Data query method, device, system, equipment and medium

Abstract

The present disclosure provides a data query method, device, system, equipment and medium. The method includes: obtaining a target termination condition; determining the ranking of each of the Y sub-datasets based on the similarity between the query data and each of the Y sub-datasets, where Y is a positive integer greater than 1; taking the top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets as the current sub-dataset and processing the current sub-dataset until at least some of the current query indicators meet the target termination condition, and taking the updated intermediate result as the query result; wherein Z is a positive integer less than Y, and the processing includes: determining at least one sample data with the highest similarity to the query data among the sample data included in the current sub-dataset to update the intermediate result, and determining the current query indicator based on the updated intermediate result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a data query method, a data query device, a data query system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Breakthroughs in deep learning have enabled unstructured data to be represented as high-dimensional feature vectors to serve a wide range of applications. Querying high-dimensional feature vectors on large unstructured datasets (with billions of items) is challenging because it is difficult to guarantee low latency and accuracy at the same time. Therefore, Approximate Nearest Neighbor (ANN) search came into being.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a data query method, a data query device, a data query system, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a data query method is provided for querying a target number of sample data with the highest similarity to the query data in a data set. The method includes: obtaining a target termination condition, the target termination condition including a target accuracy rate and / or a target latency; determining the ranking of each of the Y sub-data sets based on the similarity between the query data and each of the Y sub-data sets, wherein the Y sub-data sets are obtained by dividing the sample data according to the similarity between the sample data in the data set, and each sub-data set includes at least one sample data corresponding to the sub-data set in the data set, wherein Y is a positive integer greater than 1; taking the top Z unprocessed sub-data sets with the highest ranking among the Y sub-data sets as the current sub-data set, and processing the current sub-data set until at least some of the indicators in the current query index meet the target termination condition, Wherein, Z is a positive integer less than Y; among the sample data included in the current sub-dataset, at least one sample data with the highest similarity to the query data is determined to update the intermediate result, and the updated intermediate result includes a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-dataset ranked not lower than the current sub-dataset; and the current query index is determined based on the updated intermediate result, and the current query index includes an accuracy rate, and the accuracy rate indicates a prediction result of the proportion of the candidate data in the updated intermediate result in the target result, and the target result is an updated intermediate result obtained after processing all sub-datasets; and the updated intermediate result is used as the query result.

[0006] According to another aspect of the present disclosure, a data query method is provided, which is used to query a target number of sample data with the highest similarity to the query data in an original data set. The method is applied to a central node in a distributed cluster, the central node is communicatively connected with a plurality of running nodes in the distributed cluster, and each of the plurality of running nodes stores a data set corresponding to the running node in the original data set. The method includes: receiving query data and a global termination condition; modifying the global termination condition to obtain a target termination condition, and sending the query data and the target termination condition to each running node, so that the running node uses the above-mentioned data query method to query the target number of sample data with the highest similarity to the query data in the data set corresponding to the running node; receiving the query result corresponding to the running node from each running node, the query result including the target number of sample data; and determining the target number of sample data with the highest similarity to the query data in the query results received from each running node as the final query result; wherein the global termination condition includes a global accuracy rate, the target termination condition includes a target accuracy rate, the global accuracy rate indicates the proportion of sample data in the final query result to the target number of sample data with the highest similarity to the query data in the original data set, and / or the global termination condition includes a global delay, and the target termination condition includes a target delay.

[0007] According to another aspect of the present disclosure, a data query device is provided, which is used to query a target number of sample data with the highest similarity to the query data in a data set. The device includes: an acquisition unit, configured to acquire a target termination condition; a first determination unit, configured to determine the ranking of each of the Y sub-data sets based on the similarity between the query data and each of the Y sub-data sets, wherein the Y sub-data sets are obtained by dividing the sample data according to the similarity between the sample data in the data set, each sub-data set includes at least one sample data corresponding to the sub-data set in the data set, and Y is a positive integer greater than 1; a processing unit, configured to use the top Z unprocessed sub-data sets with the highest ranking among the Y sub-data sets as the current sub-data set, and process the current sub-data set until at least some of the indicators in the current query index meet the target termination condition, wherein Z is a positive integer less than Y, wherein the processing unit The element includes: a first determining subunit, configured to determine at least one sample data with the highest similarity to the query data among the sample data included in the current sub-data set to update the intermediate result, the updated intermediate result includes a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-data set ranked not lower than the current sub-data set; and a second determining subunit, configured to determine the current query index based on the updated intermediate result, the current query index includes an accuracy rate, the accuracy rate indicates a predicted result of the proportion of the candidate data in the updated intermediate result in the target result, the target result is the updated intermediate result obtained after processing all sub-data sets; a result generating unit, configured to use the updated intermediate result as the query result.

[0008] According to another aspect of the present disclosure, a data query device is provided for querying a target number of sample data with the highest similarity to the query data in a data set. The device is applied to a central node in a distributed cluster. The central node is communicatively connected to multiple running nodes in the distributed cluster. Each of the multiple running nodes stores a data set in the original data set corresponding to the running node. The device includes: a first receiving unit, configured to receive query data and a global termination condition; a correction unit, configured to correct the global termination condition to obtain a target termination condition, and send the query data and the target termination condition to each running node, so that the running node uses the above-mentioned data query device to query a target number of sample data with the highest similarity to the query data in the data set corresponding to the running node; a second receiving unit, configured to receive from each running node candidate data of a target number of query results corresponding to the running node, the query result including a target number of sample data; and a second determination unit, configured to determine, from the query results received from each running node, a target number of target data of sample data with the highest similarity to the query data as a final query result, wherein the global termination condition includes a global accuracy rate, the target termination condition includes a target accuracy rate, the global accuracy rate indicates the proportion of sample data in the final query result to the target number of sample data with the highest similarity to the query data in the original data set, and / or the global termination condition includes a global delay, and the target termination condition includes a target delay.

[0009] According to another aspect of the present disclosure, a data query system is provided, including an operating node applying the above-mentioned data query device for obtaining a query result for a data set and a central node applying the above-mentioned data query device for obtaining a final query result for an original data set.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned data query method.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above data query method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above data query method when executed by a processor.

[0013] According to one or more embodiments of the present disclosure, multiple sub-datasets are obtained by dividing a data set based on the similarity between sample data and sorting them according to the similarity with the query data, and then the highest-ranked part of the unprocessed sub-datasets is used as the current sub-dataset in turn, and the intermediate results are updated using the sample data with the highest similarity to the query data in the current sub-dataset, so that the updated intermediate results include a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset, until at least some of the indicators (for example, accuracy) in the current query indicators meet the target termination conditions, thereby realizing dynamic adjustment of the data query range for different query data and preset termination conditions, thereby reducing the system response time while ensuring user requests.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram showing an exemplary scenario according to an embodiment of the present disclosure;

[0017] Figure 2 A flow chart of a data query method according to an embodiment of the present disclosure is shown.

[0018] Figure 3 Shows Figure 2 A flowchart of determining the current query index in the method shown;

[0019] Figure 4 Shows Figure 3 A flowchart of determining a predicted ranking of at least one candidate data in an updated intermediate result in the method shown;

[0020] Figure 5 Shows Figure 4 A flow chart for determining a scaling factor in the method shown;

[0021] Figure 6A-6B A schematic diagram showing query data, sub-data sets, and intermediate results according to an embodiment of the present disclosure;

[0022] Figure 7A flow chart of a data query method according to an embodiment of the present disclosure is shown.

[0023] Figure 8 Shows Figure 7 A flowchart of the method for modifying the global termination condition;

[0024] Fig. 9 A structural block diagram of a data query device according to an embodiment of the present disclosure is shown;

[0025] Fig.10 A structural block diagram of a data query device according to an embodiment of the present disclosure is shown; and

[0026] Fig.11 A structural block diagram of an electronic device of a server or client according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0029] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0030] In the prior art, when performing nearest neighbor search, the data in the database can be clustered to obtain several sub-datasets, and a fixed number (such as N) of sub-datasets to be sampled are set to obtain search results with high similarity to the query data. However, using the same sampling size for different query data will result in different accuracy indicators of the query results. For example, for query data a, in sub-datasets D1-D NWhen searching in D1-D, the accuracy of the query result may far exceed the target accuracy index (for example, the target accuracy index is 90%, and the accuracy of the query result is 95%), which leads to serious redundant calculation. N When searching in the search results, the accuracy of the query results obtained may be far less than the target accuracy index (for example, the target accuracy index is 90%, and the accuracy of the query results is 75%), resulting in poor quality of the query results.

[0031] To solve the above problems, the present invention divides the data set based on the similarity between sample data to obtain multiple sub-datasets, which are sorted according to the similarity with the query data, and then sequentially uses the highest-ranking part of the unprocessed sub-datasets as the current sub-dataset, and uses the sample data with the highest similarity with the query data in the current sub-dataset to update the intermediate results, so that the updated intermediate results include a target number of candidate data with the highest similarity with the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset, until at least some of the indicators of the current query (for example, accuracy or time consumption) meet the target termination condition, thereby realizing dynamic adjustment of the data query range for different query data and target termination conditions, thereby reducing the system response time while ensuring user requirements.

[0032] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 FIG. 1 is a schematic diagram of an exemplary scenario 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the scenario 100 includes a client 110 and a system 120. The system 120 further includes a central node 121, at least one operating node 122, 123, and 124, and a database 125. Each of the client 110, the central node 121, and the operating nodes 122, 123, and 124 may be configured to execute one or more applications, which may be on one or more physical devices.

[0034] In an embodiment of the present disclosure, the central node 121 and the execution nodes 122 , 123 , and 124 may run one or more servers or software applications that enable the data query method of the present disclosure to be executed.

[0035] In some embodiments, system 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 110 under a software as a service (SaaS) model.

[0036] exist Figure 1 In the configuration shown, the central node 121 may include one or more components that implement the functions performed by the central node 121. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user operating a client device 110 may utilize a client application to interact with the central node 121 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may be different from the system 120. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0037] The user can use the client device 110 to input parameters related to data query, such as expected query accuracy, latency, etc. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user via the interface.

[0038] The client device 110 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE ChromeOS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0039] The central node 121 and the operation nodes 122, 123 and 124 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. The central node 121 and the operation nodes 122, 123 and 124 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain a server's virtual storage device). In various embodiments, the central node 121 and the operation nodes 122, 123 and 124 may run one or more services or software applications that provide the functionality described below.

[0040] The computing units in the central node 121 and the operation nodes 122, 123, and 124 may run one or more operating systems including any of the above-mentioned operating systems and any commercially available server operating systems. The central node 121 and the operation nodes 122, 123, and 124 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0041] In some embodiments, the central node 121 may include one or more applications to analyze and consolidate the data feeds and / or event updates received from the operational nodes 122, 123, and 124. The central node 121 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client device 110.

[0042] In some embodiments, system 120 may be a server of a distributed system (e.g., a distributed cluster server), or a server combined with a blockchain. System 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0043] The system 120 may also include one or more databases 125. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 125 may be used to store information such as audio files and video files. The databases 125 may reside in various locations. For example, the database used by the server 120 may be local to the system 120, or may be remote from the system 120 and may communicate with the system 120 via a network-based or dedicated connection. The databases 125 may be of different types. In some embodiments, the databases used by the system 120 may be, for example, relational databases. One or more of these databases may store, update, and query data to and from the databases in response to commands.

[0044] In some embodiments, one or more of the databases 125 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0045] Figure 1 The client 110 and the system 120 in the scenario 100 may be configured and operated in various ways to enable the application of various methods and devices described in the present disclosure.

[0046] Figure 2 A flow chart of a data query method 200 according to an embodiment of the present disclosure is shown. The method 200 can be used to query a target number of sample data with the highest similarity to the query data in a data set, including:

[0047] Step 201, obtaining target termination conditions;

[0048] Step 202: Based on the similarity between the query data and each of the Y sub-datasets, determine the ranking of each of the Y sub-datasets, wherein the Y sub-datasets are obtained by dividing the sample data in the data set according to the similarity between the sample data in the data set, and each sub-dataset includes at least one sample data in the data set corresponding to the sub-dataset, wherein Y is a positive integer greater than 1;

[0049] Step 203: The top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets are used as the current sub-dataset, and the current sub-dataset is processed until at least some of the current query indicators meet the target termination condition, where Z is a positive integer less than Y:

[0050] Step 204: Determine at least one sample data with the highest similarity to the query data among the sample data included in the current sub-dataset to update the intermediate result, wherein the updated intermediate result includes a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-dataset of the current sub-dataset, whose ranking is not lower than that; and

[0051] Step 205: determining the current query index based on the updated intermediate result, the current query index including the accuracy, the accuracy indicating the prediction result of the proportion of the candidate data in the updated intermediate result in the target result, the target result being the updated intermediate result obtained after processing all the sub-data sets;

[0052] Step 206: Use the updated intermediate result as the query result.

[0053] In some embodiments, method 200 may be performed by Figure 1 124 in the method 200. Each operating node can, for example, obtain a corresponding data set from the database 125, obtain a target termination condition from the central node 121, and transmit the query result to the central node 121, as will be described below. It can be understood that steps 204 and 205 in the method 200 can be sub-steps of the processing step in step 203.

[0054] The sample data in the data set may be, for example, a high-dimensional feature vector, and the query data may be, for example, another high-dimensional feature vector in the semantic space where the sample data is located. According to some embodiments, the query data and the sample data may include a feature vector obtained by extracting features from at least one of unstructured data such as image data, audio data, text data, molecular structure data, and sequence data, and the feature vector may be regarded as an intermediate representation of these unstructured data. By querying the vector, it is also possible to query the unstructured data. In some embodiments, the unstructured data may first be converted into a corresponding feature vector through feature extraction, and then the converted feature vector may be directly used as the query data or sample data, or the converted feature vector may be further processed (e.g., spliced, merged) to obtain the query data or sample data. It is understandable that the query data may include feature vectors obtained by extracting features from one type of data, and the sequence data may include feature vectors obtained by extracting features from the same type or another type of data. For example, the query data includes feature vectors obtained by extracting features from images, and the sequence data may include feature vectors obtained by extracting features from images (in this case, the data query scenario is image search), and may also include feature vectors obtained by extracting features from text (in this case, the data query scenario is image search for text).

[0055] The expressions such as "space", "ball", "distance", and "volume" in the present disclosure are not intended to limit the space corresponding to the data set to a three-dimensional space. It is understandable that the "space" in the present disclosure can be a two-dimensional plane or surface, a three-dimensional space, and a higher-dimensional space, the "ball" can include a circle in a two-dimensional plane or surface, a three-dimensional sphere, and a higher-dimensional hypersphere, the "distance" can be a distance on a two-dimensional plane or surface, a distance in a three-dimensional space, and a distance in a higher-dimensional space, and the "volume" can be an area in a two-dimensional plane or surface, a volume in a three-dimensional space, and a volume in a higher-dimensional space, which are not limited here.

[0056] When representing the similarity between data (e.g., feature vectors), Euclidean distance and inner product distance are two commonly used methods. The method disclosed in the present invention supports transforming a data set so as to convert the inner product distance between sample data (or between sample data and query data) into Euclidean distance. Specifically, in some embodiments, a data set using Euclidean distance as similarity can be obtained by mapping the intermediate data in an intermediate data set using inner product distance as similarity onto a unit sphere. The spherical Euclidean distance between the first sample data and the second sample data in the data set on the unit sphere indicates the inner product (i.e., inner product distance) of the first intermediate data and the second intermediate data corresponding to the first sample data and the second sample data in the intermediate data set, respectively. By converting the similarity based on the inner product distance into the similarity based on the Euclidean distance, the use scenarios of the data query method are enriched.

[0057] In some embodiments, the data set can be further divided into multiple sub-data sets (i.e., Y sub-data sets, Y is a positive integer greater than 1) according to the similarity between the sample data in the data set (e.g., the Euclidean distance between the feature vectors), so that the sub-data set can be used as the smallest unit for querying the data later. In some embodiments, the data set can be divided by a clustering method, for example, the k-means algorithm (k-means). In a uniformly distributed data set, the use of k-means can achieve multiple uniformly divided sub-data sets in a short time. In some embodiments, the data set can be divided using a KD tree. All sample data (e.g., feature vectors) can be placed in a cube (each sub-data set corresponds to a cube), and the distance between the query data and the center of mass of the cube can be used as the similarity between the query data and the sub-data set. It is understandable that, in addition to the above method, the data set can also be divided in other ways to obtain multiple sub-data sets, and each sub-data set includes at least one sample data corresponding to the sub-data set in the data set, which is not limited here. In addition, the sub-data set can also be referred to as a sample cluster. It is understandable that for the same data set, it can be divided once, and the same sub-data set can be used in subsequent queries, and when the data set changes, it can be divided again. It is understandable that when performing subsequent data queries, one or more sub-data sets can be used as units for query.

[0058] According to some embodiments, the sample data is approximately uniformly distributed in the spatial range corresponding to the data set. Multiple sub-datasets may each have a corresponding spatial range that does not overlap, and the spatial ranges of the multiple sub-datasets constitute the spatial range corresponding to the entire data set, and there is a boundary between the spatial ranges corresponding to two adjacent sub-datasets. In addition, each sample data and query data in the data set may each have a corresponding spatial position. In some embodiments, the multiple sub-datasets obtained by methods such as k-means are polyhedrons, and there is a boundary between the spatial ranges corresponding to two adjacent sub-datasets.

[0059] According to some embodiments, step 201, obtaining a target termination condition may include: receiving a target accuracy and / or a target delay.

[0060] When searching for the top N sample data with the highest similarity to the query data q in a data set, if each sample data in the data set is queried, the query result is the top N sample data with the highest similarity to the query data q in the data set, but when the amount of data is large, it often cannot meet the time performance requirements. When using ANN nearest neighbor search, no longer query each sample data in the data set, but divide the data set into multiple sub-data sets, and query only in some sub-data sets. The query result is the top N sample data with the highest similarity to the query data q among the sample data contained in the queried sub-data sets. At this time, the N sample data in the query result is likely not the sample data with the highest similarity to the query data in the entire data set. The proportion of the sample data in the query result to the target number of sample data with the highest similarity to the query data in the data set can be considered as the accuracy of the query result. If the query result is obtained by querying P sub-data sets, as P increases, the accuracy of the query result increases and the time consumption increases.

[0061] The target termination condition is used to characterize what accuracy and / or query time is acceptable. The target accuracy is the accuracy of the acceptable query result. The target accuracy may include, for example, indicators such as recall, precision, and accuracy. The target latency may be, for example, the maximum acceptable query time. It is understood that in addition to the above-mentioned target accuracy and target latency, the target termination condition may also include other conditions, such as the maximum number of sub-data sets to be queried, etc., which are not limited here.

[0062] In some embodiments, the target accuracy and / or target delay are input by the user. In some embodiments, the central node receives user input, the operating node can receive the target termination condition from the central node, and the target accuracy and / or target delay are determined based on the user input (for example, the accuracy or delay input by the user is directly used as the target accuracy or target delay, or the target accuracy or target delay is obtained by adjusting the accuracy or delay input by the user). In some embodiments, the target termination condition can also be obtained by the central node in other ways, which is not limited here.

[0063] In some embodiments, the query data may be input by a user, may be obtained from a central node, or may be determined by other means, which are not limited herein.

[0064] In some embodiments, the target quantity may be input by a user, may be obtained from a central node, or may be a default value, which is not limited here.

[0065] In some embodiments, step 202, based on the similarity between the query data and each of the Y sub-datasets, determines the ranking of each of the Y sub-datasets. For example, the Y sub-datasets can be sorted based on the spatial distance (e.g., Euclidean distance) between the query data and each of the Y sub-datasets to obtain the ranking of each of the Y sub-datasets.

[0066] It can be understood that the target number of sample data with the highest similarity to the query data in the data set has a greater probability of appearing in the sub-dataset that is closer to the query data. After determining the ranking of the sub-datasets, queries can be performed in the sub-datasets according to the ranking order of the sub-datasets. The target number of sample data with the highest similarity to the query data can be obtained with the highest probability while searching fewer sub-datasets.

[0067] It is understandable that when the query data is close to the boundary between the sub-dataset A where the query data is located and other sub-datasets B, the first N sample data closest to the query data may be located in A or in B. In this case, more sub-datasets need to be queried to meet the target accuracy requirement; and when the query data is close to the center of sub-dataset A, most of the first N sample data closest to the query data may be located in A, and querying fewer sub-datasets (for example, querying only sub-dataset A) can meet the target accuracy requirement. Therefore, the number of sub-datasets to be queried can be determined according to the spatial position of the query data and the spatial features (for example, center, boundary) corresponding to the spatial ranges of the multiple sub-datasets.

[0068] In some embodiments, the spatial distance between the query data and the target sub-dataset in the Y sub-datasets may be, for example, the spatial distance between the query data and the center of the target sub-dataset (e.g., the centroid of the sample cluster), or the vertical distance between the query data and the boundary between the sub-dataset where the query data is located and the target sub-dataset, or a distance defined in other ways that can quantitatively characterize the spatial relationship between the query data and the target sub-dataset, which is not limited here. In addition, the center of the target sub-dataset may also be defined in other ways other than the centroid of the sample cluster, which is not limited here.

[0069] In some embodiments, step 203, the top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets are used as the current sub-dataset and steps 204 and 205 are performed on the current sub-dataset until at least some of the indicators in the current query index meet the target termination condition. It can be understood that the conditional loop in step 203 is performed on the two steps of the step of using the top Z unprocessed sub-datasets with the highest ranking as the current sub-dataset and the step of processing the current sub-dataset. That is to say, after processing the top Z1 unprocessed sub-datasets with the highest ranking among the Y sub-datasets, assuming that the current query index does not meet the target termination condition at this time, it is necessary to remove the processed Z1 sub-datasets from the Y sub-datasets, and use the top Z2 sub-datasets with the highest ranking among the remaining sub-datasets as the current sub-dataset and process them, until at a certain round, Z n The query is terminated when at least some of the current query indicators determined after processing the sub-dataset as the current sub-dataset meet the target termination condition, and the updated intermediate result at this time is used as the query result. It should be noted that Z1, Z2, ..., Z n The value of can be the same (for example, each loop queries 2 sub-datasets) or different, as will be described below.

[0070] After the ranking of each sub-dataset is determined based on the distance between each of the Y sub-datasets and the query data, one or more sub-datasets (i.e., Z sub-datasets, where Z is a positive integer less than Y) can be used as the current sub-dataset one by one in the order of the sub-dataset ranking from front to back, and the sample data in the current sub-dataset is queried. When there is sample data in the current sub-dataset that is closer to the query data (closer than some sample data in the intermediate result obtained after the previous sub-dataset query), the sample data in the current sub-dataset that is closer to the query data is used to replace some sample data in the intermediate result obtained after the previous sub-dataset query. After each sub-dataset is queried, it is determined whether the target termination condition is met (it can also be determined once after each K sub-datasets are queried, K can be determined according to the number of sub-datasets, for example, K is 3. Of course, this situation can also be understood as merging the K sub-datasets into a merged sub-dataset, and determining whether the target termination condition is met after each merged sub-dataset is queried). If the target termination condition is met, the query result is obtained.

[0071] It can be understood that taking Z sub-datasets as the current sub-datasets may be taking the union of the Z sub-datasets as the current sub-datasets. In addition, each time step 203 (including the determination and processing of sub-datasets, and the processing of sub-datasets includes steps 204 and 205) is performed, the value of Z may be different. In an exemplary embodiment, the value of Y is 6, and the value of Z may be set to 3 when performing the first round of query, so that the top three unprocessed sub-datasets with the highest ranking are processed as the current sub-datasets. Assuming that after the first round of query, the target termination condition has not been met, the value of Z may be set to 2, so that the top two unprocessed sub-datasets with the highest ranking are processed as the current sub-datasets for the second round of query. Assuming that after the second round of query, the target termination condition has not been met, the value of Z may be set to 1, so that the last remaining unprocessed sub-dataset is processed as the current sub-dataset for the third round of query.

[0072] In some embodiments, the size of Z can be dynamically determined based on the current query index and the target accuracy in the target termination condition. In an exemplary embodiment, the size of Z can be proportional to the difference between the current query index and the target accuracy, that is, when the current query index differs greatly from the target accuracy, a larger Z value can be set, and when the current query index is close to the target accuracy, a smaller Z value can be set. It can be understood that the above is only an exemplary way to set the Z value, and those skilled in the art can set a fixed or dynamic Z value according to their needs, which is not limited here.

[0073] In the present disclosure, q may refer to query data, k may refer to the target number, and C i It can refer to the sub-dataset that has the ith largest similarity to the query data q (or the ith smallest spatial distance to the query data q).

[0074] Query C1, C2, ... in turn for sample data with high similarity to the query data q, and obtain the top k sample data with the highest similarity to the query data q in C1 (updated intermediate result M1), the top k sample data with the highest similarity to the query data q in C1 and C2 (updated intermediate result M2), and so on. The intermediate results can be initialized to empty. After the query in C1 is completed, the updated intermediate result M1 is obtained; the current query index is determined based on M1, and whether the termination condition is met is determined according to the current query index; if it is met, the query in C2 is not continued; if it is not met, the query in C2 is continued. After the query in C2 is completed, the updated intermediate result M2 is obtained; the current query index is determined based on M2, and whether the termination condition is met is determined according to the current query index; if it is met, the query in C3 is not continued; if it is not met, the query in C3 is continued. When the current query index determined based on M2 meets the termination condition, M2 is used as the query result. The query result is most likely not the top k sample data in the dataset with the highest similarity to the query data q. For example, k-2 of the data are the top k sample data in the dataset with the highest similarity to the query data q, and the other two are the k+3rd and k+8th sample data in the dataset with the highest similarity to the query data q.

[0075] The current query indicator may include accuracy, and the target termination condition may also include target accuracy. If the accuracy included in the current query indicator is greater than or equal to the target accuracy, the query may be terminated without querying the next sub-dataset.

[0076] In addition to accuracy, the current query indicator can also include latency, and the target termination condition can also include target latency. If the latency included in the current query indicator is greater than or equal to the target latency, the query can be terminated and the next sub-dataset will not be queried. At this time, accuracy can be understood as no longer being used as a condition for determining when to terminate the query. When returning query results, accuracy can be returned at the same time for user reference.

[0077] The target number k of sample data with the highest similarity to the query data q among the sample data included in the sub-datasets ranked not lower than the current sub-dataset is called candidate data, that is, the sample data included in the intermediate result is called candidate data. It can be understood that the sub-datasets ranked not lower than the current sub-dataset can be all sub-datasets ranked not lower than any sub-dataset among the highest-ranked Z unprocessed sub-datasets.

[0078] After determining the ranking of each sub-dataset, the sub-dataset closest to the query data can be first processed as the current sub-dataset. In some embodiments, the sub-dataset closest to the query data, i.e., the corresponding spatial range, includes the sub-dataset of the spatial position corresponding to the query data (e.g., the query data q falls into C1). The sample data in the current sub-dataset can be retrieved, and the target number of sample data with the highest similarity to the query data can be selected as the current intermediate result, i.e., the target number of candidate data. It is understandable that before processing the sub-dataset closest to the query data, the intermediate result can be an empty set, or can include the target number of sample data randomly selected in the data set.

[0079] In the present disclosure, v1, v2, ..., v k Indicates the target number k of candidate data in the intermediate result. In an exemplary embodiment, the target number k is 4, and then 4 sample data can be determined in the sub-dataset C1 as candidate data in the intermediate result, namely v1, v2, v3 and v4.

[0080] In some embodiments, each candidate data in the updated intermediate results may correspond to an intermediate ranking of the candidate data in the updated intermediate results, and may also correspond to a similarity with the query data. The intermediate ranking may be determined based on the similarity between the candidate data and the query data.

[0081] In the present disclosure, the candidate data v can be used j The superscript j indicates the intermediate ranking. In an exemplary embodiment, the updated intermediate result includes four candidate data, v1, v2, v3 and v4, where v1 is closest to the query data q, v2 is second, v3 is third, and v4 is farthest. That is, the intermediate rankings of the four candidate data v1-v4 can be 1-4 respectively.

[0082] In each subsequent round of querying the sub-datasets, the top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets can be used as the current sub-dataset for processing. The sample data in the current sub-dataset can be retrieved and the current intermediate result can be updated, so that the updated intermediate result includes the target number of candidate data with the highest similarity to the query data among the sample data included in the sub-datasets with a ranking not lower than that of the current sub-dataset.

[0083] According to some embodiments, step 204, among the sample data included in the current sub-dataset, at least one sample data having the highest similarity to the query data is determined to update the intermediate result, and the updated intermediate result includes a target number of candidate data having the highest similarity to the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset.

[0084] In some embodiments, step 204 may include: for each sample data included in the current sub-data set, in response to determining that the similarity between the sample data and the query data is higher than the similarity between each candidate data in the current intermediate result and the query data, using the sample data to replace the candidate data with the lowest similarity to the query data in the current intermediate result. Thus, the updated intermediate result can be obtained by traversing the sample data of the current sub-data set only once, thereby achieving more efficient sub-data set retrieval and intermediate result update.

[0085] In an exemplary embodiment, the current intermediate result includes four candidate data v1-v4 retrieved in C1, and sample data v3' and v4' closer to q than v3 are retrieved in the current sub-dataset C2, then v3' and v4' can be used to replace v3 and v4 to obtain updated intermediate results v1, v2, v3' and v4'. If the search continues in the current sub-dataset C2 at this time, and sample data v4" closer to q than v4' is retrieved, v4' can be further replaced with v4" to obtain updated intermediate results v1, v2, v3' and v4". If the current query indicators corresponding to the intermediate results v1, v2, v3' and v4" do not meet the target termination conditions, further search can be performed in the sub-dataset C3.

[0086] In some embodiments, other methods may be used to update the intermediate results. For example, the current sub-dataset C may be updated. i Determine the k sample data with the highest similarity to the query data q, or the candidate data v with the lowest similarity to the query data q in the current intermediate result. k The sample data of C is then updated based on the sample data queried (for example, the sample data queried in this round is updated with i-1 Determine the k sample data with the highest similarity to the query data q for comparison, and select the top k sample data with the highest similarity therefrom), which is not limited here.

[0087] In addition, it can be understood that "updating" the intermediate result with at least one sample data with the highest similarity to the query data determined by the sample data included in the current sub-dataset does not mean that at least one sample data with the highest similarity to the query data determined by the sample data included in the current sub-dataset will definitely be added to the intermediate result. If the similarity between at least one sample data and the query data is higher than that of at least one candidate data in the intermediate result, it will be added to the intermediate result, otherwise it will not be added to the intermediate result.

[0088] In some embodiments, the current sub-dataset C iThe similarity between each sample data and the query data q is lower than that of the candidate data v in the intermediate result. k If the intermediate result is similar to the query data q, the original intermediate result can be kept unchanged after being "updated".

[0089] For example, the intermediate result obtained after querying C1 includes the top four sample data v1, v2, v3, and v4 in C1 that are most similar to the query data q. Continuing to query C2, the four sample data v5, v6, v7, and v8 in C2 that are most similar to the query data q are obtained. However, the similarity between v5, v6, v7, and v8 and q is lower than the similarity between v1, v2, v3, and v4 and q. Then, v5, v6, v7, and v8 are used to update the intermediate result obtained after querying C1. The updated intermediate result is still v1, v2, v3, and v4. At this time, v1, v2, v3, and v4 can be understood as the intermediate results obtained after querying C1 and C2.

[0090] Then, step 205 can be performed. After obtaining the updated intermediate result, the current query index can be determined based on the updated intermediate result, and the current query index includes the accuracy. The accuracy indicates the proportion of candidate data in the updated intermediate result in the target result. The target result is the updated intermediate result obtained after processing all sub-data sets, that is, the target number of sample data in the data set with the highest similarity to the query data. In other words, the accuracy indicates the proportion of candidate data in the updated intermediate result in the target number of sample data in the data set with the highest similarity to the query data.

[0091] In some embodiments, the current query index is first determined based on the updated intermediate result, and then the current query index is compared with the target termination condition to determine whether at least part of the current query index meets the target termination condition. If it meets, the next sub-dataset is not queried, and the updated intermediate result is used as the query result, that is, the updated intermediate result is returned as the target number of sample data with the highest similarity to the query data in the data set. If it does not meet, continue to query the next sub-dataset.

[0092] Since the updated intermediate result contains the target number of candidate data with the highest similarity to the query data among the sample data included in all sub-datasets whose ranking is not lower than that of the current sub-dataset, therefore, when the current sub-dataset is not the sub-dataset with the lowest similarity to the query vector q (that is, when all sub-datasets are not queried), among these candidate data, only a part may belong to the k ground truth sample data with the highest similarity to the query data in the entire dataset, which are called true positives (TP); the other part does not belong to the ground truth sample data, which are called false positives (FP). By predicting which of the intermediate results belong to true positives, the prediction results of TP and FP can be obtained, and then the accuracy corresponding to the current intermediate result can be predicted.

[0093] In addition, some of the true sample data are not included in the intermediate results, which are called false negatives (FN). Since the number of candidate data and true sample data in the intermediate results can be the target number k, the values ​​of FP and FN can be the same, and TP+FP=TP+FN=k, so precision=TP / (TP+FP) can be the same as recall=TP / (TP+FN).

[0094] In some embodiments, a person skilled in the art may determine the current query index based on the updated intermediate results through a machine learning-based method, for example, inputting the candidate data in the updated intermediate results into the model, and the model outputting a prediction result for the current accuracy rate, and may also output, for each candidate data, a prediction result of whether the candidate data belongs to a target number of sample data of the closest query data in the data set, and calculate the accuracy rate based on the prediction result corresponding to each candidate data (for example, 3 out of 5 candidate data are predicted to be sample data of the target number of sample data that do not belong to the closest query data in the data set, then the accuracy rate is 40%), and so on.

[0095] It is understandable that those skilled in the art can also determine the current query index based on the updated intermediate results by other methods other than machine learning, which are not limited here. For example, the predicted ranking of candidate data a is determined by calculation, and the predicted ranking indicates the predicted result of the ranking of the similarity between candidate data a and query data q in the target result. If the predicted ranking is greater than k, the candidate data does not belong to the target number of sample data closest to the query data in the data set, so the accuracy can be determined.

[0096] It is understandable that the accuracy rate in the target termination condition can also be the error rate. Correspondingly, the current query index also includes the error rate. The effects of using the accuracy rate and the error rate are equivalent.

[0097] It can be seen that the embodiment of the present disclosure queries the sub-datasets in descending order of similarity between the sub-datasets and the query data q. i The current intermediate result obtained after the query of the sub-datasets determines the current query index including the accuracy, and determines whether it is necessary to continue querying C by judging whether at least part of the current query index meets the target termination condition. i+1 The sub-datasets are retrieved, so as to dynamically adjust the data query range according to different query data and target termination conditions, so that the query results can meet the target termination conditions while reducing the system response time.

[0098] According to some embodiments, Figure 3 As shown, step 205, determining the current query index based on the updated intermediate result includes: step 301, determining the predicted ranking of at least one candidate data in the updated intermediate result, the predicted ranking indicating the predicted result of the ranking of the similarity between the corresponding candidate data and the query data in the target result; and step 302, determining the accuracy rate based on the number of candidate data whose predicted ranking value is not greater than the target number and the target number. Thus, by determining the predicted ranking of the candidate data, the predicted result of the ranking of the candidate data in all the sample data can be obtained without querying all the sample data in the data set, and then the accuracy rate can be determined according to the predicted result and further judged whether the current intermediate result meets the target termination condition.

[0099] In some embodiments, step 302, determining the accuracy rate based on the number of candidate data whose predicted ranking values ​​are not greater than the target number and the target number may include: determining the ratio of the number of candidate data whose predicted ranking values ​​are not greater than the target number and the target number as the accuracy rate.

[0100] The number of candidate data whose predicted ranking value is not greater than the target number may be the number of candidate data that can be ranked before the target number in the order of the distances between all sample data in the data set and the query data among the current target number of candidate data, that is, the prediction result of TP. Therefore, the ratio of the number of candidate data whose predicted ranking value is not greater than the target number to the target number may be the prediction result of the precision or recall rate.

[0101] In one example, in step 301, the predicted ranking of each candidate data in the updated intermediate result can be calculated, and then the number of candidate data whose predicted ranking value is not greater than the target number is determined. In an exemplary embodiment, the target number k is 4, and the predicted rankings of the candidate data can be v1:1, v2:2, v3:5, v4:7, respectively, then it can be determined that the number of candidate data whose predicted ranking value is not greater than the target number k is 2, and the precision / recall rate can be further calculated to be 2 / (2+2)=50%.

[0102] In one example, in step 301, the predicted ranking of part or even only one candidate data in the updated intermediate result is calculated, and then the number of candidate data whose predicted ranking value is not greater than the target number is determined. For example, the predicted ranking of a certain candidate data is not greater than the target number and is the maximum value among the predicted rankings of the candidate data included in the updated intermediate result. According to the intermediate ranking of the candidate data in the updated intermediate result, the number of candidate data whose predicted ranking value is not greater than the target number can be determined. For example, if the target number k is 4, and the predicted ranking of the candidate data v4 included in the updated intermediate result is 4, it can be inferred that the predicted rankings of v1, v2, and v3 are all less than 4, and the number of candidate data whose predicted ranking value is not greater than the target number is 4. For another example, if the target number k is 4, and the predicted ranking of the candidate data v1 included in the updated intermediate result is 9, it can be inferred that the predicted rankings of v2, v3, and v4 are all greater than 9, and the number of candidate data whose predicted ranking value is not greater than the target number is 0. For another example, the target number k is 4, and the predicted ranking of the candidate data v2 included in the updated intermediate result is 4. It can be inferred that the predicted ranking of v1 is less than 4, and the predicted rankings of v3 and v4 are both greater than 4. The number of candidate data whose predicted ranking values ​​are not greater than the target number is 2.

[0103] In one embodiment, the predicted ranking of the candidate data can be determined by the high-dimensional spatial geometric relationship between the query data, the corresponding sub-data set, and the candidate data in the current sub-data set. Figure 4As shown, step 301, determining the predicted ranking of at least one candidate data in the updated intermediate result may include: step 401, determining the target candidate data in the updated intermediate result, and performing the following operations for the target candidate data until all candidate data in the updated intermediate result are used as target candidate data or the end condition is met; step 402, determining the scale factor according to the spatial position of the query data, the spatial position of the target candidate data, and the spatial characteristics corresponding to the spatial ranges of each of the Y sub-datasets; and step 403, determining the predicted ranking of the target candidate data according to the intermediate ranking and the scale factor of the target candidate data. Among them, the end condition may include that the predicted ranking of the target candidate data is not greater than the target number and is the maximum value among the predicted rankings of the candidate data included in the updated intermediate result. It can be understood that steps 402-403 can be sub-steps of step 401.

[0104] As described above, it can be assumed that the sample data in the data set is uniformly distributed or approximately uniformly distributed. Then, based on the spatial position of the query data, the spatial position of the target candidate data, and the spatial features corresponding to the spatial ranges of the Y sub-data sets, the probability that the sample data in the data set that is closer to the query data than the target candidate data falls within the current sub-data set can be inferred, and the corresponding relationship between the intermediate ranking of the target candidate data and the predicted ranking, i.e., the proportional factor, can be inversely derived based on the probability. In this way, the predicted ranking can be inferred by using the intermediate ranking of the candidate data being retrieved.

[0105] In some embodiments, the spatial feature corresponding to the spatial range of the sub-dataset may include, for example, the centroid of the sample cluster, or may include the boundary of the sub-dataset, which is not limited here.

[0106] In some embodiments, a machine learning model can be trained to achieve the determination of the proportional factor or predicted ranking corresponding to each candidate data based on the spatial information such as the above-mentioned spatial position, spatial range, spatial features, etc., and a geometric method can also be used for prediction. In contrast, the former is time-consuming, and the latter is more friendly to improving retrieval performance. For example, the volume proportion of the spatial range composed of the data set ranked not lower than the current sub-data set in the spatial range centered on the spatial position of the query data q and the spatial position of the target candidate data as the edge (can be determined by calculating the volume, or estimated by sampling or other methods), or the area proportion of the intersection of the spatial range composed of the data set ranked not lower than the current sub-data set in the closed surface (which can also be a curve) centered on the spatial position of the query data q and the spatial position of the target candidate data as the edge, can be determined as the proportional factor of the target candidate data. It can be understood that in addition to the above-mentioned method, other types of methods can also be used to determine the proportional factor or predicted ranking corresponding to the candidate data based on the above-mentioned spatial information, which is not limited here.

[0107] Furthermore, assuming that the sample data in the data set are uniformly distributed or approximately uniformly distributed, the number of sample data in a geometric body (or spatial range) is directly proportional to its volume. Therefore, the volume / area of ​​the corresponding spatial range can be calculated or estimated, and the ratio of volume / area can be used as the above-mentioned proportional factor.

[0108] According to some embodiments, the first spatial range corresponding to the target candidate data is determined with the spatial position of the query data as the center and the spatial position of the target candidate data as the edge. In one example, a sphere determined with the spatial position of the query data as the center and the distance from the query data to the target candidate data as the radius is the first spatial range corresponding to the target candidate data. The second spatial range is the intersection of the spatial range corresponding to at least one candidate sub-dataset ranked not lower than the current sub-dataset and the first spatial range. Figure 5 As shown, step 402, determining the scaling factor based on the spatial position of the query data, the spatial position of the candidate data being retrieved, and the spatial features corresponding to the spatial ranges of each of the Y sub-datasets may include: step 501, determining the volume of the first spatial range; step 502, determining the volume of the second spatial range; and step 503, determining the ratio of the volume of the first spatial range to the volume of the second spatial range as the scaling factor.

[0109] It can be understood that the first spatial range can, for example, represent the spatial range formed by the spatial positions corresponding to the sample data that are closer to the query data than the target candidate data, or the spatial range that is similar thereto, and the number of sample data included in the first spatial range can represent the predicted ranking of the target candidate data (because the target candidate data is located at the edge of the first spatial range, the predicted ranking of the sample data in the first spatial range should be higher than the target candidate data). The number of sample data included in the second spatial range can represent the intermediate ranking of the target candidate data, because the target candidate data is located at the edge of the first spatial range, and the intermediate ranking of the sample data in the second spatial range should be higher than the target candidate data. Therefore, the ratio of the volume of the first spatial range to the volume of the second spatial range can be used to estimate the ratio of the predicted ranking and the intermediate ranking of the target candidate data.

[0110] The first spatial range may have an intersection X1 (second spatial range) with the spatial range corresponding to at least one candidate sub-dataset ranked not lower than the current sub-dataset, and may also have an intersection X2 with the spatial range corresponding to the sub-dataset ranked lower than the current sub-dataset. Among them, the intersection of the first spatial range and the spatial range corresponding to at least one candidate sub-dataset ranked not lower than the current sub-dataset has been searched, and theoretically the sample data therein has been included in the updated intermediate result; while the spatial range corresponding to the sub-dataset ranked lower than the current sub-dataset has not been searched, and theoretically the sample data therein is not included in the updated intermediate result. The greater the proportion of X2 in the first spatial range (i.e., X1+X2), the smaller the proportion of X1 in the first spatial range, indicating that when searching in the sub-dataset ranked lower than the current sub-dataset, the greater the possibility of retrieving sample data with a higher similarity than the target candidate data, and the more the predicted ranking of the target candidate data should be moved back relative to the intermediate ranking.

[0111] According to some embodiments, the first spatial range is a sphere with the spatial position of the query data as the sphere center and the spatial distance between the spatial position of the query data and the spatial position of the candidate data being retrieved as the radius. The volume of the first spatial range can be calculated using a sphere volume formula.

[0112] In some embodiments, how the sub-datasets are divided is known, that is, the boundaries of each sub-dataset are known, and the boundaries of the second spatial range include partial boundaries of at least one candidate sub-dataset that is ranked not lower than the current sub-dataset, so the volume of the second spatial range can be directly calculated using these boundaries and the first spatial range (for example, by calculus, etc.). It is understandable that the volume of the second spatial range can also be calculated in other ways, which are not limited here. It is understandable that the more sub-datasets are queried, the closer the scale factor is to 1, and the higher the accuracy corresponding to the current intermediate result.

[0113] In an exemplary embodiment, Fig. 6A As shown, the current sub-dataset is C1, and the intermediate result includes four candidate data v1, v2, v3 and v4 in C1, but does not include sample data v3' and v4' in C2. When calculating the scale factor corresponding to the target candidate data v4, the corresponding first spatial range can be a sphere S1 with the query data q as the center and the distance between the query data q and v4 as the radius, and its volume can be expressed as V S The second spatial range may be the intersection of the sphere S1 and the current sub-dataset C1 (the shaded portion in the figure), and its volume may be represented as V1.

[0114] In an exemplary embodiment, Figure 6BAs shown, the current sub-dataset is C2. When the intermediate result obtained after querying C1 is updated, the sample data v3' and v4' in C2 replace the candidate data v3 and v4 in the original intermediate result that are farther away from the query data q. Therefore, the updated intermediate result includes four candidate data v1, v2, v3' and v4'. When calculating the proportional factor corresponding to the target candidate data v4', the corresponding first spatial range can be a sphere S2 with the query data q as the center and the distance between the query data q and v4' as the radius. The second spatial range can be the intersection of the sphere S2 and the sub-datasets C1 and C2 (the shaded part in the figure, that is, the union of C1 and C2, and then the intersection of S2 and C1∪C2; or, the intersection of C1 and S2, the intersection of C2 and S2, and then the union of C1∩S2 and C2∩S2).

[0115] In some embodiments, since the shape of the second spatial range is usually irregular, the volume of the part of the first spatial range that is complementary to the second spatial range can be calculated, and then the volume of the second spatial range can be determined based on the difference between the volume of the first spatial range and the volume of this part.

[0116] According to some embodiments, each of the at least one third spatial range may be the intersection of the spatial range corresponding to another sub-data set ranked higher than the current sub-data set and the first spatial range. Step 502, determining the volume of the second spatial range may include: determining the volume of at least one third spatial range; and determining the volume of the second spatial range based on the difference between the volume of the first spatial range and the sum of the volumes of at least one third spatial range. Thus, a simpler way to calculate the volume of the second spatial range is provided (for example, there are 5 sub-data sets in total, and the sum of the volumes of at least one third spatial range can be calculated by method 1: C3, C4, C5 are taken as the union, and then S2 is taken as the intersection of the union; it can also be calculated by method 2, C3 and S2 are taken as the intersection, C4 and S2 are taken as the intersection, C5 and S2 are taken as the intersection, and the three intersections are taken as the union).

[0117] It is understandable that the proportional factor can also be directly calculated based on the volume sum V3 of at least one third spatial range. For example, proportional factor = V S / (V S -V3)=1 / (1-V3 / V S ).

[0118] In an exemplary embodiment, Fig. 6AAs shown, the current sub-dataset is C1, and at least one third spatial range may include a third spatial range corresponding to the intersection of sphere S1 and sub-dataset C2, and a third spatial range corresponding to the intersection of sphere S1 and sub-dataset C3, whose volumes are V2 and V3 respectively. Since sphere S1 and sub-dataset C4 have no intersection, there is no corresponding third spatial range. V can be calculated S , V2 and V3, and V S –(V2+V3) is used as the volume of the second spatial range. The shapes of S1∩C2 and S1∩C3 are similar to the spherical cap, so the volume formula of the spherical cap can be used to substitute the parameters for calculation, and the calculation results can be used as the approximate results of V2 and V3.

[0119] In some embodiments, the calculation of the volume of the third spatial range can be further optimized. For each third spatial range, the volume of the third spatial range can be further optimized based on the shortest distance from the query data to the third spatial range (for example, the shortest distance from the query data q to C1 and C2). m The distance between the boundaries, C m is the mth sub-dataset that intersects with S1) and query data q to target candidate data v j The volume of the spherical cap is estimated by the arccosine (arccos) of the ratio of the distance (radius of sphere S1). The estimates of the volume of each spherical cap can then be combined to obtain an estimate of the total volume of at least one third spatial range. It should be noted that when using the above volume approximation method, the volume of the first spatial range can be determined as 1. In other words, the estimate of the volume of the third spatial range obtained using the above volume approximation method is actually an estimate of the ratio of the true volume of the third spatial range to the true volume of the first spatial range.

[0120] According to some embodiments, determining the volume of at least one third spatial range may include: determining the shortest spatial distance from the spatial position of the query data to each third spatial range; estimating the volume of the third spatial range based on the arccosine value of the ratio of the shortest spatial distance corresponding to each third spatial range and the spatial distance between the spatial position of the query data and the spatial position of the target candidate data.

[0121] Since the time complexity of calculating the arccosine value is a constant time, the above method can significantly reduce the overall time consumption of the data query process.

[0122] The above optimization process can be explained by the following formula.

[0123]

[0124] Among them, i is the ranking of the current sub-dataset, C i is the current sub-dataset, S iis the jth item v in the updated intermediate result j The corresponding ball, is the jth item v in the updated intermediate result j The corresponding scaling factor, P j (m) is the ball S i The mth sub-dataset C with intersection m The spherical cap obtained by approximating the corresponding third space range, l is all the spherical caps that are related to the sphere S i The number of overlapping sub-datasets, is the union of the spherical caps corresponding to the i+1th sub-dataset to the lth sub-dataset, that is, the union of at least one third spatial range, V(G) is the volume of the geometric body G, It is the sum of the volumes of the spherical caps corresponding to at least one third spatial range.

[0125] The first less than or equal to in the above formula takes into account that there may be intersections between spherical caps, and the second less than or equal to in the above formula is a further optimization of the calculation of the spherical cap volume, where:

[0126]

[0127] in,

[0128] db m To query data q to C1 and C m The distance from the boundary of m The shortest distance), λ i (j) is the current sub-dataset C i When querying data q to the jth item v in the updated intermediate result j a and b are parameters obtained by offline fitting, and a×U is a geometric upper limit of the volume ratio of the union of at least one third spatial range and the first spatial range.

[0129] Through the above method, the estimated result of the volume of the third spatial range (or the volume ratio of the third spatial range to the first spatial range) can be quickly obtained, and then the result can be used to obtain the estimated result of the volume ratio of the first spatial range to the second spatial range, that is, the proportional factor.

[0130] In some embodiments, step 403, determining the predicted ranking of the candidate data being retrieved based on the intermediate ranking and the scaling factor of the target candidate data, may include: multiplying the intermediate ranking of the candidate data and the scaling factor to obtain the predicted ranking of the candidate data.

[0131] According to some embodiments, in step 401, target candidate data is determined from the updated intermediate results, and each candidate data in the updated intermediate results is determined as the target candidate data in turn according to the intermediate ranking of the candidate data in the updated intermediate results from low to high or from high to low; or, according to the binary search method, the target candidate data is determined from the updated intermediate results.

[0132] It can be understood that our goal is to find such candidate data included in the updated intermediate result: its predicted ranking is not greater than the target number, and its predicted ranking is the maximum value among the predicted rankings of the candidate data included in the updated intermediate result.

[0133] If we want to find such candidate data, we can, of course, determine the candidate data in the updated intermediate results as the target candidate data in order of their intermediate ranking from low to high or from high to low. For example, we can calculate the predicted ranking of the candidate data with the highest intermediate ranking. If the predicted ranking is less than the target number k, we will continue to use the candidate data with the next intermediate ranking as the target candidate data until the predicted ranking of the target candidate data is greater than k. In this case, the previous target candidate data is the candidate data we are looking for. For another example, we can calculate the predicted ranking of the candidate data with the lowest intermediate ranking. If the predicted ranking is greater than the target number k, we will continue to use the candidate data with the previous intermediate ranking as the target candidate data until the predicted ranking of the target candidate data is less than k. In this case, the target candidate data is the candidate data we are looking for.

[0134] A more efficient method is to use the binary search method to determine the target candidate data. For example, initialize the two subscripts p and q to 0 and the target number k, determine the candidate data ranked p+q / 2 in the intermediate result as the target candidate data, calculate the predicted ranking of the target candidate data, and if the predicted ranking is greater than k, set q=p+q / 2, if not, set p=p+q / 2, continue to determine the candidate data ranked p+q / 2 in the intermediate result as the target candidate data, and calculate the predicted ranking of the target candidate data. In this way, the range [p,q] can be gradually narrowed until p=q, at which point we have found the candidate data we are looking for.

[0135] By using the binary search method, fast retrieval can be completed with a time complexity of O(l×log(k)) and a space complexity of O(1).

[0136] After obtaining the current query index based on the updated intermediate result, the current query index and the target termination condition can be compared to determine whether to terminate the query. According to some embodiments, at least some of the current query indexes satisfying the target termination condition may include: the accuracy in the current query index is greater than the target accuracy and / or the query time in the current query index is greater than the target delay. The query time can be, for example, the total time from the start of the query to the current moment. For example, a timer can be maintained to record the total time from the start of the query to the current moment.

[0137] In some embodiments, in response to detecting that at least some of the current query indicators do not meet the target termination conditions, the next unprocessed sub-dataset can be used as the current sub-dataset for further processing; in response to detecting that at least some of the current query indicators meet the target termination conditions, the query can be terminated and the corresponding results can be returned, such as the target number of candidate data included in the current intermediate results.

[0138] In some embodiments, the query results (target number of candidate data) obtained by the running node can be transmitted to the corresponding central node. In addition, when transmitting these candidate data, the distance between each of these candidate data and the query data can be transmitted together, so that the central node can summarize them, as described below.

[0139] In summary, multiple sub-datasets are obtained by dividing the data set based on the similarity between sample data and sorting them according to the similarity with the query data, and then the highest-ranked part of the unprocessed sub-datasets is used as the current sub-dataset in turn, and the intermediate results are updated using the sample data with the highest similarity to the query data in the current sub-dataset, so that the updated intermediate results include a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset, until at least some of the indicators in the current query indicator (for example, accuracy) meet the target termination condition, thereby realizing dynamic adjustment of the data query range for different query data and target termination conditions, thereby reducing the system response time while ensuring user requests.

[0140] Figure 7A flow chart of a data query method 700 according to an embodiment of the present disclosure is shown. Method 700 can be used to query a target number of sample data with the highest similarity to the query data in the original data set. The method can be applied to a central node in a distributed cluster, the central node is in communication connection with at least one running node in the distributed cluster, and each of the at least one running node stores a data set in the original data set corresponding to the running node. In some embodiments, the data set corresponding to each of the at least one running node is obtained by randomly and uniformly dividing the original data set, and the data sets corresponding to each of the at least one running node have no intersection, and their union is the original data set. Method 700 includes:

[0141] Step 701: receiving query data and a global termination condition, where the global termination condition is used to indicate a condition for terminating the query;

[0142] Step 702: modify the global termination condition to obtain the target termination condition, and send the query data and the target termination condition to each running node, so that the running node uses the aforementioned data query method to query the target number of sample data with the highest similarity to the query data in the data set corresponding to the running node;

[0143] Step 703: Receive a query result corresponding to each running node from the running node, where the query result includes a target number of sample data; and

[0144] Step 704: Determine the target number of sample data with the highest similarity to the query data from the query results received from each running node as the final query result. For example, the query results sent by multiple running nodes (each query result contains k sample data) are aggregated, and the k with the highest similarity are taken as the final query results (i.e., the final query results are obtained after aggregation and filtering).

[0145] In some embodiments, the use of a distributed solution can solve the problem of insufficient computing and storage resources of a single machine under a large data set. However, in a data query scenario, the result of a single running node meeting the accuracy (e.g., recall) requirement does not mean that the accuracy requirement is met as a whole. As shown in Table 1, in the data set corresponding to node 1, the three sample data closest to the query data are numbered 1, 4, and 7, and the three sample data actually queried are numbered 4, 7, and 10, then the recall rate corresponding to node 1 is 0.67. Similarly, the recall rates of nodes 2 and 3 are both 0.67, but after summarizing the results of the three nodes, the recall rate of the final result is only 0.33.

[0146] Table 1

[0147] Expected Results Query results Recall Node 1 1、4、7 <![CDATA[4、7、 10 ]]> 0.67 Node 2 2、5、8 <![CDATA[5、8、 11 ]]> 0.67 Node 3 3、6、9 <![CDATA[3、6、 12 ]]> 0.67 Summary 1、2、3 <![CDATA[3、 4、5 ]]> 0.33

[0148] Therefore, when a data query is processed in parallel by multiple running nodes, the target termination condition that each running node needs to achieve can be modified to be more stringent, so that the final query result after aggregation can meet the pre-set global termination condition for the entire original data set.

[0149] In some embodiments, the global termination condition (e.g., global accuracy, global delay) and the query data q may be input by the user using the client device 110. It is understandable that the global termination condition and the query data q may also be input by the user in other ways, or determined or accepted in other ways, which are not limited here. The target number k may be input by the user using the client device 110, or may be a default value, which is not limited here.

[0150] In some embodiments, the global termination condition may include a global accuracy rate, and the target termination condition may include a target accuracy rate. In some embodiments, the global termination condition may include a global delay, and the target termination condition may include a target delay.

[0151] According to some embodiments, Figure 8 As shown, step 702, modifying the global termination condition may include: step 801, gradually adding the target increment to the global accuracy to obtain the current accuracy; step 802, predicting the probability that the accuracy of the final query result is not lower than the global accuracy when the current accuracy is used as the target accuracy; and step 803, repeating the above steps until the determination probability is greater than the preset probability, and determining the current accuracy as the target accuracy. In this way, it can be ensured that the accuracy of the final result is greater than the global accuracy based on the user input or determined by other means.

[0152] In some embodiments, the target increment may be, for example, the inverse of the value of the target quantity, i.e., 1 / k, and the preset probability may be, for example, 99.9%. In addition, the probability that the accuracy of the target data of the target quantity is not lower than the global accuracy may be calculated or predicted in various ways (e.g., methods based on probability theory, methods based on machine learning, etc.), which are not limited herein.

[0153] In some embodiments, the number of sample data that hits the k true value sample data closest to the query data q among all the sample data included in the original data set in the final query result or the result after aggregation and before filtering can be defined as H. k Consider the worst case, where each computing node just meets the corresponding accuracy (in other words, the error rate of each computing node is exactly the maximum allowable error ε C ), then the random event H k =i has a probability of:

[0154]

[0155] The denominator represents the error rate at each operation node, which is ε C In the case of , all possible combinations of data query results that all operation nodes may obtain (equivalent to selecting k×(1–ε for each operation node) from the k sample data closest to the query data q in the data set of the operation node C ) sample data), the first term in the numerator represents all possible combinations of these i sample data when there are exactly i target data hitting k true value sample data (equivalent to selecting i from the k true value sample data as the sample data queried this time), and the second term in the numerator represents all possible combinations of the remaining non-true value sample data.

[0156] The probability that the accuracy of the target data of the target number is not lower than the global accuracy can be expressed as:

[0157]

[0158] Among them, (1-ε) is the global accuracy. Therefore, 1 / k can be gradually increased on the basis of the global accuracy (1-ε) until the probability calculated based on the above formula is greater than the preset probability (for example, 99.9%), and the current accuracy obtained is determined as the target accuracy.

[0159] According to some embodiments, modifying the global termination condition may include: multiplying a preset processing time by a preset coefficient to obtain a target delay. The preset coefficient may be determined according to at least one of a network transmission overhead, a number of running nodes, and a result aggregation overhead.

[0160] In some embodiments, the preset coefficient may be, for example, t = (1-a), where a is a decimal (e.g., 5%). In this way, each operating node needs to meet tighter time constraints when performing data queries, so that the difference between the global delay corresponding to the entire data query task and the target delay of each operating node can cover the overhead of network transmission, result aggregation and other processes. It is understandable that those skilled in the art can determine the above preset coefficients in various ways according to network transmission overhead, number of operating nodes, result aggregation overhead and / or other information, which is not limited here.

[0161] In some embodiments, each running node may execute method 200 to obtain query results corresponding to the running node, and may send the candidate data to the central node.

[0162] In some embodiments, at step 704, the central node may aggregate the query results received from each running node, and perform an accurate search on the aggregated data to obtain a target number of target data as a final result. In an exemplary embodiment, the query results received from each running node may be processed in a merge sorting manner to obtain the final query result. In addition, when the central node receives the query results from the running node, it may also receive the distance between each sample data contained in the query results and the query data, so that the central node can perform merge sorting.

[0163] In summary, by using multiple running nodes to process in parallel the multiple data sets obtained by splitting the original data set, a distributed fast data query solution is provided for super large data. By modifying the global termination condition to obtain the target termination condition for each running node, it is possible to ensure that the final query result meets the global termination condition.

[0164] Fig. 9A structural block diagram of a data query device 900 according to an embodiment of the present disclosure is shown. The data query device 900 can be used to query a target number of sample data with the highest similarity to the query data in a data set, including: an acquisition unit 910, configured to acquire a target termination condition, the target termination condition including a target accuracy rate and / or a target delay; a first determination unit 920, configured to determine the ranking of each of the Y sub-data sets based on the similarity between the query data and each of the Y sub-data sets, wherein the Y sub-data sets are obtained by dividing the sample data according to the similarity between the sample data in the data set, each sub-data set includes at least one sample data corresponding to the sub-data set in the data set, and Y is a positive integer greater than 1; a processing unit 930, configured to use the top Z unprocessed sub-data sets with the highest ranking among the Y sub-data sets as the current sub-data set, and process the current sub-data set until at least part of the current query index is indicated. The target termination condition is satisfied, wherein Z is a positive integer less than Y, wherein the processing unit comprises: a first determining subunit 932, configured to determine at least one sample data with the highest similarity to the query data among the sample data included in the current sub-data set to update the intermediate result, the updated intermediate result comprises a target number of candidate data with the highest similarity to the query data among the sample data included in the sub-data set ranked not lower than the current sub-data set; and a second determining subunit 934, configured to determine the current query index based on the updated intermediate result, the current query index comprises an accuracy rate, the accuracy rate indicates a predicted result of the proportion of the candidate data in the updated intermediate result in the target result, the target result is an updated intermediate result obtained after processing all sub-data sets; and a result generating unit 940, configured to use the updated intermediate result as the query result. It can be understood that the operations of unit 910-unit 940 in device 900 are similar to the operations of steps 201-step 203 and step 206 in method 200, and the operations of subunit 932-subunit 934 are similar to steps 204-step 205 in method 200, which will not be repeated here.

[0165] Fig.10The structural block diagram of a data query device 1000 according to an embodiment of the present disclosure is shown. The data query device 1000 can be used to query a target number of sample data with the highest similarity to the query data in a data set, and can be applied to a central node in a distributed cluster. The central node is communicatively connected with multiple running nodes in the distributed cluster, and each of the multiple running nodes stores a data set corresponding to the running node in the original data set. The device 1000 includes: a first receiving unit 1010, configured to receive query data and a global termination condition; a correction unit 1020, configured to correct the global termination condition to obtain a target termination condition, and send the query data, the target number and the target termination condition to each running node, so that the running node uses the above-mentioned device 900 to query the target number of sample data with the highest similarity to the query data in the data set corresponding to the running node; a second receiving unit 1030, configured to receive from each running node candidate data of the target number of query results corresponding to the running node, the query result includes candidate data of the target number of sample data of the target number; and a second determining unit 1040, configured to determine the target number of target data of the target number of sample data with the highest similarity to the query data as the candidate data of the target number of the final query result based on the query results received from each running node. It can be understood that the operations of the units 1010-1040 in the data query device 1000 are similar to the operations of steps 701-704 in the method 700, and are not repeated here.

[0166] In one embodiment, the global termination condition may include a global accuracy rate, and the target termination condition may include a target accuracy rate, wherein the global accuracy rate indicates the proportion of sample data in the final query result to the target number of sample data with the highest similarity to the query data in the original data set. In one embodiment, the global termination condition may include a global delay, and the target termination condition may include a target delay.

[0167] According to one aspect of the present disclosure, a data query system is provided, including a data query device 900 and a data query device 1000 .

[0168] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0169] In the following, combined Fig.11 Illustrative examples of such electronic devices, non-transitory computer-readable storage media, and computer program products are described.

[0170] Fig.11An example configuration of an electronic device 1100 that can be used to implement the methods described herein is shown. Each of the above-mentioned apparatuses 900 and 1000 can also be fully or at least partially implemented by the electronic device 1100 or a similar device or system.

[0171] The electronic device 1100 can be a variety of different types of devices. Examples of the electronic device 1100 include, but are not limited to, desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smart phones), notepad computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to display devices, game consoles), televisions or other display devices, automotive computers, and the like.

[0172] The electronic device 1100 may include at least one processor 1102, memory 1104, communication interface(s) 1106, a display device 1108, other input / output (I / O) devices 1110, and one or more mass storage devices 1112 that can communicate with each other, such as via a system bus 1114 or other appropriate connection.

[0173] The processor 1102 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 1102 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, the processor 1102 may be configured to obtain and execute computer-readable instructions stored in the memory 1104, mass storage device 1112, or other computer-readable media, such as program codes of an operating system 1116, program codes of application programs 1118, program codes of other programs 1120, and the like.

[0174] The memory 1104 and the mass storage device 1112 are examples of computer-readable storage media for storing instructions that are executed by the processor 1102 to implement the various functions described above. For example, the memory 1104 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 1112 may generally include a hard drive, a solid-state drive, a removable medium, including external and removable drives, a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network attached storage, a storage area network, etc. The memory 1104 and the mass storage device 1112 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 1102 as a specific machine configured to implement the operations and functions described in the examples herein.

[0175] A number of programs may be stored on mass storage device 1112. These programs include operating system 1116, one or more application programs 1118, other programs 1120, and program data 1122, and they may be loaded into memory 1104 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: method 200 and / or method 700 (including any suitable steps of method 200 and / or method 700), and / or additional embodiments described herein.

[0176] Although in Fig.11 1100, but modules 1116, 1118, 1120, and 1122, or portions thereof, may be implemented using any form of computer-readable media accessible by the electronic device 1100. As used herein, "computer-readable media" includes at least two types of computer-readable media, namely, computer-readable storage media and communication media.

[0177] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by electronic devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism. Computer-readable storage media defined herein do not include communication media.

[0178] One or more communication interfaces 1106 are used to exchange data with other devices, such as through a network, a direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a cellular network ... TM The communication interface 1106 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interface 1106 may also provide for communication with external storage devices (not shown) such as in a storage array, network attached storage, storage area network, etc.

[0179] In some examples, a display device 1108 such as a monitor may be included for displaying information and images to the user. Other I / O devices 1110 may be devices that receive various inputs from the user and provide various outputs to the user, and may include a touch input device, a gesture input device, a camera, a keyboard, a remote control, a mouse, a printer, an audio input / output device, and the like.

[0180] The technology described herein can be supported by these various configurations of the electronic device 1100, and is not limited to the specific examples of the technology described herein. For example, the function can also be implemented in whole or in part on the "cloud" by using a distributed system. The cloud includes and / or represents a platform for resources. The platform abstracts the underlying functions of the hardware (e.g., server) and software resources of the cloud. Resources may include applications and / or data that can be used when performing computing processing on a server away from the electronic device 1100. Resources may also include services provided over the Internet and / or through a subscriber network such as a cellular or Wi-Fi network. The platform can abstract resources and functions to connect the electronic device 1100 to other electronic devices. Therefore, the implementation of the functions described herein can be distributed throughout the cloud. For example, the functions can be implemented partially on the electronic device 1100 and partially through a platform that abstracts the functions of the cloud.

[0181] Although the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative and schematic, not restrictive; the present disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure and the appended claims, those skilled in the art will be able to understand and implement variations to the disclosed embodiments when practicing the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps that are not listed, the indefinite article "a" or "an" does not exclude a plurality, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". The mere fact that certain measures are recorded in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A data query method for searching a data set for a target number of sample data having the highest similarity to query data, the method comprising: Acquire a target termination condition, wherein the target termination condition includes a target accuracy and / or a target delay; Determine the ranking of each of the Y sub-datasets based on the similarity between the query data and each of the Y sub-datasets, wherein the Y sub-datasets are obtained by dividing the sample data in the data set according to the similarity between the sample data in the data set, each of the sub-datasets includes at least one sample data in the data set corresponding to the sub-dataset, and Y is a positive integer greater than 1; The top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets are used as the current sub-dataset and the current sub-dataset is processed until at least some of the current query indicators meet the target termination condition, wherein Z is a positive integer less than Y, and the processing includes: Determine, among the sample data included in the current sub-dataset, at least one sample data having the highest similarity to the query data to update an intermediate result, wherein the updated intermediate result includes a target number of candidate data having the highest similarity to the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset; and Determining a current query index based on the updated intermediate result, the current query index including an accuracy rate, the accuracy rate indicating a prediction result of a proportion of candidate data in the updated intermediate result in a target result, the target result being an updated intermediate result obtained after the processing is performed on all the sub-data sets; Use the updated intermediate result as the query result. Wherein, determining the previous query index based on the updated intermediate result includes: Determine a predicted ranking of at least one candidate data in the updated intermediate result, the predicted ranking indicating a predicted result of a ranking of similarities between the corresponding candidate data and the query data in the target result, wherein the Y sub-data sets each have a corresponding non-overlapping spatial range, each of the sample data and the query data each have a corresponding spatial position, wherein each candidate data in the updated intermediate result corresponds to an intermediate ranking of the candidate data in the updated intermediate result, the intermediate ranking is determined based on the similarity between the candidate data and the query data, and the predicted ranking of the at least one candidate data is determined based on the spatial position of the query data, the spatial position of the candidate data, the spatial ranges of each of the Y sub-data sets, and the intermediate ranking of the candidate data; and The accuracy rate is determined based on the number of candidate data whose predicted ranking values ​​are not greater than the target number and the target number.

2. The method according to claim 1, wherein: Determining a predicted ranking of at least one candidate data in the updated intermediate result includes: Determine the target candidate data in the updated intermediate result, and perform the following operations on the target candidate data until all the candidate data in the updated intermediate result are used as the target candidate data or the end condition is satisfied: Determining a scaling factor according to the spatial position of the query data, the spatial position of the target candidate data, and the spatial features corresponding to the spatial ranges of the Y sub-datasets; and Determining the predicted ranking of the target candidate data according to the intermediate ranking and the scaling factor of the target candidate data, The end condition includes that the predicted ranking of the target candidate data is not greater than the target number and is the maximum value among the predicted rankings of the candidate data included in the updated intermediate result.

3. The method according to claim 2, wherein: Determine target candidate data from the updated intermediate results, including: According to the order of the intermediate ranking of the candidate data in the updated intermediate results from low to high or from high to low, the candidate data in the updated intermediate results are determined as the target candidate data in sequence; Alternatively, the target candidate data is determined from the updated intermediate results according to a binary search method.

4. The method according to claim 2, wherein: The center of the first spatial range corresponding to the target candidate data is the spatial position of the query data, and the spatial position of the target candidate data is located at the edge of the first spatial range, The second spatial range is the intersection of the spatial range corresponding to at least one candidate sub-dataset ranked not lower than the current sub-dataset and the first spatial range. Wherein, determining the scaling factor according to the spatial position of the query data, the spatial position of the target candidate data, and the spatial features corresponding to the spatial ranges of the Y sub-datasets includes: determining the volume of the first spatial range; determining the volume of the second spatial range; and A ratio of the volume of the first spatial range to the volume of the second spatial range is determined as a scaling factor.

5. The method according to claim 4, wherein: Each of the at least one third spatial range is an intersection of the first spatial range and a spatial range corresponding to another sub-dataset ranked higher than the current sub-dataset. Wherein, determining the volume of the second spatial range includes: determining a volume of each of the at least one third spatial extent; and The volume of the second spatial range is determined based on a difference between the volume of the first spatial range and the sum of the volumes of the at least one third spatial range.

6. The method according to claim 5, wherein: Determining the volume of each of the at least one third spatial range comprises: Determine the shortest spatial distance from the spatial position of the query data to each of the third spatial ranges; The volume of the third spatial range is estimated based on the arccosine value of the ratio of the shortest spatial distance corresponding to each of the third spatial ranges to the spatial distance between the spatial position of the query data and the spatial position of the target candidate data.

7. The method according to any one of claims 4 to 6, wherein: The first spatial range is a sphere with the spatial position of the query data as the sphere center and the spatial distance between the spatial position of the query data and the spatial position of the target candidate data as the radius.

8. The method according to any one of claims 1 to 6, wherein: The data set is obtained by mapping intermediate data in the intermediate data set onto a unit sphere, wherein a spherical distance between first sample data and second sample data in the data set on the unit sphere indicates an inner product of first intermediate data and second intermediate data in the intermediate data set corresponding to the first sample data and the second sample data respectively.

9. The method according to any one of claims 1 to 6, wherein: Determining at least one sample data having the highest similarity to the query data among the sample data included in the current sub-data set to update the intermediate result includes: For each sample data included in the current sub-data set, in response to determining that the similarity between the sample data and the query data is higher than the similarity between each candidate data in the current intermediate result and the query data, the sample data is used to replace the candidate data in the current intermediate result with the lowest similarity to the query data.

10. The method according to any one of claims 1 to 6, wherein: The target termination conditions include: receiving the target accuracy and / or the target delay, wherein the target accuracy and / or the target delay are based on user input, Among them, at least some of the current query indicators meet the target termination conditions including: The accuracy rate in the current query indicator is greater than the target accuracy rate and / or the query time consumption in the current query indicator is greater than the target delay.

11. The method according to any one of claims 1 to 6, wherein: Each of the query data and the sample data includes a feature vector obtained by extracting features from one of image data, audio data, text data, molecular structure data, and sequence data.

12. A data query method, for querying a target number of sample data with the highest similarity to the query data in an original data set, the method being applied to a central node in a distributed cluster, the central node being communicatively connected to a plurality of running nodes in the distributed cluster, each of the plurality of running nodes storing a data set in the original data set corresponding to the running node, wherein: The method comprises: Receive query data and global termination conditions; Modifying the global termination condition to obtain a target termination condition, and sending the query data and the target termination condition to each of the running nodes, so that the running node uses the method according to any one of claims 1 to 11 to search for a target number of sample data with the highest similarity to the query data in the data set corresponding to the running node; receiving, from each of the running nodes, a query result corresponding to the running node, the query result including a target number of sample data; and Determining, from the query results received from each of the running nodes, a target number of sample data having the highest similarity to the query data as the final query result; The global termination condition includes a global accuracy rate, the target termination condition includes a target accuracy rate, and the global accuracy rate indicates the proportion of sample data in the final query result to the target number of sample data with the highest similarity to the query data in the original data set. And / or, the global termination condition includes a global delay, and the target termination condition includes a target delay.

13. The method according to claim 12, wherein: Modifying the global termination condition includes: Gradually add the target increment to the global accuracy to get the current accuracy; predicting the probability that the accuracy of the final query result is not lower than the global accuracy when the current accuracy is used as the target accuracy; and The above steps are repeated until the probability is greater than the preset probability, and the current accuracy is determined as the target accuracy.

14. The method according to claim 12 or 13, wherein: Modifying the global termination condition includes: The preset processing time is multiplied by a preset coefficient to obtain the target delay, wherein the preset coefficient is determined according to at least one of a network transmission overhead, a number of running nodes, and a result aggregation overhead.

15. A data query device, used to query a data set for a target number of sample data with the highest similarity to the query data, the device comprising: An acquisition unit is configured to acquire a target termination condition, wherein the target termination condition includes a target accuracy and / or a target delay; A first determining unit is configured to determine the ranking of each of the Y sub-datasets based on the similarity between the query data and each of the Y sub-datasets, wherein the Y sub-datasets are obtained by dividing the sample data in the data set according to the similarity between the sample data in the data set, each of the sub-datasets includes at least one sample data in the data set corresponding to the sub-dataset, and Y is a positive integer greater than 1; A processing unit is configured to use the top Z unprocessed sub-datasets with the highest ranking among the Y sub-datasets as the current sub-dataset, and process the current sub-dataset until at least some of the current query indicators meet the target termination condition, wherein Z is a positive integer less than Y, wherein the processing unit includes: A first determining subunit is configured to determine, among the sample data included in the current sub-dataset, at least one sample data having the highest similarity to the query data to update an intermediate result, wherein the updated intermediate result includes a target number of candidate data having the highest similarity to the query data among the sample data included in the sub-dataset whose ranking is not lower than that of the current sub-dataset; and a second determination subunit configured to determine a current query index based on the updated intermediate result, the current query index including an accuracy rate, the accuracy rate indicating a prediction result of a proportion of candidate data in the updated intermediate result in a target result, the target result being an updated intermediate result obtained after the processing is performed on all the sub-data sets; The result generating unit is configured to use the updated intermediate result as the query result, Wherein, determining the previous query index based on the updated intermediate result includes: Determine a predicted ranking of at least one candidate data in the updated intermediate result, the predicted ranking indicating a predicted result of a ranking of similarities between the corresponding candidate data and the query data in the target result, wherein the Y sub-data sets each have a corresponding non-overlapping spatial range, each of the sample data and the query data each have a corresponding spatial position, wherein each candidate data in the updated intermediate result corresponds to an intermediate ranking of the candidate data in the updated intermediate result, the intermediate ranking is determined based on the similarity between the candidate data and the query data, and the predicted ranking of the at least one candidate data is determined based on the spatial position of the query data, the spatial position of the candidate data, the spatial ranges of each of the Y sub-data sets, and the intermediate ranking of the candidate data; and The accuracy rate is determined based on the number of candidate data whose predicted ranking values ​​are not greater than the target number and the target number.

16. A data query device, used for querying a target number of sample data with the highest similarity to the query data in a data set, the device is applied to a central node in a distributed cluster, the central node is in communication connection with a plurality of running nodes in the distributed cluster, each of the plurality of running nodes stores a data set corresponding to the running node in the original data set, wherein: The device comprises: A first receiving unit is configured to receive query data and a global termination condition; a correction unit configured to correct the global termination condition to obtain a target termination condition, and send the query data and the target termination condition to each of the operating nodes, so that the operating node uses the device according to claim 15 to query the target number of sample data with the highest similarity to the query data in the data set corresponding to the operating node; A second receiving unit is configured to receive, from each of the running nodes, a target number of candidate data of a query result corresponding to the running node, wherein the query result includes a target number of sample data; and The second determining unit is configured to determine, from the query results received from each of the running nodes, a target number of sample data and a target number of target data with the highest similarity to the query data as the final query result, The global termination condition includes a global accuracy rate, the target termination condition includes a target accuracy rate, and the global accuracy rate indicates the proportion of sample data in the final query result to the target number of sample data with the highest similarity to the query data in the original data set. And / or, the global termination condition includes a global delay, and the target termination condition includes a target delay.

17. A data query system, comprising: A central node using the device of claim 15; as well as An operating node using the device described in claim 16.

18. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-14.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

20. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 14 when executed by a processor.

Images