Data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a data processing method, apparatus, electronic device, and storage medium, and relate to the field of computer technology. Through the embodiments of the present application, when storing data, an index creation method suitable for the original data can be determined based on the data type and data characteristics of the original data. Then, an index for the original data can be created and stored based on the index creation method suitable for the original data. Thus, upon receiving a data query request, the embodiments of the present application can query the target data based on the adapted indexing method, thereby improving data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer technology, the volume of various data is increasing, and the difficulty of data processing is also increasing. This is especially true for unstructured data, which is data that computers cannot directly understand, such as images, videos, and text.

[0003] Currently, traditional data processing methods cannot efficiently process unstructured data. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a data processing method, apparatus, electronic device, and storage medium to achieve efficient processing of unstructured data.

[0005] In a first aspect, a data query method is provided, the method being applied to an electronic device, the method comprising:

[0006] A data query request is received, where the data query request at least includes reference data.

[0007] Based on the reference data, a target indexing mode corresponding to the data query request is determined.

[0008] At least one target data corresponding to the data query request is determined based on the target indexing method.

[0009] In a second aspect, a data storage method is provided, the method being applied to an electronic device, the method comprising:

[0010] An index building request is received, where the index building request includes a data type and data characteristics of the original data.

[0011] The corresponding index creation method is determined according to the data type and data characteristics.

[0012] According to the index creation method, an index of the original data is constructed, and the original data is stored accordingly.

[0013] In a third aspect, a data query device is provided, which is applied to an electronic device and includes:

[0014] The first receiving module is configured to receive a data query request, where the data query request at least includes reference data.

[0015] The first determining module is configured to determine a target indexing method corresponding to the data query request based on the reference data.

[0016] A query module is used to determine at least one target data corresponding to the data query request based on the target index mode.

[0017] In a fourth aspect, a data storage device is provided, the device being applied to an electronic device, the device comprising:

[0018] The second receiving module is configured to receive an index building request, where the index building request includes the data type and data characteristics of the original data.

[0019] The second determining module is used to determine a corresponding index creation method according to the data type and data characteristics.

[0020] The storage module is used to construct an index of the original data according to the index creation method and store the original data accordingly.

[0021] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first and second aspects.

[0022] In a sixth aspect, an embodiment of the present application provides a computer storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods described in the first and second aspects are implemented.

[0023] Through the embodiments of the present application, when storing data, it is possible to determine an index creation method suitable for the original data based on the data type and data characteristics of the original data, and then create an index for the original data based on the index creation method suitable for the original data and store the original data. In this way, when receiving a data query request, the embodiments of the present application can query the target data based on the appropriate indexing method, thereby improving the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the embodiments of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0025] Figure 1 This is a flow chart of the data query method according to an embodiment of the present application;

[0026] Figure 2 This is a flowchart of determining target data based on a routing index according to an embodiment of the present application;

[0027] Figure 3 This is a flowchart of an embodiment of the present application for determining target data based on a routing index and a compression index;

[0028] Figure 4 This is a flowchart of an embodiment of the present application for determining target data based on a compressed index and a full index;

[0029] Figure 5 This is a flowchart of an embodiment of the present application for determining target data based on routing index, compression index, and full index;

[0030] Figure 6 This is a flowchart of the data storage method according to an embodiment of the present application;

[0031] Figure 7 This is a schematic diagram of the structure of the data query device according to an embodiment of the present application;

[0032] Figure 8 This is a structural diagram of a data storage device according to an embodiment of the present application;

[0033] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 10 A schematic structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The present application is described below based on the following embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0036] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0037] Unless the context clearly requires otherwise, words like “include”, “comprising” and the like in the specification should be interpreted as including rather than exclusive or exhaustive; that is, as “including but not limited to”.

[0038] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0039] In related technologies, unstructured data is generally queried using Approximate Nearest Neighbor Search (ANNS). ANNS uses the distance (i.e., similarity) between a reference data set and the data in a database to determine one or more target data sets that are most similar to the reference data set, thereby enabling unstructured data queries. Specifically, ANNS can represent unstructured data as vectors in a high-dimensional space, and then calculate the distance between vectors in the high-dimensional space, thereby enabling unstructured data queries.

[0040] In related technologies, the query methods primarily used by ANNS include clustering-based indexing, graph- or tree-based indexing, quantization-based indexing, and hash-based indexing. Graph- or tree-based indexing, which uses the exponent of the dimension as the index's spatial complexity and the time complexity of retrieval, is difficult to apply to high-dimensional vector spaces. Hash-based indexing, on the other hand, is not suitable for massive datasets because the hash table structure requires a large amount of storage space.

[0041] However, clustering indexing and quantization-based indexing cannot simultaneously meet the requirements of low query time and high query accuracy. Therefore, traditional data processing methods cannot efficiently process unstructured data.

[0042] To address the above-mentioned issues, embodiments of the present application provide a data query method that can be applied to an electronic device. The electronic device can be a terminal or a server, where the terminal can be a smartphone, a tablet computer, or a personal computer (PC), and the server can be a single server, a distributed server cluster, or a cloud server.

[0043] The data query method of the embodiment of the present application will be described in detail below in conjunction with specific implementation methods. Figure 1 The specific steps are as follows:

[0044] In step 11, a data query request is received.

[0045] The data query request at least includes reference data. Reference data is data used as a reference when performing a data query in the embodiment of the present application. For example, if the embodiment of the present application is performing an image query, the reference data may be an image A. The electronic device may then query a database based on the image A and determine a target image (target data) in the database that matches the image A.

[0046] In step 12, based on the reference data, a target indexing method corresponding to the data query request is determined.

[0047] In an embodiment of the present application, different query purposes can be achieved through multi-level indexing, wherein the target indexing method can include one or more of routing indexing, compression indexing and full indexing.

[0048] Specifically, the routing index can be used to determine the target index tag corresponding to the reference data, the compression index can be used to determine the target compressed data corresponding to the reference data in each compressed data set, and the full index can be used to determine the target full data corresponding to the reference data in each full data set.

[0049] In the embodiments of the present application, fast queries can be performed using routing indexes and compressed indexes, while accurate queries can be performed using full indexes. Therefore, fast and accurate queries can be achieved by combining multiple indexes. In other words, the multi-level index structure constructed using one or more of the above indexing methods can be used to address different application scenarios.

[0050] In step 13, at least one target data corresponding to the data query request is determined based on the target index mode.

[0051] Through the embodiments of the present application, a specific method can be used to query data in a database using a target index method to determine at least one target data. In this process, the embodiments of the present application can perform data queries based on appropriate index methods, so fast and accurate data queries can be achieved through the embodiments of the present application.

[0052] In a preferred embodiment, if the target indexing method determined based on the reference data in the embodiment of the present application is a routing index, step 13 can be executed as follows: based on a pre-set first indexing algorithm and reference data, determine at least one target index tag corresponding to the reference data, determine the target data set corresponding to each target index tag, and determine at least one target data based on the similarity between the reference data and the data in each target data set.

[0053] For example, Figure 2 As shown, Figure 2 This is a flowchart of determining target data based on routing indexes according to an embodiment of the present application. Data query request 21 includes reference data 211, and index tags include index tags 221, 222, 223, and 224, each of which corresponds to a target dataset (index tag 221 corresponds to target dataset A, index tag 222 corresponds to target dataset B, index tag 223 corresponds to target dataset C, and index tag 224 corresponds to target dataset D).

[0054] In the process of determining the target data 23, the embodiment of the present application can first determine the similarity between the reference data 211 and each index tag based on the first index algorithm, and then determine one or more index tags with a high similarity to the reference data 211 based on the similarity between the reference data 211 and each index tag. Figure 2 In the example, the index tags with higher similarity to the reference data 211 are the index tags 221 and the index tags 223 .

[0055] In a preferred embodiment, the first indexing algorithm may be a Hierarchical Navigable Small World (HNSW) algorithm, an Inverted File (IVF) algorithm, or an Inverted Multi-Index (IMI) algorithm.

[0056] Among them, HNSW is an approximate nearest neighbor search algorithm. Specifically, HNSW is a graph-based nearest neighbor search algorithm. HNSW constructs all vectors in the D-dimensional space into an interconnected graph, and searches for multiple points closest to a vertex based on this graph.

[0057] IVF is an algorithm that performs queries by clustering. Specifically, IVF can cluster different vector spaces and then perform queries on each cluster. This can reduce the amount of query computation and increase query speed.

[0058] IMI is a variant of the dot-product quantization algorithm. It divides the data dimension into two parts and uses a clustering algorithm to divide each half of the data dimension into different center points. Compared to IVF, IMI uses a large number of center points, making it more suitable for data-intensive scenarios.

[0059] After determining the index tag 221 and the index tag 223 , the embodiment of the present application may determine the target data sets (ie, the target data set A and the target data set C) corresponding to the index tag 221 and the index tag 223 , respectively.

[0060] After determining the target dataset A and the target dataset C, the embodiment of the present application can determine the target data 23 based on the similarity between the reference data 211 and the data in each target dataset, wherein: Figure 2 The target data 23 in the can represent either a single target data or a set of target data.

[0061] Through the embodiments of the present application, the index tags can be determined first, and then the target data can be further determined based on the index tags. In this process, there is no need to calculate the similarity between the reference data and the data in each data set; only the similarity between the reference data and the data in the target data set needs to be calculated, which saves computational effort and improves the efficiency of data query.

[0062] In addition, in a preferred embodiment, the index tag can be used to characterize the clustering characteristics of the corresponding data set.

[0063] Furthermore, the process of determining the target index label based on the first index algorithm and reference data in the embodiment of the present application can be performed as follows: based on the pre-set first index algorithm, determine the similarity between the reference data and the index labels of each data set, and then based on the pre-set first similarity condition, determine at least one target index label whose similarity meets the first similarity condition.

[0064] The first similarity condition may be that the similarity between the reference data and the index tags of each dataset is greater than a predetermined threshold, where the threshold may be a value set based on actual circumstances. When the similarity between the reference data and the index tags of one or more datasets is greater than the predetermined threshold, embodiments of the present application may determine the index tag with a similarity greater than the predetermined threshold as the target index tag.

[0065] In another preferred embodiment, if the target indexing method determined based on the reference data in the embodiment of the present application is routing index and compression index, then step 13 can be executed as follows: based on a pre-set first indexing algorithm and reference data, determine at least one target index tag corresponding to the reference data, and then based on a pre-set second indexing algorithm and reference data, determine at least one target compressed data in the compressed data set corresponding to each target index tag, and then determine the target data based on the target compressed data.

[0066] Among them, each index label corresponds to a compressed data set.

[0067] For example, Figure 3 As shown, Figure 3 This is a flowchart of determining target data based on routing indexes and compression indexes according to an embodiment of the present application. Data query request 31 includes reference data 311, and index tags include index tags 321, 322, 323, and 324, each of which corresponds to a compressed dataset (index tag 321 corresponds to compressed dataset E, index tag 322 corresponds to compressed dataset F, index tag 323 corresponds to compressed dataset G, and index tag 324 corresponds to compressed dataset H).

[0068] In the process of determining the target data 34, the embodiment of the present application can first determine the similarity between the reference data 311 and each index tag based on the first index algorithm, and then determine one or more index tags with a high similarity to the reference data 311 based on the similarity between the reference data 311 and each index tag. Figure 3 In the example, the index tags with higher similarity to the reference data 311 are the index tags 321 and 323 .

[0069] After determining the index tag 321 and the index tag 323 , the embodiment of the present application may determine the compressed data sets (ie, the compressed data set E and the compressed data set G) corresponding to the index tag 321 and the index tag 323 , respectively.

[0070] Then, the embodiment of the present application can determine the similarity between the reference data 311 and each compressed data in the compressed data set E and the compressed data set G based on the second index algorithm, and then determine the target compressed data 33 with a higher similarity to the reference data 311 based on the similarity between the reference data 311 and each compressed data in the compressed data set E and the compressed data set G, wherein the target compressed data 33 can be used to represent a single target compressed data, or can be used to represent a set of target compressed data.

[0071] In a preferred embodiment, the second index algorithm may be a scalar quantization (SQ) algorithm or a product quantization (PQ) algorithm.

[0072] SQ and PQ can use quantized data in compressed indexes, which not only effectively reduces memory requirements but also speeds up retrieval. Compressed indexes using quantized index structures can store all quantized data and index structures in memory.

[0073] After determining the target compressed data 33 , the embodiment of the present application may determine the target data 34 based on the target compressed data 33 .

[0074] Among them, the embodiment of the present application can first determine the full data corresponding to the target compressed data 33 based on the correspondence between the compressed data and the full data, and then determine the full data with a higher similarity to the reference data 311 as the target data 34 based on the similarity between the reference data 311 and the full data corresponding to the target compressed data 33.

[0075] Through the embodiments of the present application, the index tag can be determined first, and then the target compressed data can be further determined based on the index tag, and then the target data can be further determined based on the compressed data (for example, the full amount of data corresponding to the compressed data can be determined by the compressed data). In this process, there is no need to calculate the similarity between the reference data and the data in each data set. Moreover, since the compression index is indexed for the quantized compressed data, the routing index and the compression index can be used to quickly retrieve data in the memory of the electronic device, saving computing power and improving the efficiency of data query.

[0076] In addition, the process of determining the target compressed data based on the second index algorithm and reference data in the embodiment of the present application can be executed as follows: determining the compressed data set corresponding to each target index tag, and then based on the pre-set second index algorithm, determining the similarity between the reference data and the compressed data in each compressed data set, and then based on the pre-set second similarity condition, determining at least one target compressed data whose similarity meets the second similarity condition.

[0077] The second similarity condition may be that the similarity between the reference data and the compressed data in each compressed data set is greater than a predetermined threshold, where the threshold may be a value set based on actual circumstances. When the similarity between the reference data and one or more compressed data is greater than the predetermined threshold, embodiments of the present application may determine the compressed data having a similarity greater than the predetermined threshold as the target compressed data.

[0078] In another preferred embodiment, if the target indexing method determined based on the reference data in the embodiment of the present application is a compressed index and a full index, step 13 can be executed as follows: based on a pre-set second indexing algorithm and reference data, at least one target compressed data is determined in the compressed data set, and then based on a pre-set third indexing algorithm and reference data, at least one target data is determined in the full data corresponding to each target compressed data.

[0079] Among them, each compressed data corresponds to a full amount of data.

[0080] For example, Figure 4 As shown, Figure 4 This is a flow chart of determining target data based on a compressed index and a full index according to an embodiment of the present application, wherein a data query request 41 includes reference data 411 .

[0081] In the process of determining the target data 44, the embodiment of the present application can first determine the similarity between the reference data 411 and the compressed data in each compressed data set (compressed data set E, compressed data set F, compressed data set G, and compressed data set H) based on the second indexing algorithm, and then determine the target compressed data 42 with a high similarity to the reference data 411 based on the similarity between the reference data 411 and the compressed data in each compressed data set. The target compressed data 42 can be used to represent a single target compressed data or a set of target compressed data.

[0082] After determining the target compressed data 42 , the embodiment of the present application may determine the target full data 43 corresponding to the target compressed data 42 based on the correspondence between the compressed data and the full data.

[0083] After determining the target full data 43, the embodiment of the present application can determine the similarity between the reference data 411 and the target full data 43 based on the third index algorithm, and then determine the target full data 43 with a higher similarity to the reference data 411 as the target data 44 based on the similarity between the reference data 411 and the target full data 43.

[0084] In a preferred embodiment, the third indexing algorithm may be a full-quantity precise search algorithm. For example, the third indexing algorithm may be an algorithm for calculating the distance between full-quantity vectors, and the calculated distance is the data similarity.

[0085] Through the embodiments of the present application, it is possible to first determine target compressed data that is similar to the reference data, and then determine the target data based on the full data corresponding to the target compressed data. In this process, there is no need to calculate the similarity between the reference data and the data in each data set. Moreover, since the compressed index is indexed for the quantized compressed data, the use of compressed index and full index can save computational effort while ensuring accuracy, thereby improving the efficiency of data query.

[0086] In addition, in a preferred embodiment, the compressed data set may further include a corresponding code book, wherein the code book may be used to record the corresponding relationship between each compressed data in the compressed data set and each full data in the storage medium.

[0087] Furthermore, the process of determining the target data based on the third index algorithm and the reference data in the embodiment of the present application can be executed as follows: based on the code book corresponding to each compressed data set, the full data corresponding to each target compressed data is determined, and then based on the pre-set third index algorithm, the similarity between the reference data and the full data corresponding to each target compressed data is determined, and then based on the pre-set third similarity condition, at least one target data whose similarity meets the third similarity condition is determined.

[0088] The third similarity condition may be that the similarity between the reference data and the full data corresponding to each target compressed data is greater than a predetermined threshold, where the threshold may be a value set based on actual circumstances. When the similarity between the reference data and one or more full data is greater than the predetermined threshold, embodiments of the present application may determine the full data with similarity greater than the predetermined threshold as the target data.

[0089] In another preferred embodiment, if the target indexing method determined based on the reference data in the embodiment of the present application is routing index, compression index and full index, then step 13 can be executed as follows: based on the pre-set first indexing algorithm and reference data, determine at least one target index tag corresponding to the reference data, and then based on the pre-set second indexing algorithm and reference data, determine at least one target compressed data in the compressed data set corresponding to each target index tag, and then based on the pre-set third indexing algorithm and reference data, determine at least one target data in the full data corresponding to each target compressed data.

[0090] Among them, each index label corresponds to a compressed data set, and each compressed data corresponds to a full data set.

[0091] For example, Figure 5 As shown, Figure 5 This is a flowchart of determining target data based on routing indexes, compression indexes, and full indexes in an embodiment of the present application. Data query request 51 includes reference data 511, and index tags include index tags 521, 522, 523, and 524, each of which corresponds to a compressed dataset (index tag 521 corresponds to compressed dataset E, index tag 522 corresponds to compressed dataset F, index tag 523 corresponds to compressed dataset G, and index tag 524 corresponds to compressed dataset H).

[0092] In the process of determining the target data 54, the embodiment of the present application can first determine the similarity between the reference data 511 and each index tag based on the first index algorithm, and then determine one or more index tags with a high similarity to the reference data 511 based on the similarity between the reference data 511 and each index tag. Figure 5 In the example, the index tags 521 and 523 have the highest similarity to the reference data 511 .

[0093] After determining index tag 521 and index tag 523 , the embodiment of the present application may determine the compressed data sets (ie, compressed data set E and compressed data set G) corresponding to index tag 521 and index tag 523 , respectively.

[0094] Then, the embodiment of the present application can determine the similarity between the reference data 511 and each compressed data in the compressed data set E and the compressed data set G based on the second index algorithm, and then determine the target compressed data 53 with a higher similarity to the reference data 511 based on the similarity between the reference data 511 and each compressed data in the compressed data set E and the compressed data set G, wherein the target compressed data 53 can be used to represent a single target compressed data, or can be used to represent a set of target compressed data.

[0095] After determining the target compressed data 53 , the embodiment of the present application may determine the target full data 54 corresponding to the target compressed data 53 based on the correspondence between the compressed data and the full data.

[0096] After determining the target full data 54, embodiments of the present application can determine the similarity between the reference data 511 and the target full data 54 based on the third indexing algorithm, and then, based on the similarity between the reference data 511 and the target full data 54, determine the target full data 54 having a higher similarity to the reference data 511 as the target data 55. The target data 55 can be used to represent a single target data or a set of target data.

[0097] Through the embodiments of the present application, the index tag can be determined first, and then the target compressed data can be determined based on the index tag, and then the target data can be determined based on the full data corresponding to the target compressed data. In this process, there is no need to calculate the similarity between the reference data and the data in each data set, which can save computing power while ensuring accuracy and improve the efficiency of data query.

[0098] The above embodiment explains the embodiment of the present application from the perspective of data query. The following will explain the embodiment of the present application from the perspective of data storage in combination with specific implementation methods. Specifically, the embodiment of the present application provides a data storage method, such as Figure 6 The specific steps are as follows:

[0099] In step 61, an index building request is received.

[0100] The index building request includes the data type and data characteristics of the original data.

[0101] In embodiments of the present application, a data type can be used to characterize the file format of raw data. For example, the raw data can be image data, video data, or text data. Data characteristics can be used to characterize the data distribution, data sparsity, and dimensionality of the raw data.

[0102] In step 62, a corresponding index creation method is determined according to the data type and data characteristics.

[0103] In the embodiment of the present application, original data of different data types and data characteristics may correspond to different optimal index creation methods. Therefore, determining the optimal index creation method can effectively improve the efficiency of subsequent data queries.

[0104] For example, for raw data with large data volume and high data density, you can create Figure 3 or Figure 5 For example, for raw data with large amount of data and relatively sparse data, you can create the following query structure: Figure 4 The query structure shown.

[0105] In step 63, an index of the original data is constructed according to the index creation method, and the original data is stored accordingly.

[0106] After determining the index creation method, the embodiment of the present application can construct an index of the original data based on the determined index creation method and store the original data accordingly. After storing the original data, the embodiment of the present application can realize efficient data query based on the data query request and the created index method.

[0107] In a preferred embodiment, if the index creation method includes routing index creation and compression index creation, step 63 can be executed as follows: in response to the index creation method including routing index creation and compression index creation, the original data is clustered, at least one data subset is obtained, a routing index for each data subset is created, and then each data subset is quantitatively compressed to create a compression index for the data in each compressed data subset.

[0108] In the embodiment of the present application, if a routing index and a compression index are created for the original data, then after the electronic device receives a subsequent data query request, the electronic device can perform routing index and compression index according to the reference data in the data query request, and then obtain the target data. Specifically, the process of performing data query based on routing index and compression index can refer to the above Figure 3 The above contents will not be described in detail in the embodiments of this application.

[0109] In another preferred embodiment, if the index creation method includes compressed index creation and full index creation, step 63 can be executed as follows: in response to the index creation method including compressed index creation and full index creation, the original data is quantitatively compressed, a compressed index of the compressed data is created, and then a full index is created based on the correspondence between the compressed data and the original data.

[0110] In the embodiment of the present application, if a compressed index and a full index are created for the original data, then after the electronic device receives a subsequent data query request, the electronic device can perform a compressed index and a full index based on the reference data in the data query request, and then obtain the target data. Specifically, the process of performing data query based on the compressed index and the full index can refer to the above Figure 4 The above contents will not be described in detail in the embodiments of this application.

[0111] In another preferred embodiment, if the index creation method includes routing index creation, compression index creation and full index creation, step 63 can be executed as follows: in response to the index creation method including routing index creation, compression index creation and full index creation, the original data is clustered, at least one data subset is obtained, a routing index for each data subset is created, and then each data subset is quantitatively compressed, a compression index of the compressed data in each data subset is created, and then a full index is created based on the correspondence between the compressed data and the original data.

[0112] In the embodiment of the present application, if a routing index, a compression index, and a full index are created for the original data, then after the electronic device receives a subsequent data query request, the electronic device can perform routing indexing, compression indexing, and full indexing based on the reference data in the data query request, and then obtain the target data. Specifically, the process of performing data query based on routing index, compression index, and full index can refer to the above Figure 5 The above contents will not be described in detail in the embodiments of this application.

[0113] Furthermore, the process of creating the routing index may be specifically performed as follows: based on the original data, a first indexing algorithm corresponding to the routing index is trained to determine n data subsets corresponding to the original data and an index label of each data subset.

[0114] Where n is a natural number greater than or equal to 1, and the index label is used to represent the clustering characteristics of the corresponding data set.

[0115] In an embodiment of the present application, each data subset can be clustered using a trained first indexing algorithm, and an index tag corresponding to each data subset can be determined. The index tag can be used as a routing index during data query.

[0116] After determining the index labels of each data subset in the embodiment of the present application, a compressed index can be further created based on the index labels. Specifically, the process can be performed as follows: based on the index labels of each data subset, the second index algorithm corresponding to the compressed index is trained, the mapping lookup table corresponding to the n data subsets is determined, and then the data in each data subset is quantitatively compressed to determine n compressed data sets.

[0117] After creating the compressed index, the electronic device can determine the compressed data corresponding to the reference data in the data query request in each compressed data set based on the trained second index algorithm and the mapping lookup table.

[0118] Furthermore, the process of training the second index algorithm and determining the mapping lookup table can be performed as follows: determining the residual vector or original vector between each index label and the data in the corresponding data subset, and then training the second index algorithm corresponding to the compressed index based on each residual vector or each original vector, and then disassembling each residual vector or each original vector to determine the mapping lookup table corresponding to the n compressed data sets.

[0119] In the process of disassembling the residual vector, the embodiment of the present application can disassemble a single residual vector into m groups of d / m-dimensional sub-vectors, and then generate a lookup table mapping the m groups of d / m-dimensional sub-vectors to n-bit binary strings based on the m groups of d / m-dimensional sub-vectors.

[0120] After determining the mapping lookup table corresponding to the n data subsets, the embodiment of the present application can further create a full index based on the mapping lookup table corresponding to the n data subsets. Specifically, the process can be performed as follows: determine the codebook corresponding to the n compressed data sets based on the mapping lookup table, and then store each original data to a storage medium based on each compressed data set.

[0121] The codebook is used to record the full data corresponding to each compressed data in the compressed data set. During subsequent queries, after the electronic device determines the target compressed data corresponding to the data query request, it can determine the full data corresponding to the target compressed data based on the codebook generated when creating the full index. Furthermore, the target data can be found within the full data corresponding to the target compressed data, saving computing power and improving data query efficiency.

[0122] In addition, the process of determining the codebook may be specifically performed as follows: encoding each residual vector according to a mapping lookup table, and determining the codebook corresponding to the n compressed data sets.

[0123] For example, the electronic device can divide the d-dimensional residual vector into m groups of sub-vectors for combination representation, and then encode each group of sub-vectors into n-bit data according to the mapping lookup table. Then, the original d-dimensional residual vector can be represented as an m*n-bit codebook.

[0124] In a preferred embodiment, the electronic device can also perform batch insertion operations on data. Specifically, the process can be executed as follows: receiving a batch insertion request, the batch insertion request includes at least one data to be processed, then deleting the constructed index, and then constructing an index based on each data to be processed and the original data.

[0125] In practical applications, if a large amount of data needs to be inserted, creating an index for each data to be processed will reduce the efficiency of data processing. Therefore, the embodiment of the present application can first delete the original index and then create an index for all data (data to be processed and original data) in a unified manner, thereby improving the efficiency of data processing.

[0126] In another preferred embodiment, the electronic device can also perform an append-insert operation on the data. Specifically, the process can be executed as follows: receiving an append-insert request, then performing a query operation based on at least one data to be processed, determining at least one compressed data set to be updated, then compressing and inserting the at least one data to be processed into the corresponding compressed data set to be updated, then determining the code book corresponding to the updated compressed data set, and then storing the data to be processed to a storage medium.

[0127] In an embodiment of the present application, the append-in is applicable to the insertion of data to be processed with a smaller amount of data, and the append-in request includes at least one data to be processed.

[0128] In addition, if the data volume of the compressed data set after the data is inserted is too large, the following can also be executed: in response to the data volume of the updated compressed data set being greater than the predetermined data volume upper limit threshold, the updated compressed data set is split into at least 2 compressed data sets, and then the index labels corresponding to each of the split compressed data sets are determined.

[0129] Among them, the predetermined data volume upper limit threshold can be a reasonable value set according to actual conditions. By splitting the compressed data set, it can be ensured that each compressed data set has a relatively appropriate data volume size, so that data processing does not require excessive computing power, thereby improving data processing efficiency.

[0130] In another preferred embodiment, the electronic device can also perform a deletion operation on the data. Specifically, the process can be executed as follows: receiving a data deletion request, then performing a query operation based on the data identifier of the data to be deleted, determining each data to be deleted, then deleting each data to be deleted and the compressed data corresponding to each data to be deleted, and then updating the code book and index tag corresponding to each data to be deleted.

[0131] The data deletion request at least includes the data identifier of the data to be deleted.

[0132] In addition, if the data volume of the compressed data set after the data is deleted is too small, the following can also be executed: in response to the data volume of the compressed data set after the compressed data is deleted being less than the predetermined data volume lower limit threshold, the compressed data set after the compressed data is deleted is merged with other compressed data sets, the merged data set is determined, and then the index information corresponding to the compressed data set after the compressed data is deleted is deleted, and then the index information corresponding to the merged data set is updated.

[0133] The data volume of the merged dataset is less than the predetermined upper data volume threshold, and the predetermined lower data volume threshold can be a reasonable value set based on actual conditions. By merging the compressed datasets, each compressed dataset can be ensured to have a relatively appropriate data volume, without occupying excess storage paths, thereby saving storage space.

[0134] Based on the same technical concept, the embodiment of the present application also provides a data query device, such as Figure 7 As shown, the device includes: a first receiving module 71, a first determining module 72 and a query module 73.

[0135] The first receiving module 71 is configured to receive a data query request, where the data query request at least includes reference data.

[0136] A first determining module 72 is configured to determine a target indexing method corresponding to the data query request based on reference data; and

[0137] The query module 73 is configured to determine at least one target data corresponding to the data query request based on the target index mode.

[0138] Through the embodiments of the present application, a specific method can be used to query the data in the database through the target index method to determine at least one target data. In this process, the embodiments of the present application can query unstructured data based on the appropriate index method, so the embodiments of the present application can efficiently process unstructured data.

[0139] Based on the same technical concept, the embodiment of the present application also provides a data storage device, such as Figure 8 As shown, the device includes: a second receiving module 81, a second determining module 82 and a storage module 83.

[0140] The second receiving module 81 is configured to receive an index building request, where the index building request includes the data type and data characteristics of the original data.

[0141] The second determining module 82 is configured to determine a corresponding index creation method according to the data type and data characteristics.

[0142] The storage module 83 is used to construct an index of the original data according to the index creation method and store the original data accordingly.

[0143] The embodiment of the present application can construct an index of the original data based on the determined index creation method and store the original data accordingly. After storing the original data, the embodiment of the present application can realize efficient data query according to the data query request and the created index method.

[0144] Figure 9 Schematic diagram of an electronic device according to an embodiment of the present application. Figure 9 As shown, Figure 9 The electronic device shown is a general address query device, which includes a general computer hardware structure, which includes at least a processor 91 and a memory 92. The processor 91 and the memory 92 are connected via a bus 93. The memory 92 is suitable for storing instructions or programs executable by the processor 91. The processor 91 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 91 executes the instructions stored in the memory 92, thereby executing the method flow of the embodiment of the present application as described above to realize data processing and control of other devices. The bus 93 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 94 and the display device and the input / output (I / O) device 95. The input / output (I / O) device 95 can be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer and other devices known in the art. Typically, the input / output device 95 is connected to the system via an input / output (I / O) controller 96.

[0145] In a preferred embodiment, the electronic device may further include a storage medium and a coprocessor, such as Figure 10 As shown, Figure 10 This is a structural diagram of another electronic device according to an embodiment of the present application, which includes: a processor 101, a memory 102, a bus 103, a display controller 104, an input / output (I / O) device 105, an input / output (I / O) controller 106, a storage medium 107 and a coprocessor 108.

[0146] In an embodiment of the present application, the coprocessor 108 can be configured to determine the target index tag corresponding to the reference data (i.e., configured to perform routing index related steps), and the storage medium 107 can be configured to store the full amount of data. Among them, the coprocessor 108 can be a graphics processing unit (GPU), a programmable gate array, or a customized chip. The storage medium 107 can be a mechanical hard disk (HDD), a solid state drive (SSD), a flash memory (Flash), or a new type of non-volatile memory device, such as storage class memory (SCM).

[0147] It will be understood by those skilled in the art that the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present application is described with reference to flowcharts of methods, apparatuses (devices), and computer program products according to embodiments of the present application. It should be understood that each process in the flowcharts can be implemented by computer program instructions.

[0149] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0150] These computer program instructions can also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0151] Another embodiment of the present application relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used to enable a computer to execute part or all of the above method embodiments.

[0152] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by specifying relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0153] The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application are intended to be within the scope of protection of the present application.

Claims

1. A data query method, characterized in that: The method comprises: receiving a data query request, wherein the data query request includes at least reference data; Determining a target indexing method corresponding to the data query request based on the reference data, the target indexing method including one or more of a routing index, a compression index, and a full index, the routing index being used to determine a target index tag corresponding to the reference data, the compression index being used to determine target compressed data corresponding to the reference data in each compressed data set, and the full index being used to determine target full data corresponding to the reference data in each full data set; and Determining at least one target data corresponding to the data query request based on the target index mode; In response to the reference data, determining that a target indexing mode corresponding to the data query request is a routing index, a compression index, and a full index; The determining, based on the target indexing method, at least one target data corresponding to the data query request includes: Determine, based on a preset first indexing algorithm, a first similarity between the reference data in the data query request and each index tag; Determining, based on the first similarity and the reference data, at least one target index tag corresponding to the reference data, where each index tag corresponds to a compressed data set; determining a second similarity between the reference data and each compressed data in the compressed data set based on a preset second indexing algorithm; Determining at least one target compressed data in the compressed data sets corresponding to each target index tag based on the second similarity and the reference data, where each compressed data corresponds to a full data set; Determine the full amount of data corresponding to each target compressed data set based on the codebook corresponding to each compressed data set; Determine the target full data corresponding to the target compressed data based on the corresponding relationship between the compressed data and the full data; Determining a third similarity between the reference data and the target full data based on a preset third indexing algorithm; At least one target data is determined in the full data corresponding to each target compressed data according to the third similarity and the reference data.

2. The method according to claim 1, characterized in that The determining, based on the target indexing method, at least one target data corresponding to the data query request includes: Determining at least one target index tag corresponding to the reference data based on a preset first indexing algorithm and the reference data; Determine the target data set corresponding to each target index label; and At least one target data is determined based on the similarity between the reference data and the data in each target data set.

3. The method according to claim 1, characterized in that The determining, based on the target indexing method, at least one target data corresponding to the data query request includes: Determine, based on a preset first indexing algorithm and the reference data, at least one target index tag corresponding to the reference data, each index tag corresponding to a compressed data set; Determining at least one target compressed data in the compressed data set corresponding to each target index tag based on a preset second indexing algorithm and the reference data; and The target data is determined according to the target compression data.

4. The method according to claim 1, wherein The determining, based on the target indexing method, at least one target data corresponding to the data query request includes: Based on a preset second indexing algorithm and the reference data, determining at least one target compressed data in the compressed data set, each compressed data corresponding to a full data set; and Based on a preset third index algorithm and the reference data, at least one target data is determined in the full amount of data corresponding to each target compressed data.

5. The method according to any one of claims 2 and 3, characterized in that The index labels are used to characterize the clustering characteristics of the corresponding data set; The determining, based on a preset first indexing algorithm and the reference data, at least one target index tag corresponding to the reference data includes: Determining the similarity between the reference data and the index labels of each data set based on a preset first indexing algorithm; as well as Based on a preset first similarity condition, at least one target index tag whose similarity satisfies the first similarity condition is determined.

6. The method according to any one of claims 3 to 4, characterized in that The determining, based on the preset second indexing algorithm and the reference data, at least one target compressed data in the compressed data set corresponding to each target index tag includes: Determine the compressed data set corresponding to each target index label; Determining the similarity between the reference data and the compressed data in each compressed data set based on a preset second indexing algorithm; and Based on a preset second similarity condition, at least one target compressed data whose similarity satisfies the second similarity condition is determined.

7. The method according to claim 4, characterized in that The compressed data set further includes a corresponding codebook, wherein the codebook is used to record the corresponding relationship between each compressed data in the compressed data set and each full data in the storage medium; The determining, based on the preset third index algorithm and the reference data, at least one target data in the full amount of data corresponding to each target compressed data includes: Determine the full amount of data corresponding to each target compressed data set based on the codebook corresponding to each compressed data set; Determining, based on a preset third indexing algorithm, the similarity between the reference data and the full data corresponding to each target compressed data; and Based on a preset third similarity condition, at least one target data whose similarity satisfies the third similarity condition is determined.

8. The method according to claim 1, characterized in that The first indexing algorithm is a layered navigation small-world algorithm, an inverted file algorithm, or an inverted multi-index algorithm; the second indexing algorithm is a scalar quantization algorithm or a product quantization algorithm; and the third indexing algorithm is a full-quantity precise search algorithm.

9. A data storage method, characterized in that: The method comprises: receiving an index building request, wherein the index building request includes a data type and data characteristics of the original data; Determine a corresponding index creation method according to the data type and data characteristics, the index creation method including one or more of routing index creation, compression index creation, and full index creation, the routing index creation is used to train a first index algorithm based on the original data, determine a data subset and an index label, the compression index creation is used to train a second index algorithm based on the index label, determine a mapping lookup table and quantize the compressed data, and the full index creation is used to determine a codebook based on the mapping lookup table and store the original data; and Building an index of the original data according to the index creation method, and storing the original data accordingly; In response to receiving a data query request including reference data, and based on the reference data, determining that a target indexing mode corresponding to the data query request is a routing index, a compression index, or a full index; Determining at least one target data corresponding to the data query request based on the target index mode includes: Determine, based on a preset first indexing algorithm, a first similarity between the reference data in the data query request and each index tag; Determining, based on the first similarity and the reference data, at least one target index tag corresponding to the reference data, where each index tag corresponds to a compressed data set; determining a second similarity between the reference data and each compressed data in the compressed data set based on a preset second indexing algorithm; Determining at least one target compressed data in the compressed data sets corresponding to each target index tag based on the second similarity and the reference data, where each compressed data corresponds to a full data set; Determine the target full data corresponding to the target compressed data based on the corresponding relationship between the compressed data and the full data; Determine the full amount of data corresponding to each target compressed data set based on the codebook corresponding to each compressed data set; Determining a third similarity between the reference data and the target full data based on a preset third indexing algorithm; At least one target data is determined in the full data corresponding to each target compressed data according to the third similarity and the reference data.

10. The method according to claim 9, characterized in that The step of constructing an index of the original data according to the index creation method includes: In response to the index creation method including routing index creation and compression index creation, clustering the original data, obtaining at least one data subset, and creating a routing index for each of the data subsets; and Each of the data subsets is quantized and compressed, and a compression index of the data in each compressed data subset is created.

11. The method according to claim 9, characterized in that The step of constructing an index of the original data according to the index creation method includes: In response to the index creation method including compressed index creation and full index creation, performing quantitative compression on the original data to create a compressed index of the compressed data; and A full index is created based on the correspondence between the compressed data and the original data.

12. The method according to claim 9, characterized in that The step of constructing an index of the original data according to the index creation method includes: In response to the index creation mode including routing index creation, compression index creation, and full index creation, clustering the original data, obtaining at least one data subset, and creating a routing index for each of the data subsets; Performing quantitative compression on each of the data subsets, and creating a compression index of the data in each compressed data subset; and A full index is created based on the correspondence between the compressed data and the original data.

13. The method according to claim 12, characterized in that The clustering of the original data to obtain at least one data subset and creating a routing index for each data subset includes: Based on the original data, the first indexing algorithm corresponding to the routing index is trained to determine n data subsets corresponding to the original data and the index label of each data subset, where n is a natural number greater than or equal to 1, and the index label is used to characterize the clustering characteristics of the corresponding data set.

14. The method according to claim 13, characterized in that The quantized compression of each data subset to create a compressed index of the data in each compressed data subset includes: Based on the index labels of each data subset, a second indexing algorithm corresponding to the compressed index is trained to determine a mapping lookup table corresponding to the n data subsets; and The data in each data subset is quantitatively compressed to determine n compressed data sets.

15. The method according to claim 14, characterized in that The second indexing algorithm corresponding to the compressed index is trained based on the index labels of each data subset to determine the mapping lookup table corresponding to the n data subsets, including: Determine the residual vector or original vector between each index label and the data in the corresponding data subset; Training a second indexing algorithm corresponding to the compressed index based on each residual vector or each original vector; and Each residual vector or each original vector is decomposed to determine a mapping lookup table corresponding to the n compressed data sets.

16. The method according to claim 14, characterized in that The step of creating a full index based on the correspondence between the compressed data and the original data includes: Determining a codebook corresponding to n compressed data sets based on the mapping lookup table, wherein the codebook is used to record the full amount of data corresponding to each compressed data in the compressed data set; and Based on each compressed data set, each original data is stored in a storage medium.

17. The method according to claim 16, characterized in that The determining, based on the mapping lookup table, codebooks corresponding to the n compressed data sets includes: Each residual vector is encoded according to the mapping lookup table to determine a codebook corresponding to the n compressed data sets.

18. The method according to claim 9, characterized in that The method further comprises: receiving a batch insert request, wherein the batch insert request includes at least one data to be processed; Delete the built index; and An index is constructed based on the data to be processed and the original data.

19. The method according to claim 16, wherein The method further comprises: receiving an additional insert request, wherein the additional insert request includes at least one data to be processed; Performing a query operation based on the at least one data to be processed to determine at least one compressed data set to be updated; compressing the at least one to-be-processed data and inserting the compressed data set corresponding to the to-be-updated data set; Determining a codebook corresponding to the updated compressed data set; and The data to be processed is stored in a storage medium.

20. The method according to claim 19, characterized in that The method further comprises: In response to the data size of the updated compressed data set being greater than a predetermined data size upper threshold, splitting the updated compressed data set into at least two compressed data sets; and Determine the index labels corresponding to each compressed data set after splitting.

21. The method according to claim 16, wherein The method further comprises: receiving a data deletion request, wherein the data deletion request includes at least a data identifier of the data to be deleted; Perform a query operation based on the data identifier of the data to be deleted to determine each data to be deleted; Deleting each piece of data to be deleted and the compressed data corresponding to each piece of data to be deleted; and Update the codebook and index tag corresponding to each data to be deleted.

22. The method according to claim 21, characterized in that The method further comprises: In response to the data volume of the compressed data set after the compressed data is deleted being less than a predetermined data volume lower limit threshold, merging the compressed data set after the compressed data is deleted with other compressed data sets to determine a merged data set, wherein the data volume of the merged data set is less than a predetermined data volume upper limit threshold; Deleting the index information corresponding to the compressed data set after deleting the compressed data; and Update the index information corresponding to the merged data set.

23. A data query device, characterized in that: The device comprises: A first receiving module is configured to receive a data query request, wherein the data query request includes at least reference data; a first determination module, configured to determine, based on the reference data, a target indexing mode corresponding to the data query request, the target indexing mode including one or more of a routing index, a compression index, and a full index; the routing index being used to determine a target index tag corresponding to the reference data; the compression index being used to determine target compressed data corresponding to the reference data in each compressed data set; and the full index being used to determine target full data corresponding to the reference data in each full data set; and A query module, configured to determine at least one target data corresponding to the data query request based on the target index mode; In response to the reference data, determining that a target indexing mode corresponding to the data query request is a routing index, a compression index, and a full index; The query module is also used to: Determine, based on a preset first indexing algorithm, a first similarity between the reference data in the data query request and each index tag; Determining, based on the first similarity and the reference data, at least one target index tag corresponding to the reference data, where each index tag corresponds to a compressed data set; determining a second similarity between the reference data and each compressed data in the compressed data set based on a preset second indexing algorithm; Determining at least one target compressed data in the compressed data sets corresponding to each target index tag based on the second similarity and the reference data, where each compressed data corresponds to a full data set; Determine the full amount of data corresponding to each target compressed data set based on the codebook corresponding to each compressed data set; Determine the target full data corresponding to the target compressed data based on the corresponding relationship between the compressed data and the full data; Determining a third similarity between the reference data and the target full data based on a preset third indexing algorithm; At least one target data is determined in the full data corresponding to each target compressed data according to the third similarity and the reference data.

24. A data storage device, characterized in that The device comprises: A second receiving module is configured to receive an index building request, wherein the index building request includes a data type and data characteristics of the original data; a second determination module, configured to determine a corresponding index creation mode according to the data type and data characteristics, the index creation mode including one or more of routing index creation, compression index creation, and full index creation, the routing index creation being used to train a first indexing algorithm based on the original data and determine a data subset and an index label, the compression index creation being used to train a second indexing algorithm based on the index label, determine a mapping lookup table, and quantize the compressed data, and the full index creation being used to determine a codebook based on the mapping lookup table and store the original data; and A storage module, configured to construct an index of the original data according to the index creation method, and store the original data accordingly; In response to receiving a data query request including reference data, and based on the reference data, determining that a target indexing mode corresponding to the data query request is a routing index, a compression index, or a full index; The storage module is also used for: Determine, based on a preset first indexing algorithm, a first similarity between the reference data in the data query request and each index tag; Determining, based on the first similarity and the reference data, at least one target index tag corresponding to the reference data, where each index tag corresponds to a compressed data set; determining a second similarity between the reference data and each compressed data in the compressed data set based on a preset second indexing algorithm; Determining at least one target compressed data in the compressed data sets corresponding to each target index tag based on the second similarity and the reference data, where each compressed data corresponds to a full data set; Determine the target full data corresponding to the target compressed data based on the corresponding relationship between the compressed data and the full data; Determine the full amount of data corresponding to each target compressed data set based on the codebook corresponding to each compressed data set; Determining a third similarity between the reference data and the target full data based on a preset third indexing algorithm; At least one target data is determined in the full data corresponding to each target compressed data according to the third similarity and the reference data.

25. An electronic device comprising a memory and a processor, characterized in that: The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 22.

26. The electronic device according to claim 25, characterized in that The electronic device further includes a storage medium and a coprocessor; The storage medium is configured to store the full amount of data; The coprocessor is configured to determine a target index tag corresponding to reference data, where the reference data is at least data included in the data query request.

27. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 22 is implemented.

Images