Data processing method and related device

Abstract

The present application discloses a data processing method and a related device, for use in solving the problem of low index efficiency of an index structure. The method comprises: acquiring an operation request, wherein the operation request comprises an operation instruction and data to be operated; performing product quantization coding on a target residual value or said data to obtain a target key, wherein the target residual value is a residual value between said data and a center vector of a target data group, and the target data group is a data group to which said data belongs among K data groups; and executing the operation instruction on the basis of the target key and a target learning type index corresponding to the target data group.

Description

A data processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 4, 2023, with application number 202310983845.6 and invention name “A data processing method”, and priority to the Chinese patent application filed with the State Intellectual Property Office on August 25, 2023, with application number 202311086568.5 and invention name “A data processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data processing method and related equipment. Background Art

[0003] In the era of big data and cloud computing, how to quickly query and update data in large-scale databases is one of the main indicators for measuring the performance of large-scale databases.

[0004] With the development and widespread application of internet technologies, data volumes have exploded, making it difficult for existing structured data database technologies to maintain sufficient scale. Beyond the data objects themselves, their structured data representations become a new form of metadata storage, placing significant storage overhead and computing pressure on storage systems. Structured data databases primarily need to address two issues: efficient querying and efficient analysis and storage.

[0005] The product quantization (PQ) algorithm is a query algorithm proposed to improve retrieval speed. PQ uses a partitioning approach, first dividing long structured data into m subspaces, and then using the kmeans clustering method in each subspace to group similar structured data into the same category. The structured data within each center point is organized and stored using an inverted index. When querying, first find the center point corresponding to the query structured data, then find all neighboring elements in the inverted index, and then find the TopK approximate elements using a distance calculation function. The design of the inverted index leads to low index size and index search efficiency when managing big data (suitable for TopK queries of approximate images, with a single query taking approximately 100us).

[0006] Summary of the Invention

[0007] This application provides a data processing method to solve the problem of low efficiency of current index structure query.

[0008] In the first aspect, the present application provides a data processing method. The method can be applied to computing devices such as computers, servers, mobile phones, smart network cards, and programmable switches. The method comprises: obtaining an operation request, the operation request including an operation instruction and data to be operated; performing product quantization encoding on a target residual value or data to be operated to obtain a target key, the target residual value being the residual value between the data to be operated and the center vector of a target data group, the target data group being the data group to which the data to be operated belongs in K data groups, K being an integer greater than or equal to 1; executing the operation instruction according to the target learning index corresponding to the target key and the target data group.

[0009] Compared to traditional index structures, learned indexes are more lightweight, reducing storage space usage while also reducing the number of data queries and improving query efficiency. Furthermore, this application also performs product quantization compression on the data to be processed, reducing the space occupied by keys in the index structure and further reducing the storage space occupied by the index structure.

[0010] In one possible implementation, the K data groups are obtained by clustering the data in the dataset. Each data group includes data aggregated through clustering, and the data in each data group does not overlap. By clustering similar data in the dataset into the same data group, the data distribution within the data group is approximately consistent. This allows the learned index corresponding to each data group to more accurately learn the data distribution patterns within the data group, thereby improving the accuracy of the position predicted by the learned index.

[0011] In one possible implementation, the target learning index is used to output a pointer corresponding to the data to be operated based on the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit based on the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

[0012] The target learning index model learns the distribution pattern of the target key in the mapping unit, and the actual distribution of the data in the storage medium is decoupled from the target learning index model. Therefore, when writing data to the storage medium, there is no need to consider whether the writing of the data will affect the data distribution pattern in the storage medium. The data can be written to the storage medium using an append write method, which can improve write performance. In addition, when querying data, the target learning index model can quickly hit the location of the target key in the mapping unit, which can reduce the number of queries. The mapping unit stores the mapping relationship between the target key and the pointer, which can accurately obtain the storage location of the corresponding data, improving query accuracy and efficiency.

[0013] In one possible implementation, the target learning index model includes a root model, at least two intermediate models, and at least two leaf models. The root model is used to select a target intermediate model from the at least two intermediate models based on a target key. The target intermediate model is used to select a target leaf model from the at least two leaf models. The target leaf model is used to output a target position based on the target key. The target leaf model is a nonlinear function. The nonlinear function can better fit the distribution of keys, thereby more accurately predicting the key position.

[0014] In one possible implementation, at least a portion of the learned index model is stored in a memory corresponding to an external computing device, and the external computing device is used to obtain a target intermediate model, target leaf module, or target location based on at least a portion of the model. By offloading some query operations to the external computing device, query efficiency can be improved. The external computing device can be an intelligent chip, a graphics processing unit, an intelligent network card, or a programmable switch.

[0015] In one possible implementation, the operation instruction is a query instruction that instructs reading target data corresponding to the data to be operated. The operation instruction is executed based on the target key and the target learning index corresponding to the target data group. The operation instruction includes: obtaining a first pointer based on the target key through the target learning index, where the first pointer indicates the storage location of the target data; obtaining the target data based on the first pointer; wherein, when the first pointer is associated with a single data item, the target data is the data associated with the first pointer; when the first pointer is associated with multiple data items, the target data is the data with the smallest distance from the data to be operated among the multiple data items associated with the first pointer. Using the target learning index can improve data query efficiency.

[0016] In one possible implementation, the target data is stored as compressed target data, and obtaining the target data according to the first pointer includes: obtaining the compressed target data according to the first pointer; decompressing the compressed target data by a decoder to obtain the target data; the target data is structured data, and the target data includes N data segments divided according to attributes, and the compressed target data includes N compressed data segments, and the N compressed data segments are obtained by compressing the N data segments by M sub-encoders of the encoder, each sub-encoder is used to encode at least one data segment corresponding to an attribute, and the decoder includes M sub-decoders, and the M sub-decoders correspond one-to-one to the M sub-encoders, and each sub-decoder is used to decode at least one compressed data segment of the N compressed data segments. Compressing data according to the structure of the data can improve the compression rate of the data and reduce the storage space occupied by the data. M and N are both integers greater than or equal to 1.

[0017] In one possible implementation, the operation instruction is an update instruction; the operation instruction is executed according to the target learning index corresponding to the target key and the target data group, including: according to the target key, through the target learning index, obtaining a second pointer indicating the corresponding target key; according to the second pointer, writing the data to be operated into the storage medium.

[0018] In one possible implementation, the operation instruction is an update instruction. Based on the target key and the target data group corresponding to the target learned index, the operation instruction is executed, including: writing the data to be operated on to a storage medium; and updating the mapping relationship between the target key and a second pointer to the target learned index, where the second pointer indicates the storage location of the data to be operated on in the storage medium. The learned index can be modified when the data changes, thereby making the indexing results obtained by the learned index more accurate.

[0019] In one possible implementation, the operation instruction is an insert instruction. Based on the target key and the target data group corresponding to the target learned index, the operation instruction is executed, including: writing the data to be operated on to a storage medium; and inserting the mapping between the target key and a third pointer into the target learned index, where the third pointer indicates the storage location of the data to be operated on in the storage medium. The learned index can be modified when the data changes, thereby making the indexing results obtained by the learned index more accurate.

[0020] In one possible implementation, writing data to be operated on a storage medium includes: encoding and compressing the data to be operated on using an encoder corresponding to a target data group to obtain compressed data, wherein the data to be operated on is structured data and includes N data segments divided according to attributes; the compressed data includes N compressed data segments, and the N compressed data segments are obtained by encoding and compressing the N data segments using M sub-encoders of the encoder, each sub-encoder being configured to encode at least one data segment corresponding to one attribute; and writing the compressed data to the storage medium. Compressing the data based on its structure can improve the data compression rate and reduce the storage space occupied by the data.

[0021] In one possible implementation, the operation instruction is a delete instruction; based on the target key and the target learned index corresponding to the target data group, the operation instruction is executed, including: obtaining, based on the target key and the target learned index, a fourth pointer indicating the storage location of the target data corresponding to the data to be operated; deleting the data in the storage location indicated by the fourth pointer; and deleting the mapping relationship between the target key and the fourth pointer in the target learned index. When data changes, the learned index can be modified, making the indexing results obtained by the learned index more accurate.

[0022] In one possible implementation, the method further includes: when the data distribution in the K data groups satisfies a condition, re-clustering the data set to obtain K updated data groups, and updating the learned index corresponding to each updated data group. When the data distribution changes, the data is re-clustered and the corresponding learned index is trained to ensure efficient data query of the data block.

[0023] In the second aspect, the present application provides a data processing method. The method is applied to external computing devices, such as smart chips, graphics processing units, smart network cards or programmable switches. The method includes: obtaining a target key, wherein the target key is obtained by quantizing and encoding the product of the data to be operated or the target residual value in the operation request, and the target residual value is the residual value between the data to be operated and the center vector of the target data group. The target data group is the data group to which the data to be operated belongs among the K data groups, and K is an integer greater than or equal to 1; according to the target key, the pointer corresponding to the target key is obtained through the target learning index corresponding to the target data group, and the pointer indicates the storage location of the target data corresponding to the data to be operated. By offloading part of the query operation to the external computing device, the query efficiency can be improved.

[0024] In one possible implementation, obtaining the target key may involve obtaining the target key from a host, where a processor on the host performs product quantization encoding on the data to be operated to obtain the target key. In another possible implementation, obtaining the target key may involve an external computing device performing product quantization encoding on the data to be operated to obtain the target key.

[0025] In one possible implementation, the target learning index is used to output a pointer corresponding to the data to be operated based on the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit based on the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

[0026] The target learning index model learns the distribution pattern of the target key in the mapping unit, and the actual distribution of the data in the storage medium is decoupled from the target learning index model. Therefore, when writing data to the storage medium, there is no need to consider whether the writing of the data will affect the data distribution pattern in the storage medium. The data can be written to the storage medium using an append write method, which can improve write performance. In addition, when querying data, the target learning index model can quickly hit the location of the target key in the mapping unit, which can reduce the number of queries. The mapping unit stores the mapping relationship between the target key and the pointer, which can accurately obtain the storage location of the corresponding data, improving query accuracy and efficiency.

[0027] In one possible implementation, the target learning index model includes a root model, at least two intermediate models, and at least two leaf models. The root model is used to select a target intermediate model from the at least two intermediate models based on a target key. The target intermediate model is used to select a target leaf model from the at least two leaf models. The target leaf model is used to output a target position based on the target key. The target leaf model is a nonlinear function. The nonlinear function can better fit the distribution of keys, thereby more accurately predicting the key position.

[0028] On the third aspect, the present application provides a data processing device, which includes: a transceiver module for obtaining an operation request, the operation request including an operation instruction and data to be operated; a processing module for performing product quantization encoding on a target residual value or data to be operated to obtain a target key, the target residual value is the residual value between the data to be operated and the center vector of a target data group, the target data group is the data group to which the data to be operated belongs among K data groups, and K is an integer greater than or equal to 1; the processing module is used to execute the operation instruction according to the target learning index corresponding to the target key and the target data group.

[0029] In one possible implementation, the target learning index is used to output a pointer corresponding to the data to be operated based on the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit based on the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

[0030] In one possible implementation, the target learning index model includes a root model, at least two intermediate models and at least two leaf models, the root model is used to select a target intermediate model from at least two intermediate models according to a target key, the target intermediate model is used to select a target leaf model from at least two leaf models, the target leaf model is used to output a target position according to a target key, and the target leaf model is a nonlinear function.

[0031] In one possible implementation, at least part of the model in the learned index model is stored in a memory corresponding to an external computing device, and the external computing device is used to obtain a target intermediate model, a target leaf module or a target position based on at least part of the model.

[0032] In one possible implementation, the operation instruction is a query instruction, which instructs to read the target data corresponding to the data to be operated; the processing module is used to obtain a first pointer through a target learning index according to a target key, and the first pointer indicates the storage location of the target data; the processing module is used to obtain the target data according to the first pointer; wherein, when the first pointer is associated with one data, the target data is the data associated with the first pointer; when the first pointer is associated with multiple data, the target data is the data with the smallest distance to the data to be operated among the multiple data associated with the first pointer.

[0033] In one possible implementation, the target data is stored as compressed target data; a processing module is used to obtain the compressed target data according to the first pointer; the processing module is used to decompress the compressed target data through a decoder to obtain the target data; the target data is structured data, the target data includes N data segments divided according to attributes, the compressed target data includes N compressed data segments, the N compressed data segments are obtained by compressing the N data segments through M sub-encoders of the encoder, each sub-encoder is used to encode at least one data segment corresponding to an attribute, the decoder includes M sub-decoders, the M sub-decoders correspond one-to-one to the M sub-encoders, and each sub-decoder is used to decode at least one compressed data segment among the N compressed data segments.

[0034] In one possible implementation, the operation instruction is an update instruction; the processing module is used to write the data to be operated into the storage medium; the processing module is used to update the mapping relationship between the target key and the second pointer to the target learning index, and the second pointer indicates the storage location of the data to be operated in the storage medium.

[0035] In one possible implementation, the operation instruction is an insert instruction; the processing module is used to write the data to be operated into the storage medium; the processing module is used to insert the mapping relationship between the target key and the third pointer into the target learning index, and the third pointer indicates the storage location of the data to be operated in the storage medium.

[0036] In one possible implementation, the processing module is specifically used to encode and compress the data to be operated through an encoder corresponding to the target data group to obtain compressed data, where the data to be operated is structured data, the data to be operated includes N data segments divided according to attributes, and the compressed data includes N compressed data segments, which are obtained by encoding and compressing the N data segments through M sub-encoders of the encoder, and each sub-encoder is used to encode at least one data segment corresponding to an attribute; the processing module is specifically used to write the compressed data into a storage medium.

[0037] In one possible implementation, the operation instruction is a delete instruction; the processing module is used to obtain a fourth pointer indicating the storage location of the target data corresponding to the data to be operated through the target learning index based on the target key; the processing module is used to delete the data in the storage location indicated by the fourth pointer; the processing module is used to delete the mapping relationship between the target key and the fourth pointer in the target learning index.

[0038] In a possible implementation, the apparatus further includes: when the data distribution in the K data groups meets a condition, re-clustering the data set to obtain K updated data groups, and updating the learning index corresponding to each updated data group.

[0039] In a fourth aspect, the present application provides a data processing device. The device includes: a processing module for obtaining a target key, where the target key is obtained by quantizing and encoding the product of the data to be operated or the target residual value in the operation request, where the target residual value is the residual value between the data to be operated and the center vector of the target data group, where the target data group is the data group to which the data to be operated belongs among K data groups, where K is an integer greater than or equal to 1; and a processing module for obtaining a pointer corresponding to the target key through a target learning index corresponding to the target data group, based on the target key, where the pointer indicates the storage location of the target data corresponding to the data to be operated.

[0040] In a fifth aspect, the present application provides a computing device. The computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions to cause the computing device to perform the method of the first aspect or any possible implementation of the first aspect, or the method of the second aspect or any possible implementation of the second aspect.

[0041] In a sixth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer-readable instructions; the computer-readable instructions are used to implement the method in the first aspect or any possible implementation of the first aspect, or the method in the second aspect or any possible implementation of the second aspect.

[0042] In the seventh aspect, the present application provides a computer program product, characterized in that it includes computer-readable instructions; the computer-readable instructions are used to implement the method in the above-mentioned first aspect or any possible implementation of the first aspect, or the method in the second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 shows a structured data storage system combined with a large model;

[0044] Figures 2a and 2b are schematic diagrams of the software architecture of a data processing system provided by this application;

[0045] FIG3 is a schematic diagram of querying using the data processing system provided by the present application;

[0046] FIG4a is a schematic diagram of the structure of a learning index provided by this application;

[0047] FIG4 b is a schematic diagram of the structure of a learning indexing model provided by this application;

[0048] FIG5a is a schematic diagram of data encoded by a structure-aware encoder provided by the present application;

[0049] FIG5 b is a schematic diagram of a structure-aware decoder provided by the present application decoding compressed data;

[0050] FIG6 is a schematic diagram of the hardware architecture of a data processing system provided by the present application;

[0051] FIG7 is a flow chart of a data processing method provided by the present application;

[0052] FIG8 a is a schematic diagram of an index when the operation request is a query request;

[0053] FIG8 b is a schematic diagram of an index when the operation request is an update request;

[0054] FIG8c is a schematic diagram of an index when the operation request is an insert request;

[0055] FIG9 is a schematic diagram of retraining the index structure;

[0056] FIG10 is a comparison of the query test results of the solution provided by this application and the existing solution for 1 million vectors;

[0057] FIG11 is a comparison of query test results of 1 million vectors based on different databases using the solution provided by this application and the existing solution;

[0058] FIG12 is a schematic structural diagram of a data processing device provided by the present application;

[0059] FIG13 is a schematic structural diagram of another data processing device provided by the present application;

[0060] FIG14 is a schematic structural diagram of a computing device provided in this application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0062] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0063] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0064] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0065] The following is an explanation of the terms involved in this application:

[0066] Structured data refers to organized data with defined length and format. Structured data is a type of data that can be stored in tables or relational databases. This structure improves search efficiency when searching for specific data, helping to unlock data potential and realize its application value.

[0067] A learned index is an index structure that focuses on data distribution patterns. A learned index can be thought of as a machine learning model that learns data distribution patterns through training, resulting in an adaptive model. Input a key to be queried into the learned index, and it will output the location corresponding to that key.

[0068] Product quantization (PQ) is a vector quantization technique that decomposes a high-dimensional vector into low-dimensional subvectors and quantizes each subvector. PQ has been used as a method to provide faster approximate nearest neighbor searches. PQ is also applied to fields such as information retrieval and embedding matrix compression in natural language processing (NLP). Product quantization specifically includes three stages: codebook training, quantization, and query. During the codebook training stage, the n samples in the dataset (each sample is a D-dimensional vector) are divided into m subspaces. That is, each sample is divided into m D / m-dimensional subvectors, and each subspace contains n subvectors. In each subspace, the n subvectors are clustered using a clustering algorithm, such as the K-Means algorithm, to obtain K clusters. The cluster center (centroid) of each cluster is used as the representative of the cluster, and the center of each cluster is represented by a cluster ID. In this way, a codebook corresponding to each subspace can be obtained. During the quantization phase, each sample in the dataset is divided into m D / m subvectors. Each subvector is then encoded using the class number of the nearest cluster center within its subspace, resulting in an m-dimensional short vector. During the query phase, either symmetric or asymmetric distance calculation methods can be used to find the top k neighboring samples in the dataset for the query vector. Symmetric distance calculation involves calculating the distance between the short vector obtained by multiplying the query vector by quantization and encoding and the short vectors corresponding to all samples in the dataset. The top k neighboring samples are selected based on this distance. Asymmetric distance calculation involves performing the same partitioning of the query vector into m D / m-dimensional subvectors. Each subvector is then compared to the vectors of the K cluster centers in its subspace, resulting in a total of m*K distances, referred to as the distance table. The dataset samples are traversed, and the sum of the distances between the query vector and each sample is calculated according to the distance table. Based on this sum, the top k closest samples are selected.

[0069] Inverted product quantization uses a clustering algorithm to coarsely divide the samples in the data set into K classes, each class has a cluster center. For each sample in the data set, it is subtracted from the nearest cluster center to obtain a residual vector, and then the residual vector is product quantized to obtain a short vector. The process of product quantization of the residual vector is similar to the above-mentioned product quantization process and will not be repeated here. When querying, the residual vector between the query vector and each cluster center is calculated, and the class corresponding to the smallest residual vector is determined to be the class to which the query vector belongs. Then, a symmetric distance calculation method or an asymmetric distance calculation method is used to find the top k neighboring samples in the class to which the query vector belongs. The difference from product quantization is that inverted product quantization finds neighboring samples in the class to which the query vector belongs based on the short code corresponding to the residual vector. This can reduce errors, reduce the query range, and improve query efficiency.

[0070] Lossless compression uses the statistical redundancy of data to perform compression, which can completely restore the original data without causing any distortion. However, the compression rate is theoretically limited by the statistical redundancy of the data.

[0071] Structured data databases primarily store structured data, with each piece of structured data representing a data object. Structured data databases often contain a large number of data objects. The dimensionality of structured data is typically determined by the user or the data object representation. Structured data databases primarily address two challenges: efficient querying and efficient analysis and storage.

[0072] When building a structured data database, it analyzes all data objects and constructs an efficient indexing system, allowing closely related structured data to quickly find each other. When a user queries structured data, the structured data database uses its indexing system to find the k most relevant structured data objects, thereby locating the corresponding data objects.

[0073] With the advent of the big model era, structured data databases are experiencing explosive demand. As shown in Figure 1, structured data databases play two important roles: 1. As a cache: Structured data databases can cache popular questions and the answers from big models. This allows other users to quickly respond to similar questions, avoiding the computational overhead and time-consuming process of re-answering by the big model. 2. As user private data storage: Training big models based on user private data storage is time-consuming and resource-intensive. Using structured data databases, users can quickly find the relevant data objects they need and optimize processing based on the big model. This reduces training costs and improves the efficiency of big models in answering domain questions.

[0074] As user data volumes continue to grow, existing structured data database technologies have struggled to maintain sufficient scale. Beyond the data objects themselves, their structured data representations become a new form of metadata storage, placing significant storage overhead and computing pressure on storage systems.

[0075] Regarding efficient queries, inverted product quantization is used for high-dimensional vector similarity searches, which can improve query performance to a certain extent. However, when storing large-scale structured data, it still poses problems such as excessive index space usage and high query overhead. For example, in a TopK query scenario for similar images, a single inverted product quantization query takes approximately 100 μs.

[0076] Traditional compression algorithms for efficient storage and analysis include entropy coding and dictionary-based compression. Entropy coding, based on arithmetic coding and interval coding, can compress a given piece of data. Dictionary-based compression technologies such as Run-Length Encoding, LZ77 / LZW / LZAM, and zstd offer inferior compression performance compared to entropy coding, and their effectiveness is generally stronger on text. With traditional compression algorithms, randomly decompressing a piece of data often requires decompressing all the data blocks involved in the compression. This leads to the problem of read amplification. Furthermore, traditional algorithms cannot distinguish between similar data for compression, and the compression rate needs to be further improved.

[0077] In order to solve the above technical problems, this application provides the following solutions.

[0078] As shown in Figures 2a and 2b, Figures 2a and 2b are schematic diagrams of the architecture of the data processing system provided by this application. The data processing system includes a data group selection module, a quantization module, and a learning index module. The data group selection module is used to determine the data group to which the data belongs. The quantization module is used to quantize and encode the data to obtain a short code vector to compress the data length. The learning index module uses the short code vector as a key and outputs the value (pointer) corresponding to the key.

[0079] The lifecycle of a data processing system consists of the construction phase, the deployment phase, and the retraining phase. The construction phase trains the parameters in the data group selection module, quantization module, and learning index module. The deployment phase executes user commands based on the trained data processing system. The retraining phase retrains the parameters in the data group selection module, quantization module, and learning index module when changes in the data distribution of the data managed by the data processing system meet certain requirements.

[0080] Specifically, during the construction phase, a clustering algorithm (such as K-Means, etc.) is used to cluster multiple data in the data set into K data groups (or clusters), and the vector of the cluster center of each data group is obtained (i.e., the centroid of the cluster, hereinafter referred to as the center vector). The data in multiple data groups do not overlap. Similar data in the data set are clustered into the same data group. K is an integer greater than or equal to 1. After the data groups are divided, the PQ codebook in the quantization module is trained. In one possible implementation, the original data in the data group is used to train the PQ codebook, and then the corresponding PQ codebook can be trained for each data group. In another possible implementation, the residual vector between the data in the data group and the center vector of the data group is used to train the PQ codebook. In this case, all data groups can share one PQ codebook. In another possible implementation, some data groups may use original data to train the PQ codebook, and some data groups may use residual vectors to train the PQ codebook. In this case, the data groups that use original data to train the PQ codebook are trained separately to obtain corresponding PQ codebooks, that is, one data group corresponds to one PQ codebook; the data groups that use residual vectors to train the PQ codebook can jointly train one PQ codebook, that is, these data groups can share one PQ codebook.

[0081] To improve the indexing accuracy of the learned index module, in this embodiment, each data group corresponds to a learned index. For a learned index corresponding to a data group, the short code vector corresponding to the data in the data group is used as the key, and the pointer indicating the storage location of the data is used as the value to train the learned index. This allows the learned index to learn the mapping relationship between the key and pointer corresponding to the data in the data group, and thus accurately output the storage location of the corresponding data based on the key.

[0082] After the construction is complete, the center vector of each data group (a parameter in the data group selection module), the PQ codebook (a parameter in the quantization module), and the learned index corresponding to each data group (a parameter in the learned index module) can be obtained. Thus, the constructed data processing system can handle user operation requests. Operation requests include, for example, query requests, update requests, insert requests, and delete requests. Operation requests include the data to be operated on. Operation requests can be single operations or batch operations.

[0083] During the use phase, as shown in Figure 3 , the data group selection module is used to determine the target data group to which the data to be operated belongs. There are various ways to determine the data group to which the data to be operated belongs. For example, the data group selection module calculates the residual between the data to be operated and the center vectors of K data groups, and selects the data group with the smallest residual value as the target data group. Another example is the data group selection module calculates the distance between the data to be operated and the center vectors of the K data groups, and selects the data group with the smallest distance as the target data group.

[0084] The quantization module is used to quantize the data to be operated into a short code vector, or quantize the target residual vector between the data to be operated and the target data group into a short code vector. The short code vector is the target key for input into the learning index module. When the data quantized by the quantization module is the data to be operated, the quantization module selects the PQ codebook corresponding to the target data group to quantize the data to be operated, and obtains the short code vector corresponding to the data to be operated. When the data quantized by the quantization module is the target residual vector, the quantization module quantizes the target residual vector using the PQ codebook obtained by residual vector training, and obtains the short code vector corresponding to the data to be operated.

[0085] The learning index module includes a learning index corresponding to multiple data groups. As shown in Figure 4a, the learning index includes a learning index model and a mapping unit. It should be understood that Figure 4a is merely an example and should not be construed as limiting the present application. The learning index model learns the distribution pattern of keys (short code vectors) in the mapping unit through training and can accurately predict the location of keys in the mapping unit. The mapping unit stores multiple key-value (or key-pointer) pairs, where the value is a pointer indicating the storage location of the data corresponding to the key. For any data in the data group, there is a corresponding pointer in the mapping unit indicating its storage location. Therefore, the key-pointer pair in the mapping unit can accurately address the data in the data group. The process of querying the pointer of the data corresponding to a key by the learning index is as follows: the learning index model outputs the location of the key in the mapping unit based on the key. The location stores a pointer that has a mapping relationship with the key. The mapping unit obtains the pointer of the location from the location output by the learning index model and outputs the pointer. In this embodiment, the pointer can be the storage location of the data. The pointer can also be an identifier of the data, which can be used to obtain the storage location of the data.

[0086] In one possible implementation, as shown in FIG4b , the learned index model has a hierarchical structure, comprising a root model, at least two intermediate models, and at least two leaf models. The root model is used to select an intermediate model from the at least two intermediate models based on a key. The intermediate model selected by the root model is used to select a leaf model from the at least two leaf models based on a key. In one possible implementation, the learned index is stored in the form of a linear table. The root model can output the offset of the selected intermediate model, and the intermediate model can output the offset of the selected leaf model, thereby retrieving and running the corresponding model based on the offset. In another implementation, the root model can output the model identifier of the selected intermediate model, and the intermediate model can output the model identifier of the selected leaf model. The storage location of the model is determined based on the mapping relationship between the model identifier and the storage location of the model corresponding to the model identifier in memory, thereby retrieving and running the model corresponding to the model identifier. The leaf model selected by the intermediate model is used to output the position of the key in the mapping unit based on the key. Optionally, the leaf model can be a nonlinear function, such as ReLU, sigmoid, and Tanh. Nonlinear functions can better fit the distribution of keys, thereby more accurately predicting the position of keys. Of course, in other possible implementations, the learned index model may also be a fitting-tree index, an adaptive learned index (ALEX), a piecewise geometric model index (PGM), an updateable learned index with precise positions (LIPP), etc. In this embodiment, the K learned index models in the K learned indexes may use the same learning index model or different learning index models, which is not limited here.

[0087] Optionally, the learned index model is a lightweight model. For example, the learned index model does not include a complex neural network model, so that the storage space occupied by the learned index model is small and can be cached in the memory. In one possible implementation, the learned index model can be cached in its entirety in the memory on the host side. In another possible implementation, the learned index model can be cached in its entirety in the memory on the device side. In yet another possible implementation, the learned index model can be partially cached in the memory on the host side and partially cached in the memory on the device side. For example, the root model and the intermediate model are cached in the memory on the device side, while the leaf model is cached in the memory on the host side. Alternatively, the root model and the leaf model are cached in the memory on the host side, while the intermediate model is cached in the memory on the device side, and so on. Examples are not given here one by one.

[0088] A mapping unit includes multiple sub-mapping units (also called buckets), each of which stores at least one key-pointer pair. The learned indexing model outputs the position of the key in the mapping unit based on the key, which indicates the location of the sub-mapping unit storing the key-pointer pair corresponding to that key. A sub-mapping unit may store only one key-pointer pair, meaning that a sub-mapping unit only stores the mapping relationship between one key and one pointer. A sub-mapping unit may also store multiple key-pointer pairs. In one possible implementation, the multiple key-pointer pairs stored in a sub-mapping unit may be mapping relationships between the same key and multiple pointers. This is because when high-dimensional data is compressed into low-dimensional short code vectors through quantization, conflicts may occur. That is, different but similar data may be quantized and compressed to produce the same short code vector, resulting in a mapping relationship between one short code vector and pointers to multiple data. Alternatively, the multiple key-pointer pairs stored in a sub-mapping unit are mapping relationships between multiple keys and multiple pointers, with one pointer only mapping to one key. A sub-mapping unit stores multiple key-pointer pairs. This situation is caused by conflicts caused by the learned indexing model itself, mapping different short code vectors to the same sub-mapping unit. This conflict can be reduced by optimizing the learned index. For example, the learned index model can be trained in an insertion manner, that is, by inserting key-pointer pairs into the mapping unit, and only inserting a key-pointer pair corresponding to one key into a sub-mapping unit to ensure that the keys are distributed in an orderly manner in the mapping unit. Then, the learned index model is trained based on the key and its position in the mapping unit. The learned index model is trained and optimized through a bottom-up training method, so that when querying, the learned index model can accurately predict the position of the key in the mapping unit and obtain the corresponding pointer.

[0089] A pointer may indicate the storage location of only one piece of data. It may also indicate the storage locations of multiple pieces of data. For example, different but similar data may be quantized and compressed to produce the same short code vector. In one possible implementation, each piece of data may correspond to a pointer indicating its storage location, with the sub-mapping unit storing the mapping between a key and multiple pointers. In another possible implementation, the data may be stored continuously, with a single pointer indicating the storage location of the data. The sub-mapping unit also stores the mapping between a key and a pointer.

[0090] As shown in Figure 4a, the possible relationships between sub-mapping units, keys, pointers, and data are summarized below:

[0091] 1. A sub-mapping unit stores a key-pointer pair, and a pointer indicates the storage location of a piece of data. In this case, the learned indexing module outputs a pointer that uniquely indicates the storage location of a piece of data. For example, as shown in Figure 4a, sub-mapping unit B1 stores a key-pointer mapping relationship (key1, pointer1), and pointer pointer1 only indicates the storage location of data1.

[0092] 2. A sub-mapping unit stores a key-pointer pair, and a pointer indicates the storage location of multiple data. That is, the short code vectors corresponding to multiple data are the same, and these data are stored continuously, and the storage location of these data is indicated by a pointer. In this case, the learning index module outputs a pointer that indicates the storage location of multiple data. For example, as shown in Figure 4a, the sub-mapping unit B2 stores a key-pointer mapping relationship (key2, pointer2), and the pointer pointer1 indicates the storage location of data data2 and data3.

[0093] 3. A sub-mapping unit stores multiple key-pointer pairs, where the key in each key-pointer pair is the same key, and different pointers indicate the storage locations of different data. In this case, the learning index module outputs multiple pointers, each indicating the storage location of a piece of data. For example, sub-mapping unit B3 in Figure 4a stores two key-pointer mapping relationships (key3, pointer3) and (key3, pointer4), where pointer pointer3 indicates the storage location of data data4, and pointer pointer4 indicates the storage location of data5.

[0094] 4. A sub-mapping unit stores multiple key-pointer pairs, and the keys in the multiple key-pointer pairs include different keys. For example, the sub-mapping unit B4 in Figure 4a stores two key-pointer mapping relationships (key4, pointer5) and (key5, pointer6), pointer pointer5 indicates the storage location of data data6, and pointer pointer6 indicates the storage location of data7. In this case, the learning index module can match the input key with the multiple key-pointer pairs in the sub-mapping unit, and only output the pointer that has a mapping relationship with the input key. When the key input to the learning index module has a mapping relationship with only one pointer and the pointer only indicates the storage location of one data, the learning index module outputs a pointer that uniquely indicates the storage location of one data. When the key input to the learning index module has a mapping relationship with only one pointer and the pointer indicates the storage location of multiple data, the learning index module outputs a pointer that indicates the storage location of multiple data. When a key input to the learning index module has a mapping relationship with multiple pointers and one pointer only indicates a storage location of one data, the learning index module outputs multiple pointers, each pointer indicating a storage location of one data.

[0095] In the following, a pointer indicating the storage location of a data is used as an example for explanation.

[0096] When the operation instruction is a query instruction, an update instruction or a delete instruction, it is necessary to query the storage location of the target data corresponding to the data to be operated through the learning index module according to the target key corresponding to the data to be operated. Specifically, according to the target data group to which the data to be operated belongs determined by the data group selection module, the target learning index corresponding to the target data group can be determined in multiple learning indexes, and the target learning index module can output the pointer corresponding to the target data according to the target key corresponding to the data to be operated. The target data can be the data to be operated, the data containing the data to be operated, the data associated with the data to be operated, or the data similar to the data to be operated. For the case where the target key corresponds to multiple data (there is a mapping relationship between the target key and multiple pointers), multiple data can be read out according to the multiple pointers, and these data are candidate data. Then the distance between each candidate data and the data to be operated is calculated respectively, and the candidate data closest to the data to be operated is determined as the target data, and the pointer corresponding to the candidate data closest to the data to be operated is the target pointer, and then the corresponding operation is performed on the target data and / or the target pointer.

[0097] When the operation instruction is an update instruction, an insert instruction or a delete instruction, it may cause the mapping relationship between the key and the pointer to change, or generate a new mapping relationship between the key and the pointer. In this case, the mapping relationship between the newly added or changed key and the pointer needs to be updated to the mapping unit.

[0098] When the operation instruction is an insert or delete instruction, the corresponding key is inserted or deleted in the mapping unit, which may cause the key distribution in the mapping unit to change. When the key distribution in a certain area of ​​the mapping unit changes or accumulates a certain amount of change, the leaf model corresponding to that area can be retrained, that is, the learned index model can be locally adjusted. This can ensure the accuracy of the learned index model's predictions, reduce conflicts, and improve training speed.

[0099] By PQ encoding the data to obtain a short code vector, on the one hand, the dimension of the data is reduced, which can reduce the storage space occupied by the mapping unit, reduce the cost of learning index training and prediction, and improve the prediction performance. On the other hand, it can maintain order and ensure the orderly and uniform distribution of keys, thereby improving the prediction accuracy of the learning index model. Through the learning index model, the position of the key in the mapping unit can be quickly and accurately located without having to determine the position of the key in the mapping unit through traversal or binary search. The mapping unit stores the mapping relationship between the key and the pointer corresponding to the data. The pointer indicates the precise storage location of the data. According to the position of the key, the pointer that has a mapping relationship with the key can be obtained. The pointer directly or indirectly indicates the storage location of the data, so that the precise storage location of the data can be obtained.

[0100] During the retraining phase, due to operations such as updating, inserting, and deleting, there may be large differences in the data distribution between the data groups, which in turn affects the query performance and accuracy of the learning index module. Therefore, when the data distribution differences between the data groups meet the conditions, the data set can be retrained. The data distribution differences between the data groups meeting the conditions may mean that the difference in the number of elements in different data groups exceeds a threshold, or the mean square error of the number of elements in different data groups exceeds a threshold. Retraining is to re-divide the multiple data clusters in the data set into K data groups, and obtain the vector of the cluster center of each data group, as well as retrain the PQ codebook and the learning index corresponding to each data group. The retraining process is similar to the construction process and will not be repeated here.

[0101] Optionally, to improve the utilization of the storage medium, when writing data to the storage medium for persistent storage, the data can be first compressed to obtain compressed data, and then the compressed data can be written to the storage medium. Accordingly, when reading data from the storage medium, the compressed data needs to be decompressed to obtain uncompressed data. The storage medium is, for example, a persistent storage device such as a solid-state drive or a mechanical hard drive.

[0102] Therefore, the data processing system may further include a codec module and a compression array. The codec module includes an encoder and a decoder. The encoder is used to encode and compress data to be written to the storage medium to obtain compressed data. The compressed data is stored in the compression array on the storage medium. The decoder is used to decode the compressed data read from the storage medium and restore it to uncompressed data.

[0103] During the system construction phase, the encoding and decoding modules are also trained. For example, all data in the dataset is structured data, with a consistent structure. Each data segment consists of multiple data segments with different attributes. The encoder is a structure-aware encoder, meaning it can perceive data segments with different attributes within the data and train a corresponding sub-encoder and sub-decoder for each attribute segment. The encoder can perceive the structure of the data based on hints and thus divide the data into data segments with different attributes. The encoder can also perceive the structure of the data by analyzing data patterns and thus divide the data into data segments with different attributes. For example, if the data includes gender and age fields, then the data segment corresponding to the gender field is an attribute, and the data segment corresponding to the age field is an attribute. Assuming the data is a 100-dimensional vector, with the age field corresponding to dimensions 0-3 and the gender field corresponding to dimensions 4-5, dimensions 0-3 can be divided into one data segment, and dimensions 4-5 into one data segment.

[0104] Optionally, to improve the compression rate, an encoder and a decoder can be trained for each data group. Of course, K data groups can also share one encoder and one decoder, which is not a limitation here. Taking the example of training an encoder for each data group, when training the encoder, the corresponding sub-encoder is trained using data segments with the same attributes in the data group. The data in the data segments with the same attributes have certain regularities, such as the higher frequency of occurrence of certain specific characters and the concentration of character types, so that the sub-encoder at the training location has a higher compression rate for the data segments with the same attributes. During retraining, the codec module can be untrained, that is, the codec module remains unchanged, avoiding the need to read, process, decompress, and then recompress the compressed data from the storage medium to avoid affecting the normal use of the database. The encoder can be a structure-aware Huffman encoder, etc. Of course, the encoder can also be other structure-aware lossless encoders, such as structure-aware Shannon coding or structure-aware arithmetic coding, etc. entropy encoders.

[0105] When writing data to a storage medium, a trained encoder is used to compress and encode the data. Assume that the data includes N data segments divided according to attributes, and the encoder includes M sub-encoders. Each sub-encoder is used to encode a data segment corresponding to an attribute, and each sub-encoder is used to encode at least one data segment among the N data segments. The sub-encoder is trained based on the data segment corresponding to the attribute in the data group to which the data belongs. N is greater than or equal to M, and both M and N are integers greater than 1. Since the data before compression is structured data, the lengths of different data segments of the data are fixed. Therefore, the data can be split into multiple data segments according to the length corresponding to each data segment, and then the data segments of the corresponding attributes are encoded and compressed in parallel by each sub-encoder, thereby improving the compression efficiency. Of course, the data can also be encoded serially through each sub-encoder in turn, which is not limited here.

[0106] For example, as shown in Figure 5a, the data includes four data segments corresponding to four attributes: data segment 1 corresponding to attribute 1, data segment 2 corresponding to attribute 2, data segment 3 corresponding to attribute 3, and data segment 4 corresponding to attribute 4. The encoder includes four sub-encoders: sub-encoder 1, sub-encoder 2, sub-encoder 3, and sub-encoder 4. Sub-encoder 1 is trained for data segments corresponding to attribute 1 of multiple data in the data group, sub-encoder 2 is trained for data segments corresponding to attribute 2 of multiple data in the data group, sub-encoder 3 is trained for data segments corresponding to attribute 3 of multiple data in the data group, and sub-encoder 4 is trained for data segments corresponding to attribute 4 of multiple data in the data group. Sub-encoder 1 encodes data segment 1 to obtain compressed data segment 1, sub-encoder 2 encodes data segment 2 to obtain compressed data segment 2, sub-encoder 3 encodes data segment 3 to obtain compressed data segment 3, and sub-encoder 4 encodes data segment 4 to obtain compressed data segment 4. The four sub-encoders can simultaneously encode the corresponding data segments and then concatenate compressed data segments 1, 2, 3, and 4 to obtain the complete compressed data. The four encoders can also encode the corresponding data segments in sequence. For example, sub-encoder 1 first encodes data segment 1, then sub-encoder 2 encodes data segment 2, then sub-encoder 3 encodes data segment 3, and finally data segment 4 encodes data segment 4.

[0107] When data stored in a storage medium needs to be read, a decoder is required to decompress the data. Assume that the compressed data consists of N compressed data segments. The decoder includes M sub-decoders, each corresponding to one of the M encoders. Each sub-decoder is used to decode a compressed data segment corresponding to an attribute, and each sub-decoder is used to decode at least one of the N compressed data segments. Since the length of the data segments in the data is not fixed after compression, the M sub-decoders can decompress the compressed data serially, and each sub-decoder can identify the end position of its corresponding compressed data segment in the compressed data.

[0108] For example, as shown in Figure 5b, the compressed data includes four compressed data segments corresponding to four attributes: compressed data segment 1 corresponding to attribute 1, compressed data segment 2 corresponding to attribute 2, compressed data segment 3 corresponding to attribute 3, and compressed data segment 4 corresponding to attribute 4. The decoder includes four sub-decoders: sub-decoder 1, sub-decoder 2, sub-decoder 3, and sub-decoder 4. Sub-decoder 1 corresponds to sub-encoder 1, sub-decoder 2 corresponds to sub-encoder 2, sub-decoder 3 corresponds to sub-encoder 3, and sub-decoder 4 corresponds to sub-encoder 4. After the compressed data is input into the decoder, sub-decoder 1 first decompresses the compressed data from the beginning until it recognizes compressed data segment 1. Sub-decoder 2 then begins decompressing the compressed data from undecompressed locations until it recognizes compressed data segment 2. Sub-decoder 3 then begins decompressing the compressed data from undecompressed locations until it recognizes compressed data segment 3. Finally, sub-decoder 4 begins decompressing the compressed data from undecompressed locations until it completes decompression of compressed data segment 4, thereby obtaining the complete decompressed data.

[0109] The compressed data belonging to the same data group can be stored continuously in the storage medium. Since the length of the compressed data is not fixed, this embodiment records the starting position information of the compressed data in the storage medium through the compression header. Since multiple compressed data are stored continuously, the compression header records the data block where the starting position of the compressed data is located, as well as which bit in the data block. Optionally, the compression header can be compressed and stored in the storage medium to reduce the storage space occupied by the compression header. Of course, the compression header can also be stored directly in the storage medium without compression. Optionally, the compression header can be set before the compressed data of the same data group stored continuously as the header of the data group.

[0110] When it is necessary to read the target data from the storage medium, the starting position of the compressed target data corresponding to the target data can be determined based on the pointer and the compression header. In one possible implementation, the pointer can be an identifier (ID) of the target data. Each data in the data group will be assigned an ID, and the ID number is consistent with the storage order of the compressed data. The starting position information of each data in the data header can be stored in the form of an array, and the starting position information of each data in the compression header has the same length. According to the pointer and the length of the starting position information, it is possible to calculate at which position in the compression header the starting position information of the compressed target data is stored, thereby obtaining the starting position information of the compressed target data from this position, and then the compressed target data can be read according to the starting position information. In another possible implementation, the pointer can be the position of the starting position information of the compressed target data in the compression header, thereby obtaining the starting position information of the compressed target data from the compression header according to the pointer, and then the compressed target data can be read according to the starting position information.

[0111] When the compression header is stored in a storage medium in a compressed form, the compression header can be read and decompressed first. In one possible implementation, the entire compression header can be decompressed and cached in the memory. In another possible implementation, the compression header can be decompressed on demand, that is, it can be decompressed to the point where the starting position information of the compressed target data can be obtained. The decompressed portion of the compression header can be cached in the memory. When the starting position information of the queried data is subsequently cached in the memory, there is no need to decompress the compression header again, and the actual position information of the queried data can be directly obtained from the memory, thereby improving the query efficiency. When the starting position information of the queried data has not been decompressed into the memory, decompression can be started from the uncompressed position of the compression header until the starting position information of the queried data can be obtained, thereby eliminating the need to decompress the compression header from the beginning and improving the data query efficiency.

[0112] In this embodiment, when data is stored in a storage medium in the form of compressed data, the starting position information of each compressed data item is recorded in the compression header. This allows for random reading of any compressed data without having to read and decompress all other unrelated data. This reduces write amplification and improves compressed data reading performance. Furthermore, in this embodiment, data is segmented and compressed according to the data structure, which improves the data compression rate, further reduces the storage space occupied by data storage, and improves the utilization of the storage medium.

[0113] As shown in Figure 6, Figure 6 is a schematic diagram of the hardware system architecture provided by this application. The hardware system architecture in this embodiment. The hardware system includes a host central processing unit (CPU), an external computing device and a memory. The host CPU and the external computing device are respectively connected to the memory. The memory is used to store query requests, and the external computing device can obtain the query request from the memory to perform corresponding calculations and obtain calculation results (for example, a pointer indicating the data storage location, or information indicating the selected leaf node, etc.). The calculation results calculated by the external computing device are stored in the memory, and the host CPU obtains the calculation results from the memory and performs subsequent query operations based on the calculation results. The external computing device is used to accelerate the target position of the output key of the learning index model in the mapping unit. The external computing device can be a device with computing power such as a programmable switch, an intelligent network card and an intelligent chip, so that part of the calculation during the query process can be offloaded to the external computing device, thereby improving query efficiency. Intelligent chips include, for example, graphics processing units (GPUs), tensor processing units (TPUs), neural network processors (NPUs) or data processors (DPUs). When processing batch query requests, the external computing device can process multiple query requests in parallel, significantly improving query efficiency in batch query request scenarios. Specifically, the external computing device includes multiple computing units, such as multiplication and accumulation units (MACs) or matrix computing units. These multiple computing units can process query requests in parallel, thereby increasing the throughput of batch query processing.

[0114] The memory may be a high-speed memory such as a volatile memory or a non-volatile memory. Specifically, the memory may be a double data rate (DDR) synchronous dynamic random access memory (SDRAM) or a high bandwidth memory (HBM).

[0115] The learned index model can be fully stored in the memory corresponding to the external computing device. When processing a query request, the external computing device runs the target learned index model corresponding to the data group to which the data to be operated belongs, uses the target key corresponding to the data to be operated as the input of the target learned index model, and outputs the target location corresponding to the key to the memory.

[0116] In another implementation, partial models (e.g., root models and / or intermediate models) of the learned index model are stored in the memory of an external computing device. When processing a query request, the external computing device runs the partial model of the target learned index model corresponding to the data group to which the data to be operated belongs, uses the target key corresponding to the data to be operated as input to the partial model, and outputs the selected intermediate model or leaf model to the memory.

[0117] Optionally, product quantization of the data may also be performed by an external computing device to accelerate the output of the short code vector.

[0118] The mapping unit is stored in, for example, a host-side memory. After obtaining the target location from the memory, the host CPU retrieves the pointer corresponding to the target location. If the target location only stores a mapping relationship between the target key and one pointer, and the pointer indicates the storage location of only one piece of data, the host CPU can return the target data corresponding to the pointer. If the target location stores mapping relationships between the target key and multiple pointers, the host CPU calculates the distance between the data corresponding to the multiple pointers and the data to be operated on, and returns the data with the shortest distance to the data to be operated on.

[0119] The above-mentioned external computing device is an optional device, that is, the product quantization and index query of data are all implemented by the host CPU and are not restricted here.

[0120] As shown in FIG7 , FIG7 is a flow chart of a data processing method provided by the present application. The method is executed by the above-mentioned data processing system. The method includes:

[0121] S701: Acquire an operation request, where the operation request includes an operation instruction and data to be operated.

[0122] An operation request includes an operation instruction and the data to be operated. An operation instruction may be, for example, a query instruction, an insert instruction, an update instruction, or a delete instruction. An operation request may be a batch operation request or a single operation request.

[0123] S702: Perform product quantization encoding on the target residual value or the data to be operated to obtain a target key. The target residual value is the residual value between the data to be operated and the center vector of the target data group. Each data group includes data gathered by clustering, and K is an integer greater than or equal to 1.

[0124] When the PQ codebook is trained based on residual values, the target residual value is PQ-encoded, and the resulting short code vector serves as the target key. The target residual value is the residual value between the target data and the center vector of the target data group. The target data group is the data group to which the target data belongs, among the K data groups. Each data group includes similar data gathered through a clustering algorithm. The clustering and PQ encoding of data groups can be found in the relevant description above and will not be repeated here.

[0125] When the PQ codebook is trained based on the original data, the data to be operated is PQ encoded, and the obtained short code vector is used as the target key.

[0126] S703: Execute the operation instruction according to the target learning index corresponding to the target key and the target data group.

[0127] In this embodiment, the data in the dataset is clustered to group similar data into the same data group. A corresponding learning index is then trained for each data group to improve index accuracy and performance. The training and structure of the learning index can be found in the relevant description above and will not be repeated here.

[0128] When querying data, the target learned index corresponds to the target data group to which the data to be operated belongs. The target learned index takes the target key as input, and it outputs a pointer to the storage location of the target data corresponding to the data to be operated. When data is modified, a new key-pointer pair is generated, which needs to be updated to the target learned index.

[0129] The following describes how to execute an operation instruction when the operation request is a query request, an insert request, an update request, or a delete request.

[0130] 1. The operation request is a query request

[0131] As shown in Figure 8a, for the data to be operated, after obtaining the corresponding target key, the target key is used as the input of the target learning index to obtain the first pointer output by the target learning index. Then, the target data is obtained according to the pointer. When the first pointer includes multiple pointers, the candidate data indicated by each pointer is obtained respectively, and the distance between each candidate data and the data to be operated is calculated respectively, and the candidate data with the smallest distance is determined to be the target data. When the first pointer is a pointer, and the pointer only indicates the storage location of one data, the data obtained according to the first pointer is the target data. When the first pointer is a pointer, and the storage location indicated by the pointer includes multiple candidate data, the distance between each candidate data and the data to be operated is calculated respectively, and the candidate data with the smallest distance is determined to be the target data.

[0132] If the target data is stored as compressed target data in the storage medium, the compressed target data needs to be decompressed. Specifically, when the first pointer indicates the storage location of multiple compressed candidate data, the compressed candidate data needs to be decompressed, and then the distance between the decompressed candidate data and the data to be operated is calculated to determine the closest data to be operated as the target data. When the first pointer only indicates the storage location of data in a compressed state (i.e., compressed target data), the compressed target data is read out and decompressed into uncompressed target data. The decompression method refers to the relevant description above and will not be repeated here.

[0133] 2. The operation request is an update request

[0134] In one implementation, after obtaining the corresponding target key for the data to be operated on, the target key is used as the input of the target learned index to obtain the pointer output by the target learned index. The data in the storage location indicated by the pointer output by the target learned index is then updated to the data to be operated on. In this case, there is no need to modify the target learned index.

[0135] In one implementation, the data to be operated is written to a blank position in the storage medium in an append manner to obtain a mapping relationship between the target key and the second pointer indicating the position where the data to be operated is written. The mapping relationship between the target key and the original pointer in the target learning index is then modified to a mapping relationship between the target key and the second pointer. In the present application, since the target learning index includes a target learning index model and a mapping unit, the target learning index model learns the distribution pattern of the target key in the mapping unit, and the actual distribution of the data in the storage medium is decoupled from the target learning index model. Therefore, when writing data to the storage medium, there is no need to consider whether the writing of the data will affect the data distribution pattern in the storage medium. The data can be written to the storage medium in an append manner, which can improve the writing performance.

[0136] As shown in Figure 8b, if the data is stored in a compressed form in the storage medium, after obtaining the corresponding target key, the target key is used as the input of the target learning index to obtain the pointer output by the target learning index. Then compare the lengths of the first compressed data corresponding to the pointer output by the target learning index with the second compressed data corresponding to the data to be operated. If the length of the first compressed data is greater than or equal to the second compressed data, the first compressed data is replaced with the second compressed data at the position indicated by the pointer output by the target learning index. If the length of the first compressed data is less than the second compressed data, the second compressed data is written to a blank position in the storage medium to obtain a mapping relationship between the target key and the second pointer indicating the position where the second compressed data is written. Then modify the mapping relationship between the target key and the original pointer in the target learning index to a mapping relationship between the target key and the second pointer. The method of compressing the data to be operated to obtain the second compressed data can be found in the relevant description above and will not be repeated here.

[0137] 3. The operation request is an insert request

[0138] As shown in Figure 8c, the data to be operated is written to the storage medium in an append-write manner, obtaining a third pointer indicating the storage location of the data to be operated on the storage medium. The mapping relationship between the target key and the third pointer is then inserted into the target learning index, so that the corresponding pointer can be retrieved through the target learning index when the data to be operated is subsequently queried. Specifically, the mapping relationship between the target key and the third pointer is inserted into the mapping unit of the target learning index according to the order of the target keys, thereby ensuring the orderly distribution of the keys in the mapping unit and the accuracy of the target learning index model's output of the target key's location in the mapping unit based on the target key. Because the target key corresponding to the data to be operated did not previously exist, the target learning index model in the target learning index had not yet learned the mapping relationship between the target key and the target key's location in the mapping unit. To improve the prediction accuracy of the target learning index model, the target learning index model can be retrained after obtaining the mapping relationship between the target key and the third pointer, or retrained after a period of time, or after accumulating a certain number of mapping relationships.

[0139] Optionally, the compressed data obtained by encoding and compressing the data to be operated on can be written to a storage medium, and a third pointer indicating the storage location of the compressed data corresponding to the data to be operated on can be obtained. In this case, the third pointer can be an ID assigned to the compressed data corresponding to the data to be operated on, or a location in the compression header that stores the starting location information of the compressed data.

[0140] 4. The operation request is a deletion request

[0141] When the target data corresponding to the data to be operated is deleted, the mapping relationship between the target key and the pointer indicating the target data storage location no longer exists, so it is necessary to delete the mapping relationship in the mapping unit of the target learning index. It is possible to delete only the mapping relationship between the target key and the pointer indicating the target data storage location, so that the target data cannot be queried. The target data in the storage medium can also be further deleted. Specifically, after obtaining the target key corresponding to the data to be operated, the target key is used as the input of the target learning index to obtain the pointer output by the target learning index. If the target key is associated with only one data, the data in the storage location indicated by the pointer output by the target learning index is deleted. If the target key is associated with multiple candidate data, the candidate data that is closest to the data to be operated among the candidate data is deleted.

[0142] Optionally, before S702, an existence check can be performed through a Bloom filter to check whether the data to be operated exists in the memory. If the data to be operated exists in the memory, the read operation (query request) or write operation (insert request or update request) of the data to be operated is directly completed through the data to be operated-pointer mapping relationship in the hash map of the memory. If a write operation is involved, a corresponding pre-write log is generated and the pre-write log is persisted. If the data to be operated does not exist in the memory, execute S702 and S703.

[0143] In this embodiment, a short code vector is obtained by PQ encoding the data to be operated, which reduces the dimension of the data, can reduce the cost of learning index training and prediction, and improve prediction performance. Through the learning index model, the position of the key in the mapping unit can be quickly and accurately located without having to determine the position of the key in the mapping unit through traversal or binary search. The mapping unit stores the mapping relationship between the pointer corresponding to the key and the data. According to the position of the key, the pointer with the mapping relationship with the key can be obtained. The pointer directly or indirectly indicates the storage location of the data, so that the precise storage location of the data can be obtained. When the data is stored in the storage medium in the form of compressed data, the starting position information of each compressed data is recorded in the compression header, so that random reading of any compressed data can be supported without reading and decompressing all other unrelated data together, which can reduce write amplification and improve the reading performance of compressed data. In addition, in this embodiment, the data is compressed in segments according to the data structure, which can improve the compression rate of the data, further reduce the storage space occupied by data storage, and improve the utilization rate of the storage medium.

[0144] It should be noted that the above steps S701-S703 can all be executed by the host CPU on the host side. Optionally, the above step S702 can also be executed by an external computing device. Optionally, the operation of obtaining the pointer corresponding to the target key through the target learning index based on the target key in the above step S703 can also be executed by an external computing device to accelerate the query rate.

[0145] Optionally, as shown in FIG9 , when the data distribution differences between data groups meet a condition, the data set can be retrained. The data set is re-clustered to obtain K updated data groups, and the learning index corresponding to each updated data group is updated. The data distribution differences between data groups meeting the condition can mean that the difference in the number of elements in different data groups exceeds a threshold, or the mean square difference of the number of elements in different data groups exceeds a threshold.

[0146] In order to verify the technical effect of this solution, tests were conducted based on the solution provided in this application. The following are the test results in different scenarios.

[0147] Figure 10 shows a comparison of the query performance of this solution and existing solutions for a query of 1 million vectors in a structured data database storage scenario. It can be seen that query performance is improved by 6x-38x, and accuracy is also improved.

[0148] The query operation is completed collaboratively based on the GPU and CPU. Compared with the query operation completed only based on the CPU, the experimental results show that the acceleration throughput of index lookup using GPU V100 is 150x to 700x, and the acceleration throughput of index lookup using GPU A100 is 975x-2800x.

[0149] It can be seen that the use of learning-based indexes can optimize index storage overhead and search performance. The use of diversified computing power such as GPUs can further accelerate query throughput.

[0150] In the fingerprint table storage scenario, as shown in Figure 11, the overall compression effect on the product dataset reaches 2x. The average access latency of TinyKV based on the learning index is less than 6.27us, which is a 44.12% reduction in latency compared to LevelDB and an 84.68% reduction in latency compared to RocksDB.

[0151] We optimize index storage overhead and search performance by using a learned index. We also optimize storage performance by using similar value structured compression. We also optimize storage overhead by using structure-aware data compression methods.

[0152] In summary, at a scale of 1 million, average access latency is less than 6.27µs, a 44.12% reduction compared to LevelDB and an 84.68% reduction compared to RocksDB. This reduction is 78.5%-93.5% compared to LevelDB and 77.4%-94.2% compared to RocksDB. The longer the structured data, the greater the reduction rate with TinyKV. Tests on a GPU V100 show query throughput increases of 150x to 700x.

[0153] The following table shows compression tests of the structured compression algorithm proposed in this application based on similar structured data. Tested on a metadata dataset, compression ratios reached 4.5x–5.9x. Compression performance reached 994.5 MB / s on x86 and 503 MB / s on ARM. Decompression performance reached 1443.9 MB / s on x86 and 1158.8 MB / s on ARM, representing over 10x performance improvement over zstd.

[0154] Similar data partitioning solves the compression ratio issue. SIMD instructions address performance issues. Supporting random access metadata solves the read amplification problem. The compression method provided in this application offers fast performance, high compression ratios, and supports random access, avoiding performance bottlenecks caused by read amplification.

[0155] Based on the same inventive concept, the present application also provides a corresponding device embodiment. As shown in Figure 12, the present application provides a structural schematic diagram of a data processing device. The device 1200 includes: a transceiver module 1201, which is used to obtain an operation request, and the operation request includes an operation instruction and data to be operated; a processing module 1202, which is used to perform product quantization encoding on the target residual value or the data to be operated to obtain a target key, and the target residual value is the residual value between the data to be operated and the center vector of the target data group. The target data group is the data group to which the data to be operated belongs in K data groups, and K is an integer greater than or equal to 1; the processing module 1202 is used to execute the operation instruction according to the target learning index corresponding to the target key and the target data group.

[0156] In one possible implementation, the target learning index is used to output a pointer corresponding to the data to be operated based on the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit based on the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

[0157] In one possible implementation, the target learning index model includes a root model, at least two intermediate models and at least two leaf models, the root model is used to select a target intermediate model from at least two intermediate models according to a target key, the target intermediate model is used to select a target leaf model from at least two leaf models, the target leaf model is used to output a target position according to a target key, and the target leaf model is a nonlinear function.

[0158] In one possible implementation, at least part of the model in the learned index model is stored in a memory corresponding to an external computing device, and the external computing device is used to obtain a target intermediate model, a target leaf module or a target position based on at least part of the model.

[0159] In one possible implementation, the operation instruction is a query instruction, which instructs to read the target data corresponding to the data to be operated; the processing module 1202 is used to obtain a first pointer through a target learning index according to the target key, and the first pointer indicates the storage location of the target data; the processing module 1202 is used to obtain the target data according to the first pointer; wherein, when the first pointer is associated with one data, the target data is the data associated with the first pointer; when the first pointer is associated with multiple data, the target data is the data with the smallest distance to the data to be operated among the multiple data associated with the first pointer.

[0160] In one possible implementation, the target data is stored as compressed target data; the processing module 1202 is used to obtain the compressed target data according to the first pointer; the processing module 1202 is used to decompress the compressed target data through a decoder to obtain the target data; the target data is structured data, the target data includes N data segments divided according to attributes, the compressed target data includes N compressed data segments, the N compressed data segments are obtained by compressing the N data segments through M sub-encoders of the encoder, each sub-encoder is used to encode at least one data segment corresponding to an attribute, the decoder includes M sub-decoders, the M sub-decoders correspond one-to-one to the M sub-encoders, and each sub-decoder is used to decode at least one compressed data segment among the N compressed data segments.

[0161] In one possible implementation, the operation instruction is an update instruction; the processing module 1202 is used to write the data to be operated into the storage medium; the processing module 1202 is used to update the mapping relationship between the target key and the second pointer to the target learning index, and the second pointer indicates the storage location of the data to be operated in the storage medium.

[0162] In one possible implementation, the operation instruction is an insert instruction; the processing module 1202 is used to write the data to be operated into the storage medium; the processing module 1202 is used to insert the mapping relationship between the target key and the third pointer into the target learning index, and the third pointer indicates the storage location of the data to be operated in the storage medium.

[0163] In one possible implementation, the processing module 1202 is specifically used to encode and compress the data to be operated through the encoder corresponding to the target data group to obtain compressed data, where the data to be operated is structured data, and the data to be operated includes N data segments divided according to attributes, and the compressed data includes N compressed data segments, which are obtained by encoding and compressing the N data segments through M sub-encoders of the encoder, and each sub-encoder is used to encode at least one data segment corresponding to an attribute; the processing module 1202 is specifically used to write the compressed data into a storage medium.

[0164] In one possible implementation, the operation instruction is a delete instruction; the processing module 1202 is used to obtain a fourth pointer indicating the storage location of the target data corresponding to the data to be operated based on the target key through the target learning index; the processing module 1202 is used to delete the data in the storage location indicated by the fourth pointer; the processing module 1202 is used to delete the mapping relationship between the target key and the fourth pointer in the target learning index.

[0165] In a possible implementation, the apparatus further includes: when the data distribution in the K data groups meets a condition, re-clustering the data set to obtain K updated data groups, and updating the learning index corresponding to each updated data group.

[0166] As shown in Figure 13, the present application provides a structural schematic diagram of a data processing device. The device 1300 includes: a processing module 1301, which is used to obtain a target key, where the target key is obtained by quantizing and encoding the product of the data to be operated or the target residual value in the operation request, and the target residual value is the residual value between the center vector of the data to be operated and the target data group, and the target data group is the data group to which the data to be operated belongs among K data groups, where K is an integer greater than or equal to 1; the processing module 1301 is used to obtain a pointer corresponding to the target key based on the target key through the target learning index corresponding to the target data group, and the pointer indicates the storage location of the target data corresponding to the data to be operated.

[0167] In one possible implementation, the target learning index is used to output a pointer corresponding to the data to be operated based on the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit based on the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

[0168] The target learning index model learns the distribution pattern of the target key in the mapping unit, and the actual distribution of the data in the storage medium is decoupled from the target learning index model. Therefore, when writing data to the storage medium, there is no need to consider whether the writing of the data will affect the data distribution pattern in the storage medium. The data can be written to the storage medium using an append write method, which can improve write performance. In addition, when querying data, the target learning index model can quickly hit the location of the target key in the mapping unit, which can reduce the number of queries. The mapping unit stores the mapping relationship between the target key and the pointer, which can accurately obtain the storage location of the corresponding data, improving query accuracy and efficiency.

[0169] In one possible implementation, the target learning index model includes a root model, at least two intermediate models, and at least two leaf models. The root model is used to select a target intermediate model from the at least two intermediate models based on a target key. The target intermediate model is used to select a target leaf model from the at least two leaf models. The target leaf model is used to output a target position based on the target key. The target leaf model is a nonlinear function. The nonlinear function can better fit the distribution of keys, thereby more accurately predicting the key position.

[0170] In one possible implementation, the device also includes a storage module 1302, which is used to store the target learning index model or a partial model of the target learning index model, so that when querying, the corresponding pointer is obtained through the target learning index model according to the target key.

[0171] As shown in Figure 14, the present application provides a schematic diagram of the structure of a computing device. The computing device includes at least one processor 1401 and at least one memory 1402. The at least one memory 1402 stores computer-readable instructions. The at least one processor 1401 executes the computer-readable instructions to enable the computing device to perform the method of any of the above embodiments.

[0172] The present application provides a computer-readable storage medium comprising computer-readable instructions for implementing the method in any of the above embodiments.

[0173] The present application provides a computer program product, characterized in that it includes computer-readable instructions; the computer-readable instructions are used to implement the method in any of the above embodiments.

[0174] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0176] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining an operation request, wherein the operation request includes an operation instruction and data to be operated; Perform product quantization encoding on a target residual value or the data to be operated to obtain a target key, wherein the target residual value is a residual value between the data to be operated and a center vector of a target data group, wherein the target data group is a data group to which the data to be operated belongs among K data groups, and K is an integer greater than or equal to 1; The operation instruction is executed according to the target key and the target learning index corresponding to the target data group.

2. The method according to claim 1, characterized in that The target learning index is used to output the pointer corresponding to the data to be operated according to the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit according to the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

3. The method according to claim 2, characterized in that The target learning index model includes a root model, at least two intermediate models and at least two leaf models, the root model is used to select a target intermediate model from the at least two intermediate models according to the target key, the target intermediate model is used to select a target leaf model from the at least two leaf models, the target leaf model is used to output the target position according to the target key, and the target leaf model is a nonlinear function.

4. The method according to claim 2 or 3, characterized in that: At least part of the model in the learned index model is stored in a memory corresponding to an external computing device, and the external computing device is used to obtain the target intermediate model, the target leaf module or the target position based on the at least part of the model.

5. The method according to any one of claims 1 to 4, characterized in that The operation instruction is a query instruction, the query instruction instructs to read the target data corresponding to the data to be operated, and the operation instruction is executed according to the target key and the target learning index corresponding to the target data group, including: According to the target key, obtaining a first pointer through the target learned index, wherein the first pointer indicates a storage location of the target data; The target data is obtained according to the first pointer; wherein, when the first pointer is associated with one data, the target data is the data associated with the first pointer; when the first pointer is associated with multiple data, the target data is the data with the smallest distance from the data to be operated among the multiple data associated with the first pointer.

6. The method according to claim 5, characterized in that The target data is stored as compressed target data, and obtaining the target data according to the first pointer includes: Acquire the compressed target data according to the first pointer; The compressed target data is decompressed by a decoder to obtain the target data; the target data is structured data, the target data includes N data segments divided according to attributes, the compressed target data includes N compressed data segments, the N compressed data segments are obtained by compressing the N data segments by M sub-encoders of the encoder, each of the sub-encoders is used to encode at least one of the data segments corresponding to one of the attributes, the decoder includes M sub-decoders, the M sub-decoders correspond one-to-one to the M sub-encoders, each of the sub-decoders is used to decode at least one compressed data segment among the N compressed data segments, and both M and N are integers greater than or equal to 1.

7. The method according to any one of claims 1 to 4, characterized in that The operation instruction is an update instruction; the executing the operation instruction according to the target key and the target learning index corresponding to the target data group includes: Writing the data to be operated into a storage medium; A mapping relationship between the target key and a second pointer is updated to the target learning index, wherein the second pointer indicates a storage location of the data to be operated in the storage medium.

8. The method according to any one of claims 1 to 4, characterized in that The operation instruction is an insert instruction; and executing the operation instruction according to the target key and the target learning index corresponding to the target data group includes: Writing the data to be operated into a storage medium; A mapping relationship between the target key and the third pointer is inserted into the target learning index, and the third pointer indicates a storage location of the data to be operated in the storage medium.

9. The method according to claim 7 or 8, characterized in that: The step of writing the data to be operated into a storage medium comprises: The data to be operated is compressed by encoding through the encoder corresponding to the target data group to obtain compressed data, wherein the data to be operated is structured data, the data to be operated includes N data segments divided according to attributes, and the compressed data includes N compressed data segments, the N compressed data segments are obtained by encoding and compressing the N data segments through M sub-encoders of the encoder, each of the sub-encoders is used to encode at least one data segment corresponding to one of the attributes, and both M and N are integers greater than or equal to 1; The compressed data is written to the storage medium.

10. The method according to any one of claims 1 to 4, characterized in that The operation instruction is a delete instruction; and executing the operation instruction according to the target key and the target learning index corresponding to the target data group includes: According to the target key, through the target learning index, obtain a fourth pointer indicating a storage location of target data corresponding to the data to be operated; Deleting the data in the storage location indicated by the fourth pointer; The mapping relationship between the target key and the fourth pointer in the target learned index is deleted.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: When the data distribution in the K data groups meets the conditions, the data set is re-clustered to obtain the updated K data groups, and the learning index corresponding to each updated data group is updated.

12. A data processing method, characterized in that: The method comprises: Obtain a target key, where the target key is obtained by quantizing and encoding the product of the data to be operated or the target residual value in the operation request, where the target residual value is the residual value between the data to be operated and the center vector of the target data group, where the target data group is the data group to which the data to be operated belongs among K data groups, where K is an integer greater than or equal to 1; According to the target key, a pointer corresponding to the target key is obtained through a target learning index corresponding to the target data group, and the pointer indicates a storage location of the target data corresponding to the data to be operated.

13. A data processing device, characterized in that: The device comprises: A transceiver module, used for obtaining an operation request, wherein the operation request includes an operation instruction and data to be operated; a processing module, configured to perform product quantization encoding on a target residual value or the data to be operated to obtain a target key, wherein the target residual value is a residual value between the data to be operated and a center vector of a target data group, wherein the target data group is a data group to which the data to be operated belongs among K data groups, and K is an integer greater than or equal to 1; The processing module is used to execute the operation instruction according to the target key and the target learning index corresponding to the target data group.

14. The device according to claim 13, characterized in that The target learning index is used to output the pointer corresponding to the data to be operated according to the target key. The target learning index includes a learning index model and a mapping unit. The learning index model is used to output the target position of the target key in the mapping unit according to the target key. The target position stores the mapping relationship between the target key and the pointer corresponding to the data to be operated.

15. The device according to claim 14, characterized in that The target learning index model includes a root model, at least two intermediate models and at least two leaf models, the root model is used to select a target intermediate model from the at least two intermediate models according to the target key, the target intermediate model is used to select a target leaf model from the at least two leaf models, the target leaf model is used to output the target position according to the target key, and the target leaf model is a nonlinear function.

16. The device according to claim 14 or 15, characterized in that At least part of the model in the learned index model is stored in a memory corresponding to an external computing device, and the external computing device is used to obtain the target intermediate model, the target leaf module or the target position based on the at least part of the model.

17. The device according to any one of claims 13 to 16, characterized in that The operation instruction is a query instruction, and the query instruction instructs to read the target data corresponding to the data to be operated; The processing module is configured to obtain a first pointer through the target learning index according to the target key, wherein the first pointer indicates a storage location of the target data; The processing module is used to obtain the target data according to the first pointer; wherein, when the first pointer is associated with one data, the target data is the data associated with the first pointer; when the first pointer is associated with multiple data, the target data is the data with the shortest distance from the data to be operated among the multiple data associated with the first pointer.

18. The device according to claim 17, characterized in that The target data is stored in the form of compressed target data; The processing module is used to obtain the compressed target data according to the first pointer; The processing module is used to decompress the compressed target data through a decoder to obtain the target data; The target data is structured data, the target data includes N data segments divided according to attributes, the compressed target data includes N compressed data segments, the N compressed data segments are obtained by compressing the N data segments through M sub-encoders of the encoder, each of the sub-encoders is used to encode at least one of the data segments corresponding to the attribute, the decoder includes M sub-decoders, the M sub-decoders correspond one-to-one to the M sub-encoders, each of the sub-decoders is used to decode at least one compressed data segment among the N compressed data segments, and M and N are both integers greater than or equal to 1.

19. The device according to any one of claims 13 to 16, characterized in that The operation instruction is an update instruction; The processing module is used to write the data to be operated into a storage medium; The processing module is used to update the mapping relationship between the target key and the second pointer to the target learning index, and the second pointer indicates the storage location of the data to be operated in the storage medium.

20. The device according to any one of claims 13 to 16, characterized in that The operation instruction is an insert instruction; The processing module is used to write the data to be operated into a storage medium; The processing module is used to insert the mapping relationship between the target key and the third pointer into the target learning index, and the third pointer indicates the storage location of the data to be operated in the storage medium.

21. The device according to claim 19 or 20, characterized in that The processing module is specifically used to compress the data to be operated by encoding through the encoder corresponding to the target data group to obtain compressed data, the data to be operated is structured data, the data to be operated includes N data segments divided according to attributes, the compressed data includes N compressed data segments, the N compressed data segments are obtained by encoding and compressing the N data segments by M sub-encoders of the encoder, each of the sub-encoders is used to encode at least one data segment corresponding to one of the attributes, and both M and N are integers greater than or equal to 1; The processing module is specifically configured to write the compressed data into the storage medium.

22. The device according to any one of claims 13 to 16, characterized in that The operation instruction is a delete instruction; The processing module is used to obtain, according to the target key and through the target learning index, a fourth pointer indicating a storage location of target data corresponding to the data to be operated; The processing module is used to delete the data in the storage location indicated by the fourth pointer; The processing module is used to delete the mapping relationship between the target key and the fourth pointer in the target learning index.

23. The device according to any one of claims 13 to 22, characterized in that The device also includes: When the data distribution in the K data groups meets the conditions, the data set is re-clustered to obtain the updated K data groups, and the learning index corresponding to each updated data group is updated.

24. A data processing device, characterized in that: The device comprises: a processing module, configured to obtain a target key, wherein the target key is obtained by quantizing and encoding the product of the data to be operated or the target residual value in the operation request, wherein the target residual value is a residual value between the data to be operated and a center vector of the target data group, wherein the target data group is a data group to which the data to be operated belongs among K data groups, and K is an integer greater than or equal to 1; The processing module is used to obtain a pointer corresponding to the target key according to the target key through a target learning index corresponding to the target data group, wherein the pointer indicates a storage location of the target data corresponding to the data to be operated.

25. A computing device, characterized in that The computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device performs the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 12.

27. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 12.

Images