Search method, device, equipment and system

Abstract

A retrieval method, device, equipment and system are disclosed, relating to the field of artificial intelligence. The method comprises determining a plurality of cluster centers according to the features of a plurality of data contained in a data set, quantifying and encoding the plurality of data contained in the data set according to the plurality of cluster centers to obtain encoded data and a codebook indicating the plurality of cluster centers. The encoded data also retains the features of the data set, reducing the storage space occupied by the encoded data while ensuring data accuracy. After obtaining a retrieval request, the encoded data of the data set is decoded to obtain decoded data, the plurality of cluster centers corresponding to the decoded data are determined according to the mapping relationship between the decoded data and the codebook, ensuring the data accuracy of the restored data set. The plurality of cluster centers corresponding to the decoded data are retrieved according to the query data included in the retrieval request to obtain a retrieval result. Since the plurality of cluster centers contain the features of the data set, the retrieval in the plurality of cluster centers improves the retrieval accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a retrieval method, apparatus, device, and system. Background Technology

[0002] Currently, with the widespread application of big data technology, the scale of data is growing exponentially. This scale is primarily reflected in the volume and dimensions of the data. As the scale of data increases, data storage consumes a significant amount of storage space. Typically, to save storage space, data is converted to lower precision to reduce the data volume and thus decrease the occupied storage space. For example, high-precision floating-point numbers are converted to low-precision integers. However, retrieving target data from low-precision data results in low retrieval accuracy. Furthermore, even with a large data volume, low-precision data can still occupy a considerable amount of storage space. Summary of the Invention

[0003] This application provides a retrieval method, apparatus, device, and system that improves retrieval accuracy and reduces the storage space occupied by data.

[0004] Firstly, a retrieval method is provided, comprising: obtaining a retrieval request, the retrieval request including query data; decoding encoded data of a dataset to obtain decoded data; determining multiple cluster centers corresponding to the decoded data based on the mapping relationship between the decoded data and a codebook; retrieving the multiple cluster centers corresponding to the decoded data based on the query data; and obtaining retrieval results. The retrieval results include data corresponding to at least one cluster center. Specifically, the encoded data of the dataset is obtained by quantizing and encoding multiple data points contained in the dataset based on multiple cluster centers, and the codebook is used to indicate the multiple cluster centers.

[0005] The method provided in this application obtains multiple cluster centers based on the features of multiple data points in the dataset. By quantizing and encoding the data using these multiple cluster centers, the encoded data retains the features of the dataset. This reduces the storage space occupied by the encoded data while ensuring data accuracy. Upon receiving a retrieval request, the encoded data is decoded, and the multiple cluster centers corresponding to the decoded data are determined based on the mapping relationship between the decoded data and the codebook, thus ensuring the accuracy of the reconstructed dataset. The retrieval is then performed on the multiple cluster centers corresponding to the decoded data based on the query data included in the retrieval request, yielding the retrieval results. Because multiple cluster centers contain the features of the dataset, retrieval within these multiple cluster centers improves retrieval accuracy.

[0006] In one possible implementation, determining multiple cluster centers corresponding to the decoded data based on the mapping relationship between the decoded data and the codebook includes: determining multiple cluster centers corresponding to the decoded data from the codebook based on the bit correspondence between the decoded data and the codebook.

[0007] By using the digital correspondence to find multiple cluster centers corresponding to the decoded data in the codebook, we can ensure accurate data reconstruction and improve retrieval accuracy.

[0008] In another possible implementation, conditional constraints are used to indicate the relationship between multiple cluster centers. The multiple cluster centers corresponding to the decoded data are determined from the codebook based on the digit correspondence between the decoded data and the codebook. This includes: determining the data in the codebook corresponding to the decoded data based on the digit correspondence between the decoded data and the codebook, and determining the multiple cluster centers corresponding to the decoded data based on the conditional constraints and the data in the codebook corresponding to the decoded data.

[0009] Based on the constraints, the data in the codebook corresponding to the decoded data is restored to the cluster centers, ensuring accurate data restoration and improving retrieval accuracy.

[0010] In another possible implementation, multiple cluster centers corresponding to the decoded data are determined based on the mapping relationship between the decoded data and the codebook, including: determining multiple cluster centers corresponding to the decoded data based on the mapping relationship between the data of each dimension in the decoded data and the codebook.

[0011] By using smaller granular dimensions, the data in each dimension of the decoded data is restored to cluster centers with high similarity to those in that dimension, increasing the accuracy of the identified cluster centers and thus improving retrieval precision.

[0012] In another possible implementation, decoding the encoded data of the dataset to obtain the decoded data includes: performing binary decoding on the encoded data of the dataset to obtain the decoded data.

[0013] The encoded data is decoded into binary data, which makes it easier to select data bit by bit from the binary data and determine the cluster center based on the mapping relationship with the codebook.

[0014] In another possible implementation, the encoded data of the dataset is binary decoded to obtain intermediate data, and adjacent bits of the intermediate data are merged to obtain the decoded data.

[0015] By merging adjacent bits of the intermediate data obtained from binary decoding, and using the merged adjacent bits as the same dimension, it is easier to determine the cluster center from the codebook of the same dimension, making the determined cluster center more accurate and improving the retrieval accuracy.

[0016] Secondly, a data encoding method is provided. The method includes determining multiple cluster centers based on the characteristics of multiple data points contained in the dataset, quantizing the multiple data points contained in the dataset based on the multiple cluster centers to obtain quantized values ​​of the multiple data points, encoding the quantized values ​​of the multiple data points to obtain encoded data of the dataset. The encoded data of the dataset is used to restore the dataset based on a codebook during retrieval. The codebook is used to indicate the multiple cluster centers.

[0017] The method provided in this application obtains multiple cluster centers based on the features of multiple data points in the dataset. By quantizing and encoding the data according to these multiple cluster centers, the encoded data retains the features of the dataset. This reduces the storage space occupied by the encoded data while ensuring data accuracy.

[0018] In another possible implementation, the quantized values ​​of multiple data points are encoded to obtain the encoded data of the dataset, including: dividing the quantized values ​​of multiple data points into multiple groups of quantized values ​​based on the number of cluster centers, encoding the multiple groups of quantized values, and obtaining the encoded data of the dataset.

[0019] By encoding each quantized value in multiple quantized values, encoded data is obtained for each quantized value. The storage space occupied by the encoded data of each quantized value is the same as the storage space occupied by each quantized value in the quantized value, thus reducing the storage space occupied.

[0020] For example, each set of quantized values ​​contains k quantized values, and each quantized value in each set occupies n storage spaces, resulting in n*k storage spaces for the entire set. The encoded data for each set of quantized values ​​occupies n storage spaces. Compared to n*k storage spaces per set, the encoded data for each set of quantized values ​​occupies only 1 / k of the storage space occupied by each set of quantized values.

[0021] For example, a dataset contains multiple data points with dimension dim, number n, and a data size of n*dim. Assuming the number of cluster centers is k, the encoded data has dimension dim / k, a data size of n*dim / k, and the storage space occupied by the encoded data is 1 / k of the dataset size.

[0022] In another possible implementation, multiple cluster centers satisfy conditional constraints, which are used to indicate the relationship between the multiple cluster centers.

[0023] By setting conditional constraints on cluster centers, the decoded data can be restored to cluster centers based on these constraints during decoding, resulting in more accurate cluster centers and improved retrieval accuracy.

[0024] In another possible implementation, multiple data points in the dataset are quantized based on multiple cluster centers to obtain quantized values ​​for multiple data points. This includes determining the quantized value of each data point based on its similarity to multiple cluster centers, thereby obtaining quantized values ​​for multiple data points.

[0025] In another possible implementation, the quantization value of each data point is determined based on the similarity between each data point and multiple cluster centers, thus obtaining the quantization values ​​of multiple data points. This includes: determining the quantization value of each data point based on the similarity between the data in each dimension and the multiple cluster centers corresponding to each dimension, thus obtaining the quantization values ​​of multiple data points.

[0026] In another possible implementation, a codebook is generated based on the mapping relationship and multiple cluster centers.

[0027] By generating codebooks for multiple data entries, multiple cluster centers can be determined during retrieval based on the mapping relationship between the decoded data and the codebooks. This increases the accuracy of the determined cluster centers and improves retrieval precision.

[0028] In another possible implementation, the quantized values ​​of multiple data points are encoded to obtain the encoded data of the dataset, including: binary encoding of the quantized values ​​of multiple data points to obtain the encoded data of the dataset.

[0029] By binary encoding the quantized value of each data point, encoded data is obtained for each data point. The encoded data reduces the storage space occupied compared to the quantized value.

[0030] In another possible implementation, the dataset contains multiple data points that are residual data determined based on the original data in the dataset.

[0031] The residual data contains more local information than the original data, which reduces the information loss in the encoded data obtained from the residual data and improves the retrieval accuracy.

[0032] In another possible implementation, the original data of the dataset is clustered into two or more clusters, and the original data of the dataset is converted into residual data based on the relationship between the data in each cluster and the center of the cluster.

[0033] The data in each cluster contains local information features of the original data. The cluster center is representative data within its cluster, reflecting the characteristics of the data within the cluster. The residual data obtained by analyzing the relationship between the data in each cluster and the cluster center preserves the local information features contained in the original data, thus improving retrieval accuracy.

[0034] Thirdly, a retrieval apparatus is provided, comprising modules for performing the methods of the first aspect or any possible design of the first aspect. For example, the retrieval apparatus includes a communication module, a decoding module, and a retrieval module.

[0035] The communication module is used to acquire retrieval requests, which include query data. The decoding module decodes the encoded data of the dataset to obtain decoded data. Based on the mapping relationship between the decoded data and the codebook, it determines multiple cluster centers corresponding to the decoded data. The encoded data of the dataset is obtained by quantizing and encoding multiple data points contained in the dataset based on multiple cluster centers. The codebook is used to indicate the relationship between the multiple cluster centers. The retrieval module retrieves the data from the multiple cluster centers corresponding to the decoded data based on the query data, obtaining retrieval results. The retrieval results include data corresponding to at least one cluster center.

[0036] In one possible implementation, when the decoding module determines multiple cluster centers corresponding to the decoded data based on the mapping relationship between the decoded data and the codebook, it is specifically used to determine multiple cluster centers corresponding to the decoded data from the codebook based on the bit correspondence between the decoded data and the codebook.

[0037] In another possible implementation, when the decoding module determines multiple cluster centers corresponding to the decoded data from the codebook based on the bit correspondence between the decoded data and the codebook, it specifically determines the data in the codebook corresponding to the decoded data based on the bit correspondence between the decoded data and the codebook, and determines multiple cluster centers corresponding to the decoded data based on the conditional constraints and the data in the codebook corresponding to the decoded data. The conditional constraints are used to indicate the relationship between the multiple cluster centers.

[0038] In another possible implementation, when the decoding module determines multiple cluster centers based on the mapping relationship between the decoded data and the codebook, it is specifically used to determine multiple cluster centers corresponding to the decoded data based on the mapping relationship between the data of each dimension in the decoded data and the codebook.

[0039] In another possible implementation, when the decoding module decodes the encoded data of the dataset to obtain the decoded data, it is specifically used to perform binary decoding on the encoded data of the dataset to obtain the decoded data.

[0040] In another possible implementation, the decoding module performs binary decoding on the encoded data of the dataset to obtain intermediate data, and merges adjacent bits of the intermediate data to obtain the decoded data.

[0041] Fourthly, an encoding apparatus is provided, comprising modules for performing the methods of the second aspect or any possible design of the second aspect. For example, the encoding apparatus includes a quantization module and an encoding module.

[0042] The quantization module determines multiple cluster centers based on the characteristics of multiple data points within the dataset. It also quantizes the data points based on these cluster centers, obtaining quantized values ​​for each data point. The encoding module encodes these quantized values, producing encoded dataset data. This encoded data is used to reconstruct the dataset during retrieval based on a codebook, which indicates the multiple cluster centers.

[0043] In one possible implementation, when the encoding module encodes the quantized values ​​of multiple data points to obtain the encoded data of the dataset, it specifically divides the quantized values ​​of multiple data points into multiple groups of quantized values ​​based on the number of cluster centers, encodes the multiple groups of quantized values, and obtains the encoded data of the dataset.

[0044] In another possible implementation, multiple cluster centers satisfy conditional constraints, which are used to indicate the relationship between the multiple cluster centers.

[0045] In another possible implementation, the quantization module quantizes multiple data points in the dataset based on multiple cluster centers. When obtaining the quantized values ​​of multiple data points, it specifically determines the quantized value of each data point based on the similarity between each data point and multiple cluster centers, thus obtaining the quantized values ​​of multiple data points.

[0046] In another possible implementation, the quantization module determines the quantization value of each data point based on the similarity between each data point and multiple cluster centers. When obtaining the quantization values ​​of multiple data points, it is specifically used to determine the quantization value of each data point based on the similarity between the data in each dimension and the multiple cluster centers corresponding to each dimension, thus obtaining the quantization values ​​of multiple data points.

[0047] In another possible implementation, the encoding module is also used to generate a codebook based on the mapping relationship and multiple cluster centers.

[0048] In another possible implementation, the encoding module encodes the quantized values ​​of multiple data points to obtain the encoded data of the dataset. Specifically, it performs binary encoding of the quantized values ​​of multiple data points to obtain the encoded data of the dataset.

[0049] In another possible implementation, the dataset contains multiple data points that are residual data determined based on the original data in the dataset.

[0050] In another possible implementation, when the dataset contains multiple data points that are residual data determined from the original data in the dataset, the encoding module is specifically used to cluster the original data of the dataset into two or more clusters, and convert the original data of the dataset into residual data based on the relationship between the data in each cluster and the cluster center.

[0051] Fifthly, a retrieval system is provided, which executes the operational steps of the method in the first aspect or any possible implementation of the first aspect, decoding encoded data to obtain decoded data, determining multiple cluster centers corresponding to the decoded data based on the decoded data, and retrieving the multiple cluster centers corresponding to the decoded data to obtain retrieval results. The retrieval system is also used to execute the operational steps of the method in the second aspect or any possible implementation of the second aspect, determining multiple cluster centers based on the characteristics of multiple data points contained in the dataset, quantizing the multiple data points contained in the dataset based on the multiple cluster centers to obtain quantized values ​​of the multiple data points, and encoding the quantized values ​​of the multiple data points to obtain encoded data of the dataset.

[0052] In a sixth aspect, a computer device is provided, the computer device including a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the plurality of processors jointly execute the operation steps of the method in the first aspect or any possible implementation of the first aspect, or execute the operation steps of the method in the second aspect or any possible implementation of the second aspect.

[0053] A seventh aspect provides a computer-readable storage medium, comprising: computer software instructions; which, when executed in a processor, cause the processor to perform operational steps of the method as described in the first aspect or any possible implementation thereof. Alternatively, they may perform operational steps of the method as described in the second aspect or any possible implementation thereof.

[0054] Eighthly, a computer program product is provided that, when run on a computer, causes the computer to perform operational steps of the method as described in the first aspect or any possible implementation thereof, or to perform operational steps of the method as described in the second aspect or any possible implementation thereof.

[0055] The technical effects of any of the design methods in aspects two through eight can be found in aspect one or in different design methods in aspect one, and will not be repeated here.

[0056] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0057] Figure 1 A flowchart of a dimensionality reduction algorithm provided for existing technology, and a schematic diagram of the retrieval speed and recall rate of data with different precision;

[0058] Figure 2 A schematic diagram of the architecture of a retrieval system provided in this application;

[0059] Figure 3 A flowchart illustrating a data encoding method provided in this application;

[0060] Figure 4 This application provides a schematic diagram for determining cluster centers;

[0061] Figure 5 This application provides a schematic diagram for determining residual data;

[0062] Figure 6 A schematic diagram illustrating the encoding of quantized values ​​provided in this application;

[0063] Figure 7 A flowchart illustrating a retrieval method provided in this application;

[0064] Figure 8 This application provides a schematic diagram of reconstructing cluster centers;

[0065] Figure 9 A flowchart illustrating another retrieval method provided in this application;

[0066] Figure 10 A schematic diagram of an encoding and decoding process provided for this application;

[0067] Figure 11 A schematic diagram illustrating data encoding and decoding provided in this application;

[0068] Figure 12 A flowchart illustrating a method for determining cluster centers based on residual data provided in this application;

[0069] Figure 13 A schematic diagram of the structure of an encoding device provided in this application;

[0070] Figure 14 A schematic diagram of the structure of a retrieval device provided in this application;

[0071] Figure 15 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0072] To better understand the embodiments of this application, some terms or technologies involved in the embodiments of this application will be explained below.

[0073] Data compression refers to reducing data storage space or transmission bandwidth usage while minimizing information loss through specific algorithms and techniques. It has been widely applied in fields such as computers, communications, and multimedia. Data compression mainly includes dimensionality reduction and quantization.

[0074] Dimensionality reduction refers to the process of mapping high-dimensional data to a low-dimensional space. It reduces the number of dimensions in the data while retaining key information, thereby reducing storage space or computational costs when processing or analyzing the data.

[0075] In some embodiments, the Principal Components Analysis (PCA) algorithm is used to reduce the dimensionality of the data.

[0076] like Figure 1 Figure (a) shows a flowchart of a dimensionality reduction algorithm. The PCA algorithm is used as an example. The PCA algorithm includes the following steps: Data standardization (step 110). Calculate the mean and standard deviation of each dimension of the data to standardize the data for each dimension. Calculate the covariance matrix (step 120). Calculate the covariance matrix for the standardized data. The covariance matrix is ​​used to measure the linear correlation between different dimensions. Calculate eigenvalues ​​and eigenvectors (step 130). Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and their corresponding eigenvectors. Eigenvalues ​​represent the variance of the principal components, and eigenvectors represent the principal components. Select principal components (step 140). Sort the eigenvalues ​​in descending order and select the eigenvectors corresponding to the k largest eigenvalues. Project to a lower-dimensional space (step 150). Construct a projection matrix from the k eigenvectors, multiply the original data by the projection matrix, and the result is the dimensionality-reduced data.

[0077] However, when using the PCA algorithm to reduce the dimensionality of the original data, some information contained in the original data will be lost. For example, if the PCA algorithm reduces the original 256-dimensional data to 64-dimensional data, the retrieval accuracy will decrease because the information contained in the original data is lost in the dimensionality reduction data.

[0078] Retrieving data from dimensionality-reduced data results in a lower recall rate. Recall refers to the process of retrieving target data from dimensionality-reduced data. The actual recalled data refers to the data retrieved from the dimensionality-reduced data. Correctly recalled data refers to the data within the actually recalled data that belongs to the target data set. Recall rate is used to measure the effectiveness of the retrieval process. For example, recall rate is the number of correctly recalled data divided by the number of data actually recalled. The closer the recall rate is to 1, the more accurate the retrieval; the closer the recall rate is to 0, the less accurate the retrieval.

[0079] For example, if the actual number of data points recalled is 100 and the number of data points correctly recalled is 13, the recall rate is 13 / 100 = 0.13.

[0080] Table 1 shows the statistical relationship between actual recalled data and recall rate. As shown in Table 1, target data was obtained by searching the data after dimensionality reduction using the PCA algorithm. Assume the number of target data is 1. When the actual number of recalled data is 100, the recall rate is 0.13. When the actual number of recalled data is 500, the recall rate is 0.23. When the actual number of recalled data is 1000, the recall rate is 0.27. When the actual number of recalled data is 5000, the recall rate is 0.34.

[0081] Table 1

[0082]

[0083] Quantization is the process of converting continuous numerical values ​​into discrete data, which reduces the amount of data by reducing the number of bits. For example, continuous percentage values ​​(e.g., floating-point numbers from 0.0 to 100.0) can be converted into discrete level labels (e.g., 1, 2, 3, 4, 5 representing five intervals).

[0084] Alternatively, quantization also refers to the process of converting high-precision data into low-precision data, which reduces the amount of data by lowering the precision. Low-precision data includes, but is not limited to, int1, int2, int4, or int8. For example, converting a high-precision floating-point number (e.g., a 32-bit floating-point number) into low-precision data (e.g., an int8 type).

[0085] In some embodiments, data is converted to lower precision data using a quantization method, and the target data is retrieved from the lower precision data. For example, the data is quantized into data of type int8, int4, int2, or int1.

[0086] Data types int8 and int4 have higher precision, while int2 and int1 have lower precision. Even when quantizing data into int8 or int4, large datasets still consume significant storage space. While int8 or int4 data offers higher precision and recall, it requires more computational resources, resulting in slower retrieval speeds.

[0087] Quantizing data into int1 or int2 types occupies less storage space compared to int8 or int4 types. When retrieving target data from int2 or int1 data, the recall rate is lower due to the lower precision of int2 or int1 types. However, the retrieval consumes fewer computational resources and is faster.

[0088] Figure 1(b) in the diagram illustrates the retrieval speed and recall rate for data with different levels of precision. For example... Figure 1 As shown in (b), from int8 data to int1 data, the precision of the data gradually decreases, the retrieval speed gradually increases, and the recall rate gradually decreases.

[0089] To address the issue of low retrieval accuracy and significant storage space consumption caused by retrieving target data from low-precision data after it has been converted to low precision, this application provides a retrieval method. The method includes determining multiple cluster centers based on the characteristics of multiple data points in a dataset, quantizing and encoding the multiple data points to obtain encoded data, and a codebook indicating the multiple cluster centers. Since the multiple cluster centers are derived from the characteristics of multiple data points in the dataset, quantizing and encoding the data based on these cluster centers preserves the characteristics of the dataset. This reduces the storage space occupied by the encoded data while ensuring data accuracy. Upon receiving a retrieval request, the encoded data of the dataset is decoded to obtain decoded data. The mapping relationship between the decoded data and the codebook is used to determine the multiple cluster centers corresponding to the decoded data, thus ensuring the accuracy of the reconstructed dataset. The retrieval is performed on the multiple cluster centers corresponding to the decoded data based on the query data included in the retrieval request to obtain the retrieval results. Because the multiple cluster centers contain the characteristics of the dataset, retrieval within these multiple cluster centers improves retrieval accuracy.

[0090] The following is a detailed description of a retrieval method provided in this application, with reference to the accompanying drawings.

[0091] Figure 2 This is a schematic diagram of the architecture of a retrieval system provided in this application. Figure 2 As shown, the retrieval system 200 includes a client 210, a computing cluster 220, and a storage cluster 230.

[0092] The computing cluster 220 contains multiple computing nodes 221. For example, a computing node 221 is a computing device, such as a server.

[0093] In some embodiments, the computing cluster 220 is a heterogeneous computing architecture to provide high-performance computing. For example, computing nodes 221 include computing units with computing capabilities such as central processing units (CPUs), graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), and neural-network processing units (NPUs) to provide high-performance computing.

[0094] In this embodiment of the application, computing node 221 is used to obtain a retrieval request, decode the encoded data of the dataset to obtain decoded data, determine multiple cluster centers according to the mapping relationship between the decoded data and the codebook, and retrieve the multiple cluster centers corresponding to the decoded data according to the query data contained in the retrieval request to obtain retrieval results.

[0095] Optionally, computing node 221 is used to determine multiple cluster centers based on the characteristics of multiple data points contained in the dataset, quantize the multiple data points contained in the dataset based on the multiple cluster centers to obtain quantized values ​​of the multiple data points, and encode the quantized values ​​of the multiple data points to obtain the encoded data of the dataset.

[0096] Optionally, the retrieval and encoding processes can be performed by a single computing node or by multiple different computing nodes.

[0097] In other embodiments, multiple computing nodes 221 are connected via network devices (such as switches, network interface cards, etc.) based on high-speed interconnect technology, enabling communication between the multiple computing nodes 221.

[0098] Optionally, the computing cluster 220 also includes a control node 222. For example, the control node and the computing nodes are independent physical devices. Alternatively, the control node and multiple computing nodes may reside on the same physical device. The control node is the CPU. The multiple computing nodes include computing units such as GPUs, NPUs, and DPUs. The control node 222 manages and distributes retrieval requests, allowing multiple computing nodes to execute retrievals in parallel to improve retrieval efficiency.

[0099] Storage cluster 230 comprises multiple storage nodes 231. A storage node 231 includes one or more controllers, a network interface card (NIC), and multiple hard drives. Hard drives are used to store data. Hard drives are disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording (SMR) hard drives. The NIC is used to communicate with the compute nodes 221 included in compute cluster 220. The controller is used to write data to or read data from the hard drives based on read / write data requests sent by the compute nodes 221. During the read / write process, the controller needs to translate the address carried in the read / write data request into an address that the hard drive can recognize.

[0100] Optionally, storage cluster 230 is used to store the encoded data. Compute node 221 obtains the encoded data from storage cluster 230 to decode the encoded data and obtain the decoded data.

[0101] Optionally, storage cluster 230 is also used to store codebooks. Compute node 221 obtains the codebook from storage cluster 230 to determine multiple cluster centers based on the mapping relationship between the decoded data and the codebook.

[0102] In some embodiments, client 210 communicates with computing cluster 220 and storage cluster 230 via network 240. For example, client 210 sends a request to computing cluster 220 via network 240, requesting computing cluster 220 to perform a retrieval. Network 240 refers to an enterprise intranet (e.g., a local area network, LAN) or the Internet. Client 210 refers to a computer connected to network 240, also known as a workstation. Different clients share network resources (e.g., computing resources, storage resources).

[0103] In other embodiments, client 210 has client program 211 installed. Client 210 runs client program 211 and displays a user interface (UI). User 250 operates the user interface to submit a request. For example, user 250 operates the user interface to submit a search request. After receiving the search request, control node 222 sends an instruction message to at least one computing node, causing at least one computing node to search for multiple cluster centers corresponding to the decoded data and obtain the search results.

[0104] Optionally, the system administrator 260 can configure system information, such as setting the number of cluster centers, by calling the application platform interface (API) 212 or command-line interface (CLI) 213 through the client 210.

[0105] Figure 2 This is merely an illustrative diagram; the embodiments of this application do not limit the device connection method or the number of devices in the retrieval system. For example, the retrieval system includes multiple clients. One client connects to multiple computing nodes. Different clients establish connections with different computing nodes.

[0106] The data encoding method provided in this application will now be described in detail with reference to the accompanying drawings.

[0107] Figure 3 This is a flowchart illustrating a data encoding method provided in this application. Figure 3 As shown, the method includes the following steps.

[0108] Step 310: Determine multiple cluster centers based on the characteristics of the multiple data points contained in the dataset.

[0109] A dataset is a collection of data. In this application, a dataset is a collection containing vectorized data. The dataset contains multiple data items that are vector data.

[0110] Vectorization refers to the process of converting non-vectorized data into vector data. Vector data is easier for computers to process and analyze.

[0111] Optionally, a dataset is a collection containing non-vectorized data. Converting non-vectorized data into vectorized data yields multiple data points within the dataset. For example, non-vectorized data includes unstructured data such as text, images, audio, or video. Converting unstructured data such as text, images, audio, or video into vector data yields vector data.

[0112] Optionally, features are the attributes that the data possesses. For example, a data point indicates student data, which includes attributes such as age, height, and class. The vector data [15, 165, 3] represents a student who is 15 years old, 165cm tall, and in class 3.

[0113] Cluster centers are the core points used in clustering algorithms to represent each cluster; they are the representation of the data contained in the dataset within a specific feature space. Cluster centers reflect the distribution characteristics of the data within a cluster.

[0114] In some embodiments, a clustering algorithm is performed on multiple data points contained in the dataset to determine multiple cluster centers based on the characteristics of the multiple data points in the dataset. The clustering algorithm includes, but is not limited to, partition-based clustering algorithms, hierarchical clustering algorithms, density-based clustering algorithms, or probabilistic model-based clustering algorithms.

[0115] For example, by performing a clustering algorithm on the data in the dataset, multiple clusters are obtained, each containing at least one data point. Cluster centers are then determined for each cluster based on the at least one data point contained within it.

[0116] In the first implementation, the cluster center of each cluster is obtained by calculating the average of at least one data point within each cluster using mathematical methods. These mathematical methods include, but are not limited to, the mean, median, or weighted average. For example, the cluster center is obtained by calculating the average of at least one data point within each cluster.

[0117] In the second implementation, the cluster center is a single data point from at least one data point in a cluster. For example, the sum of similarities between each data point in each cluster and the remaining data points in that cluster is calculated, and the data point with the highest similarity is designated as the cluster center for that cluster.

[0118] Optionally, the number of cluster centers determined based on the characteristics of multiple data points in the dataset satisfies the first condition. For example, the first condition is that the number of cluster centers is a power of 2, where n is a positive integer. For instance, when n is 1, the number of cluster centers is 2. When n is 2, the number of cluster centers is 4. When n is 3, the number of cluster centers is 8.

[0119] Optionally, the number of cluster centers is set by the user. Alternatively, the number of cluster centers is determined by the clustering algorithm when it is executed.

[0120] Vector data can contain one-dimensional or multi-dimensional data. Optionally, each dimension contains vector data representing a feature. Different dimensions may contain data representing different or the same features.

[0121] For example, vector data represents clothing features. Vector data has three dimensions: the first dimension contains data representing size features, the second dimension contains data representing color features, and the third dimension contains data representing fabric features.

[0122] In some embodiments, the data in each dimension is clustered based on the data features of each dimension of the multiple data sets to determine multiple cluster centers. These multiple cluster centers include the cluster centers corresponding to each dimension. The number of cluster centers corresponding to each dimension may be the same or different.

[0123] For example, by performing a clustering algorithm on each dimension of multiple data points, multiple clusters are obtained for each dimension. Each cluster contains data from its respective dimension. Based on the data contained in each cluster within each dimension, a cluster center is determined for each cluster. Each cluster contains at least one cluster center.

[0124] In the first implementation, mathematical methods are used to calculate the cluster centers for the data in each dimension contained within each cluster. For example, the cluster centers for each cluster are obtained by calculating the average value of the data in each dimension contained within each cluster.

[0125] In the second implementation, the cluster center is a single data point within the dimension of a cluster. For example, the similarity between the data points in the dimension of a cluster and the remaining data points in that cluster is calculated, and the data point with the highest similarity is designated as the cluster center.

[0126] Combination Figure 4 Explain how to perform a clustering algorithm on each dimension of multiple data points to determine the cluster centers for each dimension.

[0127] Figure 4 This is a schematic diagram illustrating a method for determining cluster centers provided in this application. As shown in the figure, the dataset contains vector data V1-V6, where any vector data in V1-V6 has a dimension of 4. The data in the first dimension includes 1, 2, 3, 8, 9, and 10; the data in the second dimension includes 3, 4, 5, 6, 7, and 8; the data in the third dimension includes 2, 3, 4, 7, 8, and 9; and the data in the fourth dimension includes 4, 5, 6, 8, 8, and 7.

[0128] Clustering was performed on the data for each dimension, resulting in multiple cluster centers. These cluster centers comprised the cluster centers for the first through fourth dimensions. The cluster centers for the first dimension were 1, 3, 8, and 10. The cluster centers for the second dimension were 4, 5, 7, and 8. The cluster centers for the third dimension were 3, 4, 7, and 8. The cluster centers for the fourth dimension were 4, 5, 6, and 7.

[0129] In some embodiments, when n is 2 or more, i.e., when the number of cluster centers is 4 or more, multiple cluster centers satisfy the conditional constraint. The conditional constraint is used to indicate the relationship between multiple cluster centers. In the first implementation, when n is 2, the number of cluster centers is 4, and the conditional constraint is that the sum of two of the four cluster centers equals the sum of the other two cluster centers.

[0130] For example, the four cluster centers are w, x, y, and z, and the constraint that the four cluster centers need to satisfy is x + y = w + z.

[0131] In the second implementation, when n is greater than 2, four cluster centers are selected from multiple cluster centers, and condition constraints are set for each of the four selected cluster centers. The condition constraints are that the sum of two of the four cluster centers is equal to the sum of the other two cluster centers.

[0132] For example, when n is 3, we get 8 cluster centers: a, b, c, d, w, x, y, and z. We randomly select 4 cluster centers (a, b, c, d) from these 8, where a + b = c + d is satisfied. The other 4 cluster centers (w, x, y, z) also need to satisfy the constraint x + y = w + z.

[0133] Optionally, the dataset may contain multiple data points that are residual data determined from the original data in the dataset.

[0134] In some embodiments, the original data contained in the dataset is clustered to obtain two or more clusters, each cluster containing at least one original data record and each cluster containing a center. Based on the relationship between the original data in each cluster and the center of that cluster, the original data of the dataset is transformed into residual data.

[0135] For example, a clustering algorithm can be used to cluster multiple original data points in a dataset, resulting in two or more clusters and the center of each cluster. The difference between the data points in each cluster and the center of that cluster is then calculated to obtain the residual data.

[0136] Combination Figure 5 Explain the process of converting the original data of a dataset into residual data.

[0137] Figure 5 This is a schematic diagram illustrating how residual data is determined, as provided in this application. Figure 5 As shown, V1-V6 are the original data. A clustering algorithm is performed on the original data to obtain the first cluster and the second cluster. The first cluster contains data V1, V2, and V3, and the second cluster contains data V4, V5, and V6. The center of the first cluster is V2[2, 4, 3, 5], and the center of the second cluster is V5[9, 7, 8, 8].

[0138] Calculate the difference between the data contained in each cluster and the center of that cluster to obtain the residual data. For example, data V6 belongs to the second cluster, and the center of the second cluster is V5. Calculate the difference between V6[10, 8, 9, 7] and V5[9, 7, 8, 8] to convert data V6 into residual data [1, 1, 1, -1].

[0139] Optionally, a subset of data can be selected from the dataset, and multiple cluster centers can be determined based on the characteristics of this subset. The more subset of data selected, the more accurately the determined cluster centers will represent the data information. Selection methods include, but are not limited to, random sampling.

[0140] Step 320: Quantize multiple data points in the dataset based on multiple cluster centers to obtain the quantized values ​​of the multiple data points.

[0141] In some embodiments, the cluster center to which each data point in the dataset belongs is determined, and each data point is quantized into data associated with its respective cluster center, resulting in quantized values ​​for the multiple data points. For example, the quantized values ​​are the identifiers of the cluster centers. For instance, four cluster centers are w, x, y, and z. The identifier for cluster center w is 0, the identifier for cluster center x is 1, the identifier for cluster center y is 2, and the identifier for cluster center z is 3. The quantized values ​​for the multiple data points include the identifiers of the four cluster centers.

[0142] In some embodiments, the cluster center to which each data point belongs is determined based on the similarity between each data point and multiple cluster centers, and each data point is quantized into data related to its respective cluster center to obtain quantized values ​​for multiple data points.

[0143] For example, the similarity between each data point and each of the multiple cluster centers can be calculated separately. Each data point can then be quantified as the identifier of the cluster center with the highest similarity to it, thus determining the quantized value of each data point and obtaining the quantized values ​​of multiple data points. Similarity calculation methods include, but are not limited to, Euclidean distance, cosine similarity, or inner product distance.

[0144] For example, the dataset contains data V1[1, 3, 2, 4], two cluster centers a and b, cluster center a[1, 2, 3, 4] is labeled 0, and cluster center b[3, 5, 7, 8] is labeled 1. V1 is more similar to cluster center a than to cluster center b, therefore, V1[1, 3, 2, 4] is quantized to the label 0 of cluster center a.

[0145] In some embodiments, the cluster center to which the data of each dimension belongs is determined based on the similarity between the data of each dimension in each data and the multiple cluster centers corresponding to each dimension. The data of each dimension is then quantized into data related to the cluster center to which it belongs, resulting in quantized values ​​for multiple data.

[0146] For example, calculate the similarity between each dimension of data in each data point and the multiple cluster centers corresponding to each dimension, quantize each dimension of data in each data point into the identifier of the cluster center with the highest similarity to the data in that dimension, determine the quantization value of each data point, and obtain the quantization values ​​of multiple data points.

[0147] For example, the dataset contains vector data V1-V6. Vector data V1[1, 5, 5, 8] has a dimension of 4.

[0148] The four cluster centers in the first dimension are 1, 3, 8, and 10, with corresponding labels of 0, 1, 2, and 3, respectively.

[0149] The four cluster centers in the second dimension are 4, 5, 7, and 8, with corresponding labels of 0, 1, 2, and 3, respectively.

[0150] The four cluster centers in the third dimension are 3, 4, 7, and 8, with corresponding labels of 0, 1, 2, and 3, respectively.

[0151] The four cluster centers in the fourth dimension are 4, 5, 6, and 7, with corresponding labels of 0, 1, 2, and 3, respectively.

[0152] The data for the first dimension of V1 is 1. It has the highest similarity to cluster center 1 among the four cluster centers in the first dimension. Therefore, the data 1 in the first dimension of V1 is quantized to the identifier 0 of cluster center 1.

[0153] The data for the second dimension of V1 is 5. It has the highest similarity to the four cluster centers 4, 5, 7, and 8 in the second dimension. Therefore, the data 5 in the second dimension of V1 is quantized to the identifier 1 of cluster center 5.

[0154] The data for the third dimension of V1 is 5. It has the highest similarity to cluster center 4 among the four cluster centers 3, 4, 7 and 8 in the third dimension. Therefore, the data 5 in the third dimension of V1 is quantized to the identifier 1 of cluster center 4.

[0155] The data for the fourth dimension of V1 is 8. It has the highest similarity to 7 among the four cluster centers 4, 5, 6 and 7 in the fourth dimension. Therefore, the data 8 in the fourth dimension of V1 is quantized to the identifier 3 of cluster center 7.

[0156] Therefore, the quantization value of V1[1, 5, 5, 8] is [0, 1, 1, 3].

[0157] Step 330: Encode the quantized values ​​of multiple data points to obtain the encoded data of the dataset.

[0158] In some embodiments, the quantized value of each data point in the multiple data points is binary encoded to obtain the encoded data of the dataset.

[0159] For example, such as Figure 6 As shown in (a), L1-L4 are the quantized values ​​of four data points. Binary encoding of L1 is calculated as 1*2^0 + 1*2^2 + 0*2^4 + 2*2^6, resulting in the encoded data 133. Binary encoding of L2 is calculated as 0*2^1 + 1*2^2 + 1*2^4 + 2*2^6, resulting in the encoded data 148. Binary encoding of L3 is calculated as 3*2^0 + 0*2^2 + 1*2^4 + 2*2^6, resulting in the encoded data 147. Binary encoding of L4 is calculated as 1*2^0 + 1*2^2 + 0*2^4 + 2*2^6, resulting in the encoded data 97.

[0160] In some embodiments, the quantized values ​​of multiple data points are divided into multiple groups of quantized values ​​based on the number of cluster centers, and the multiple groups of quantized values ​​are encoded to obtain the encoded data of the dataset.

[0161] For example, the quantized values ​​of multiple data points can be divided into multiple groups, with the number of quantized values ​​in each group matching the number of cluster centers. Each group of quantized values ​​is then binary-encoded to obtain the encoded data of the dataset.

[0162] In the first implementation, the number of cluster centers is less than the dimension of the data. Each data point's quantized value is divided into multiple groups of quantized values, and each group is binary-encoded. The number of quantized values ​​in each group is the same as the number of cluster centers.

[0163] For example, such as Figure 6 As shown in (b), L5 is the quantized value of a single data point in the dataset, with a dimension of 8. Assuming there are 4 cluster centers, L5 is divided into two groups of quantized values, with the number of cluster centers as the unit. The first group of quantized values ​​is [2, 2, 1, 2], and the second group of quantized values ​​is [0, 3, 1, 3]. The dimensions of the first and second groups of quantized values ​​are the same as the number of cluster centers.

[0164] The first set of quantized values ​​is encoded into binary using the formula 2*2^0 + 2*2^2 + 1*2^4 + 2*2^6, resulting in the encoded data 154. The second set of quantized values ​​is encoded into binary using the formula 0*2^0 + 3*2^2 + 1*2^4 + 3*2^6, resulting in the encoded data 220.

[0165] In the second implementation, the number of cluster centers is equal to the dimension of the data, and the quantized value of each data point is grouped together and binary encoded.

[0166] For example, the number of cluster centers is 4. A data point in the dataset has a quantized value of [0, 1, 1, 3] and a dimension of 4. These quantized values ​​are grouped together and encoded using binary encoding: 0*2^0 + 1*2^2 + 1*2^4 + 3*2^6, resulting in the data 2^12.

[0167] In some embodiments, after obtaining the encoded data, the encoded data is stored.

[0168] For example, the encoded data can be stored in a storage medium, which may include, but is not limited to, memory and hard disk.

[0169] For example, the encoded data can be stored in a processor. Processors include, but are not limited to, graphics processing units (GPUs) and central processing units (CPUs). For instance, the encoded data can be stored in the GPU's video memory or the CPU's cache.

[0170] A processor operator is a basic unit that performs a specific computational task within a processor. Operators are classified into various types, the most common of which include: arithmetic operators, nonlinear operators, convolution operators, pooling operators, and fusion operators.

[0171] Arithmetic operators are used for basic mathematical operations such as addition, subtraction, multiplication, and division. Nonlinear operators are used to introduce nonlinear characteristics; these include activation functions such as ReLU, Sigmoid, and Tanh. Convolution operators are used for feature extraction in Convolutional Neural Networks (CNNs). Pooling operators are used to reduce the spatial resolution of feature maps and decrease computational complexity; examples include max pooling and average pooling. Fusion operators combine multiple smaller operators into a single larger operator to reduce memory read / write operations and computational complexity, thereby improving inference efficiency.

[0172] In some embodiments, a codebook is generated based on multiple cluster centers and mapping relationships, so that the encoded data of the dataset can be restored to multiple cluster centers based on the codebook. The mapping relationship refers to the bit correspondence between the decoded data and the codebook determined by the processor's operators.

[0173] For example, the bit correspondence between the decoded data and the codebook determined by the processor's operators includes: selecting the data corresponding to the bit position of the data 1 in the decoded data in the first codebook, and selecting the data corresponding to the bit position of the data 0 in the second codebook based on the bit position of the data 0 in the decoded data.

[0174] For example, suppose the first codebook includes [a, b] and the second codebook includes [c, d], where a, b, c, and d are parameters. The decoded data includes four cases: 00, 01, 10, and 11. Data 00 corresponds to cluster center w, data 01 corresponds to cluster center x, data 10 corresponds to cluster center y, and data 11 corresponds to cluster center z.

[0175] In data 00, the data 0 corresponds to the second codebook. The first bit in data 00 corresponds to the first bit in the second codebook, and the second bit in data 00 corresponds to the second bit in the second codebook. Therefore, the data 0 in the first bit of data 00 corresponds to the parameter c of the first bit in the second codebook, and the data 0 in the second bit of data 00 corresponds to the parameter d of the second bit in the second codebook, resulting in c + d = w.

[0176] In data 01, data 0 corresponds to the second codebook, and data 1 corresponds to the first codebook. The first bit of data 01 corresponds to the first bit of the second codebook, and the second bit of data 01 corresponds to the second bit of the first codebook. Therefore, the data 0 in the first bit of data 01 corresponds to the parameter c of the first bit of the second codebook, and the data 1 in the second bit of data 01 corresponds to the parameter b of the second bit of the first codebook. Thus, c + b = x.

[0177] In data 10, data 1 corresponds to the first codebook, and data 0 corresponds to the second codebook. The first bit in data 10 corresponds to the first bit in the first codebook, and the second bit in data 10 corresponds to the second bit in the second codebook. Therefore, data 1 in the first bit of data 10 corresponds to parameter 'a' in the first codebook, and data 0 in the second bit of data 10 corresponds to parameter 'd' in the second codebook. Thus, a + d = y.

[0178] In data 11, data 1 corresponds to the first codebook; the first bit in data 11 corresponds to the first bit in the first codebook; and the second bit in data 11 corresponds to the second bit in the first codebook. Therefore, the data 1 in the first bit of data 11 corresponds to parameter 'a' in the first codebook, and the data 1 in the second bit of data 11 corresponds to parameter 'b' in the second bit of the first codebook, resulting in a + b = z.

[0179] Let d = 0. According to c + d = w, we get c = w. According to a + d = y, we get a = y.

[0180] Rearranging a + b = z, we get b = za. Since d = 0 and a = y, therefore b = zy.

[0181] Therefore, the first codebook generated based on the bit correspondence between the decoded data and the codebook determined by the processor's operators is [y, zy], and the second codebook is [w, 0].

[0182] Different processors use different operators, and the corresponding bitwise relationships between the decoded data and the codebook may differ for different processors. Furthermore, this application does not limit the number of codebooks generated.

[0183] In some embodiments, multiple codebooks are generated for multiple data entries contained in the dataset, including a first codebook and a second codebook.

[0184] In the first implementation, a first codebook and a second codebook are generated for all dimensions of the multiple data points contained in the dataset. The first codebook contains two cluster centers for all dimensions of the data, and the second codebook contains two additional cluster centers for all dimensions of the data.

[0185] For example, when n is 2, the number of cluster centers for each dimension is 4. The four cluster centers for the first dimension are w, x, y, z; the four cluster centers for the second dimension are a, b, c, d; and the four cluster centers for the third dimension are e, f, g, h. The first codebook for all dimensions is [zy, y, dc, c, hg, g]. The second codebook for all dimensions is [0, w, 0, a, 0, e].

[0186] For example, when n is greater than 2, the number of cluster centers in each dimension is greater than 4. Generate a first codebook and a second codebook for the 4 cluster centers in all dimensions, and generate a first codebook and a second codebook for the remaining 4 cluster centers in all dimensions.

[0187] For example, the eight cluster centers in the first dimension are a, b, c, d, e, f, g, and h, and the eight cluster centers in the second dimension are m, n, o, p, w, x, y, and z.

[0188] Generate a first codebook [dc, c, po, o] and a second codebook [0, a, 0, m] for the four cluster centers a, b, c, d in the first dimension and the four cluster centers m, n, o, p in the second dimension.

[0189] Generate a first codebook [hg, g, zy, y] and a second codebook [0, e, 0, w] for the remaining 4 cluster centers e, f, g, h in the first dimension and the remaining 4 cluster centers w, x, y, z in the second dimension.

[0190] In the second implementation, based on the cluster centers of each dimension, a first codebook and a second codebook are generated for each dimension of the multiple data points contained in the dataset. The first codebook contains the two cluster centers of the dimension, and the second codebook contains the other two cluster centers of the dimension.

[0191] For example, when n is 2, the number of cluster centers in each dimension is 4. The four cluster centers for the first dimension are w, x, y, and z, and the four cluster centers for the second dimension are a, b, c, and d.

[0192] Generate codebooks for the four cluster centers of the first dimension data. The first codebook is [zy, y] and the second codebook is [0, w].

[0193] Generate codebooks for the four cluster centers of the second dimension data. The first codebook is [dc, c] and the second codebook is [0, a].

[0194] For example, when n is greater than 2, the number of cluster centers in each dimension is greater than 4. A first codebook and a second codebook are generated for the 4 cluster centers in each dimension, and a first codebook and a second codebook are generated for the remaining 4 cluster centers in each dimension.

[0195] For example, the 8 cluster centers in the first dimension are a, b, c, d, w, x, y, z. Generate a first codebook [dc, c] and a second codebook [0, a] for the 4 cluster centers a, b, c, d in the first dimension. Generate a first codebook [zy, y] and a second codebook [0, w] for the remaining 4 cluster centers in the first dimension.

[0196] In some embodiments, a codebook is generated for multiple data entries contained in the dataset.

[0197] In the first implementation, the first half of the codebook includes two cluster centers of all dimensional data, and the second half of the codebook includes the other two cluster centers of all dimensional data.

[0198] For example, when n is 2, the number of cluster centers for each dimension is 4. The four cluster centers for the first dimension are w, x, y, z; the four cluster centers for the second dimension are a, b, c, d; and the four cluster centers for the third dimension are e, f, g, h. Generate a codebook [zy, y, dc, c, hg, g, 0, w, 0, a, 0, e] for all dimensions.

[0199] The first half of the codebook [zy, y, dc, c, hg, g, 0, w, 0, a, 0, e] consists of [zy, y, dc, c, hg, g], which is composed of two cluster centers from all dimensions of the data. The second half of the codebook consists of [0, w, 0, a, 0, e], which is composed of the other two cluster centers from all dimensions of the data.

[0200] In the second implementation, the codebook is composed of cluster centers of data in each dimension arranged sequentially.

[0201] For example, when n is 2, the number of cluster centers for each dimension is 4. The four cluster centers for the first dimension are w, x, y, z; the four cluster centers for the second dimension are a, b, c, d; and the four cluster centers for the third dimension are e, f, g, h. Generate a codebook [zy, y, 0, w, dc, c, 0, a, hg, g, 0, e] for all dimensions.

[0202] The codebook [zy, y, 0, w, dc, c, 0, a, hg, g, 0, e] includes four cluster centers in the first dimension ([zy, y, 0, w]), four cluster centers in the second dimension ([dc, c, 0, a]), and four cluster centers in the third dimension ([hg, g, 0, e]). By determining the cluster centers of multiple data points in the dataset, these data points are converted into quantized values, and then encoded to obtain the encoded dataset. The encoded dataset has a smaller data size compared to the original dataset containing multiple data points, thus reducing storage space usage. A computing card with 48GB of memory can store 150 million data points in the dataset. After encoding the dataset using the data encoding method provided in this application, the storage capacity of the computing card with 48GB of memory increases to 400 million data points.

[0203] For example, a dataset contains multiple data points with dimension dim, number n, and size n*dim. Assuming the number of cluster centers is k, the encoded dataset has dimension dim / k, size n*dim / k, and occupies 1 / k of the storage space of the original dataset.

[0204] Next, the retrieval method will be explained in detail with reference to the attached diagram.

[0205] Figure 7 This is a flowchart illustrating a retrieval method provided in this application. Figure 7 As shown, the method includes the following steps.

[0206] Step 710: Obtain the search request.

[0207] In some embodiments, a retrieval request is received from a client. The retrieval request contains query data. For example, the query data includes unstructured data such as text, images, audio, or video.

[0208] In other embodiments, a retrieval request is retrieved from a storage medium, and the retrieval request contains query data. For example, the retrieval request is a historical retrieval request.

[0209] Optionally, the query data contained in the retrieval request can be converted into vector data. For example, image data can be converted into vector data using a convolutional neural network.

[0210] Step 720: Decode the encoded data of the dataset to obtain the decoded data, and determine the cluster center corresponding to the decoded data based on the mapping relationship between the decoded data and the codebook.

[0211] Upon receiving a retrieval request, the encoded data of the dataset is obtained. Optionally, the encoded data can be retrieved from memory or the processor's cache.

[0212] In some embodiments, the encoded data of the dataset is binary decoded to obtain the decoded data.

[0213] For example, the encoded data of each data point in the dataset is binary-decoded to obtain the decoded data.

[0214] For example, the dataset contains first data and second data. The encoded data of the first data is 154, and the encoded data of the second data is 220. Decoding the encoded data of the first data into binary yields the decoded data 10011010. Decoding the encoded data of the second data into binary yields the decoded data 11011100.

[0215] In some embodiments, the encoded data of the dataset is binary decoded to obtain intermediate data, the intermediate data is arranged from low to high bits, and adjacent bits are merged to obtain the decoded data.

[0216] For example, the encoded data of each data point in the dataset is binary-decoded to obtain the intermediate data for each data point. The intermediate data for each data point is then arranged from least significant bit to most significant bit, and adjacent bits are merged to obtain the decoded data.

[0217] For example, the dataset contains the first data, which is encoded as 154. The encoded data of the first data is binary decoded to obtain the intermediate data 10011010. The intermediate data is then arranged from least significant bit to most significant bit, resulting in 01011001. Adjacent bits are merged to obtain the decoded data [01, 01, 10, 01].

[0218] In some embodiments, multiple cluster centers corresponding to the decoded data are determined from the codebook based on the bit correspondence between the decoded data and the codebook.

[0219] The correspondence between the decoded data and the codebook refers to the relationship between the digits in the decoded data and the digits in the codebook. Multiple cluster centers corresponding to the decoded data are determined based on the corresponding digits in the codebook.

[0220] In one implementation, multiple cluster centers corresponding to the decoded data are determined from the codebook based on the correspondence between the data of each dimension in the decoded data and the digits in the codebook.

[0221] For example, the first dimension in the decoded data corresponds to the first bit in the codebook, and the cluster center of the first bit in the codebook is determined as the cluster center corresponding to the first dimension of the decoded data. The second dimension in the decoded data corresponds to the second bit in the codebook, and the cluster center of the second bit in the codebook is determined as the cluster center corresponding to the second dimension of the decoded data.

[0222] For example, when each dimension corresponds to two codebooks, data 0 in each dimension of the decoded data corresponds to the second codebook, and data 1 in each dimension of the decoded data corresponds to the first codebook. Based on the bits of data 1 in each dimension of the decoded data, determine the cluster center corresponding to the corresponding bit in the first codebook for that dimension. Based on the bits of data 0 in each dimension of the decoded data, determine the cluster center corresponding to the corresponding bit in the second codebook for that dimension. Summing the cluster centers selected from the first and second codebooks yields the cluster center of the decoded data in that dimension.

[0223] Combination Figure 8 (a) in the text explains that the cluster center is determined based on the correspondence between the data in each dimension of the decoded data and the digits in the codebook.

[0224] Figure 8 This is a schematic diagram illustrating the reconstruction of cluster centers provided in this application. Figure 8 As shown in (a), the decoded data includes [10, 01, 10, 01]. The first dimension of the data includes 10, the second dimension includes 01, the third dimension includes 10, and the fourth dimension includes 01.

[0225] The first codebook of the first dimension is [1, 5], and the second codebook of the first dimension is [0, 1].

[0226] In the first dimension of data 10, data 1 corresponds to the first codebook of the first dimension, data 0 corresponds to the second codebook of the first dimension, the first bit in data 10 corresponds to the first bit in the first codebook of the first dimension, and the second bit in data 10 corresponds to the second bit in the second codebook of the first dimension.

[0227] The data 1 in the first bit of data 10 corresponds to the data 1 in the first bit of the first codebook in the first dimension, and the data 0 in the second bit of data 10 corresponds to the data 1 in the second bit of the second codebook in the first dimension.

[0228] Summing the cluster centers obtained from the first and second codebooks in the first dimension, we find that the cluster center of the decoded data in the first dimension is 2.

[0229] The first codebook of the fourth dimension is [2, 4], and the second codebook of the fourth dimension is [0, 2].

[0230] In the fourth dimension, data 0 corresponds to the second codebook of the fourth dimension, data 1 corresponds to the first codebook of the fourth dimension, the first bit in data 01 corresponds to the first bit in the second codebook of the fourth dimension, and the second bit in data 01 corresponds to the second bit in the first codebook of the fourth dimension.

[0231] The data 0 in the first bit of data 01 corresponds to the data 0 in the first bit of the second codebook in the fourth dimension, and the data 1 in the second bit of data 01 corresponds to the data 4 in the second bit of the first codebook in the fourth dimension.

[0232] Summing the cluster centers obtained from the first and second codebooks in the fourth dimension, we get a cluster center of 4 for the decoded data in the fourth dimension.

[0233] Summing the cluster centers obtained from the first and second codebooks in the fourth dimension, we get a cluster center of 4 for the decoded data in the fourth dimension.

[0234] In some embodiments, the data in the codebook corresponding to the decoded data is determined based on the bit correspondence between the decoded data and the codebook, and multiple cluster centers corresponding to the decoded data are determined based on the condition constraints and the data in the codebook corresponding to the decoded data.

[0235] In one implementation, the data in the codebook corresponding to the decoded data is determined based on the digit correspondence between the data in each dimension of the decoded data and the codebook. Multiple cluster centers corresponding to the decoded data are then determined based on the conditional constraints and the data in the codebook corresponding to the decoded data.

[0236] For example, based on the bits of data 1 in each dimension of the decoded data, data with the corresponding bits are selected from the cluster centers of that dimension in the first codebook; similarly, based on the bits of data 0 in each dimension of the decoded data, data with the corresponding bits are selected from the cluster centers of that dimension in the second codebook. The data selected from the codebooks is then compared with the conditions after the shifting process to obtain multiple cluster centers corresponding to the decoded data.

[0237] Combination Figure 8 (b) in the text explains that the cluster center is determined based on the conditional constraints and the correspondence between the data in each dimension of the decoded data and the digits in the codebook.

[0238] Figure 8This is a schematic diagram illustrating the reconstruction of cluster centers provided in this application. Figure 8 As shown in (b), the decoded data includes [10, 00, 01, 01]. The first dimension of the data includes 10, the second dimension includes 00, the third dimension includes 01, and the fourth dimension includes 01.

[0239] Assume the four cluster centers of the first dimension data are w, x, y, z; the four cluster centers of the second dimension data are a, b, c, d; the four cluster centers of the third dimension data are e, f, g, h; and the four cluster centers of the fourth dimension data are m, n, o, p. Generate a first codebook [zy, y, dc, c, hg, g, po, o] for all dimensions of data, and a second codebook [0, w, 0, a, 0, e, 0, m] for all dimensions of data.

[0240] In the first dimension of data 10, data 1 corresponds to the first codebook, data 0 corresponds to the second codebook, the first bit in data 10 corresponds to the first bit in the first dimension of the first codebook, and the second bit in data 10 corresponds to the second bit in the first dimension of the second codebook.

[0241] The data 1 in the first bit of data 10 corresponds to the data zy in the first dimension of the first codebook, and the data 0 in the second bit of data 10 corresponds to the data w in the second dimension of the second codebook.

[0242] Summing the data obtained from the codebook yields z - y + w. Rearranging the terms from the constraint x + y = w + z satisfied by the cluster centers, we get x = z - y + w. Therefore, x is the cluster center corresponding to the first dimension of the decoded data.

[0243] In the second dimension, data 00 corresponds to the second codebook. The first bit in data 00 corresponds to the first bit in the second dimension of the second codebook, and the second bit in data 00 corresponds to the second bit in the second dimension of the second codebook.

[0244] The data 0 in the first bit of data 00 corresponds to the data 0 in the first bit of the second dimension of the second codebook, and the data 0 in the second bit of data 00 corresponds to the data a in the second bit of the second dimension of the second codebook.

[0245] The data obtained from the codebook is summed to obtain 0 + a = a, and the cluster center corresponding to the first dimension of the decoded data is determined to be a.

[0246] In the third dimension of data 01, data 0 corresponds to the second codebook, and data 1 corresponds to the first codebook. The first bit in data 01 corresponds to the first bit in the third dimension of the second codebook, and the second bit in data 01 corresponds to the second bit in the third dimension of the first codebook.

[0247] The data 0 in the first bit of data 01 corresponds to the data 0 in the first bit of the third dimension of the second codebook, and the data 1 in the second bit of data 01 corresponds to the data g in the second bit of the third dimension of the first codebook.

[0248] The data obtained from the codebook is summed to obtain 0 + g = g. The cluster center corresponding to the third dimension of the decoded data is determined to be g.

[0249] In the fourth dimension, data 0 corresponds to the second codebook, and data 1 corresponds to the first codebook. The first bit in data 01 corresponds to the first bit in the fourth dimension of the second codebook, and the second bit in data 01 corresponds to the second bit in the fourth dimension of the first codebook.

[0250] The data 0 in the first bit of data 01 corresponds to the data 0 in the first bit of the fourth dimension of the second codebook, and the data 1 in the second bit of data 01 corresponds to the data o in the second bit of the fourth dimension of the first codebook.

[0251] The data obtained from the codebook is summed to obtain 0 + o = o, and the cluster center corresponding to the fourth dimension of the decoded data is determined to be o.

[0252] The cluster centers corresponding to each dimension of the decoded data are combined to determine the multiple cluster centers corresponding to the decoded data as [x, a, g, o].

[0253] In some embodiments, multiple cluster centers corresponding to the decoded data are determined based on the mapping relationship between the data of each dimension in the decoded data and the codebook.

[0254] For example, when a codebook is set up for all dimensions of the data, the cluster centers of the corresponding dimensions are found in the codebook based on the data of each dimension of the decoded data, and multiple cluster centers corresponding to the decoded data are determined.

[0255] For example, the four cluster centers of the first dimension data are w, x, y, z; the four cluster centers of the second dimension data are a, b, c, d; and the four cluster centers of the third dimension data are e, f, g, h. A first codebook [zy, y, dc, c, hg, g] is set for all dimensions of the data. A second codebook [0, w, 0, a, 0, e] is set for all dimensions of the data.

[0256] The decoded data is [01, 00, 10, 11]. The cluster center of the first dimension in the first codebook is [zy, y], and the cluster center of the first dimension in the second codebook is [0, w].

[0257] Based on the first digit 0 of the data 01 in the first dimension of the decoded data, select the first digit 0 from the first dimension cluster center [0, w] of the second codebook. Based on the second digit 1 of the data 01 in the first dimension of the decoded data, select the second digit y from the first dimension cluster center [zy, y] of the first codebook.

[0258] The data obtained from the codebook is summed to obtain 0 + y = y, and the cluster center corresponding to the first dimension of the decoded data is determined to be y.

[0259] Step 730: Search for multiple cluster centers corresponding to the decoded data based on the query data to obtain the search results.

[0260] In some embodiments, search results are obtained based on the similarity between the query data and multiple cluster centers.

[0261] For example, the search results include at least one cluster center from multiple cluster centers.

[0262] For example, the similarity between the query data and each of the multiple cluster centers determined by the decoded data is calculated, and at least one cluster center with a high similarity to the query data is selected as the retrieval result.

[0263] For example, the retrieval result is the original data in the dataset corresponding to at least one of the multiple cluster centers.

[0264] In one implementation, at least one cluster center with high similarity to the query data is obtained, and at least one original data corresponding to at least one cluster center is selected from the dataset as the retrieval result.

[0265] For example, the dataset contains four original data points V1, V2, V3, and V4. The data points V1, V2, V3, and V4 are encoded to obtain the encoded data.

[0266] The encoded data is decoded to obtain the decoded data. Based on the mapping relationship between the decoded data and the codebook, the multiple cluster centers corresponding to data V1 are determined as [x, a, g, o], the multiple cluster centers corresponding to data V2 are [w, b, e, m], the multiple cluster centers corresponding to data V3 are [y, c, f, n], and the multiple cluster centers corresponding to data V4 are [z, d, h, p].

[0267] The similarity between the query data and multiple cluster centers corresponding to V1, V2, V3, and V4 was calculated, and the query data showed high similarity with [x, a, g, o] and [z, d, h, p]. The original data V1 and V4 in the datasets corresponding to [x, a, g, o] and [z, d, h, p] were used as the search results.

[0268] The retrieval method provided in this application, combined with the Inverted File with Product Quantization (IVFPQ) algorithm, outperforms the IVFPQ algorithm in both retrieval efficiency and memory usage while achieving the same retrieval accuracy. The IVFPQ algorithm is a type of quantization algorithm.

[0269] For example, the retrieval method provided in this application achieves a 99% recall rate on a dataset containing 25 million data points, while the IVFPQ algorithm saves 11% of GPU memory compared to the open-source IVFPQ algorithm and has a retrieval speed 1.06 times faster.

[0270] On a dataset containing 100 million data points, the retrieval method provided in this application saves 69% of GPU memory compared to the open-source IVFPQ algorithm, and the retrieval speed is 1.42 times that of the IVFPQ algorithm.

[0271] Furthermore, the retrieval method provided in this application uses a 48G computing power card to search on a dataset containing 400 million data points, achieving a recall rate of 99%.

[0272] Next, refer to Figure 9 The process of explaining the search method provided in this application is described.

[0273] Figure 9 This is a flowchart illustrating a retrieval method provided in this application. Figure 9 As shown, the retrieval method includes a data encoding stage and a retrieval stage.

[0274] The base database, also known as the dataset, contains multiple data entries.

[0275] Cluster the data for each dimension of the base database, and determine multiple cluster centers for each dimension. Refer to step 310 for the specific implementation method.

[0276] Based on the similarity between each data point in the base database and multiple cluster centers, the base database data is quantized to the identifier of the cluster center, thus obtaining the quantized value of the base database data. For specific implementation details, please refer to step 320.

[0277] Low-bit data, also known as encoded data, is compressed into low-bit data by encoding the quantized values ​​of the base database data. For details on the implementation, please refer to step 330.

[0278] The low-bit data is decoded to obtain the decoded data. Based on the mapping relationship between the decoded data and the codebook, multiple cluster centers are determined. That is, the low-bit data is decoded back to the cluster centers. For specific implementation details, please refer to step 720.

[0279] Based on the query data included in the retrieval request, the distance between the query data and the reconstructed cluster centers is calculated to obtain the retrieval results. For specific implementation details, please refer to step 730.

[0280] Next, refer to Figure 10 Explain the encoding and decoding process provided in this application.

[0281] Figure 10 This is a schematic diagram illustrating an encoding and decoding process provided in this application. For example... Figure 10 As shown, the method includes the following steps.

[0282] Step 1010: Determine M cluster centers for each dimension of the data.

[0283] Cluster the data in each dimension of the dataset, determine M cluster centers for each dimension, and generate a codebook based on the cluster centers and mapping relationships. Refer to step 310 for the specific implementation.

[0284] Step 1020: Quantize the data in the dataset into [0, 1, ..., M-1].

[0285] For each of the M cluster centers obtained in step 1010, assign a representation: the first cluster center is identified as 0, the second as 1, and the Mth as M-1. Calculate the similarity between the data in each dimension of the dataset and the M cluster centers in that dimension. Quantize the data in each dimension of the dataset into the identifier of the cluster center with the highest similarity, obtaining quantized values ​​for multiple data points. Refer to step 320 for the specific implementation method.

[0286] Step 1030: Compress the M adjacent quantization values.

[0287] The quantized values ​​of each data point obtained in step 1020 are divided into multiple groups of quantized values, with the number of quantized values ​​in each group matching the number of cluster centers, M. Adjacent M quantized values ​​are then compressed, i.e., each group of quantized values ​​is binary encoded to obtain the encoded dataset. For details, refer to step 330.

[0288] Suppose a dataset contains n data points, has a dimension of dim, and a size of n*dim. By compressing the M adjacent quantized values ​​of multiple data points, the size of the dataset is reduced to n*dim / M.

[0289] Step 1040: Binary Decoding.

[0290] The encoded data obtained in step 1030 is binary decoded to obtain intermediate data. For specific implementation details, please refer to step 720.

[0291] Step 1050: Merge adjacent bit data.

[0292] The intermediate data obtained in step 1050 is arranged from least significant bit to most significant bit, and adjacent bits are merged to obtain the decoded data. For specific implementation details, please refer to step 720.

[0293] Step 1060: Restore to multiple cluster centers.

[0294] Based on the mapping relationship between the decoded data obtained in step 1050 and the codebook obtained in step 1010, the decoded data is restored to the corresponding multiple cluster centers. For specific implementation details, please refer to step 720.

[0295] Next, refer to the example. Figure 11 Explain the process of data encoding and decoding.

[0296] Figure 11 This is a schematic diagram of data encoding and decoding provided for this application. For example... Figure 11 As shown, the dataset contains six data points V1-V6, and the dimension of any vector data point in V1-V6 is 8. Taking data point V3 as an example, the data encoding method is explained.

[0297] Perform a clustering algorithm on each dimension of the multiple data points to determine the cluster center for each dimension. For details, please refer to step 310; further explanation is omitted here.

[0298] For example, if the number of cluster centers is 4, the cluster centers in the first dimension are 1, 2, 3, and 4, and the corresponding labels are 0, 1, 2, and 3.

[0299] The cluster centers for the second dimension are 3, 4, 5, and 6, with corresponding labels of 0, 1, 2, and 3.

[0300] The cluster centers in the third dimension are 2, 3, 5, and 6, with corresponding labels of 0, 1, 2, and 3.

[0301] The cluster centers for the fourth dimension are 4, 5, 6, and 7, with corresponding labels of 0, 1, 2, and 3.

[0302] The cluster centers for the fifth dimension are 5, 6, 8, and 9, with corresponding labels of 0, 1, 2, and 3.

[0303] The cluster centers for the sixth dimension are 7, 8, 10, and 11, with corresponding labels of 0, 1, 2, and 3.

[0304] The cluster centers for the seventh dimension are 6, 7, 9, and 10, with corresponding labels of 0, 1, 2, and 3.

[0305] The cluster centers of the eighth dimension are 8, 9, 11, and 12, with corresponding labels of 0, 1, 2, and 3.

[0306] Based on the determined cluster centers, a codebook is generated for each dimension of the data. The method for generating the codebook can be referred to in step 330, and will not be elaborated here.

[0307] For example, the first codebook of the first dimension is [1, 3], and the second codebook of the first dimension is [0, 1].

[0308] The first codebook of the second dimension is [1, 5], and the second codebook of the first dimension is [0, 3].

[0309] The first codebook of the third dimension is [1, 5], and the second codebook of the first dimension is [0, 2].

[0310] The first codebook of the fourth dimension is [1, 6], and the second codebook of the first dimension is [0, 4].

[0311] The first codebook of the fifth dimension is [1, 8], and the second codebook of the first dimension is [0, 5].

[0312] The first codebook of the sixth dimension is [1, 10], and the second codebook of the first dimension is [0, 7].

[0313] The first codebook of the seventh dimension is [1, 9], and the second codebook of the first dimension is [0, 6].

[0314] The first codebook of the eighth dimension is [1, 11], and the second codebook of the first dimension is [0, 8].

[0315] The dataset contains multiple data points that are quantized based on multiple cluster centers, resulting in quantized values ​​for each data point. The quantization method is detailed in step 320 and will not be repeated here.

[0316] For example, to quantize V3, the data of each dimension of V3 is quantized into the identifier of the cluster center with the highest similarity, resulting in the quantized value L3[2, 2, 2, 2, 0, 2, 2, 2].

[0317] The quantized values ​​of multiple data points are divided into multiple groups of quantized values ​​based on the number of cluster centers. These multiple groups of quantized values ​​are then encoded to obtain the encoded data of the dataset.

[0318] For example, with 4 cluster centers, the quantized value L3 is divided into two groups: [2, 2, 2, 2] and [0, 2, 2, 2]. The number of quantized values ​​in each group is consistent with the number of cluster centers. Binary encoding of [2, 2, 2, 2] yields 2*2^0 + 2*2^2 + 2*2^4 + 2*2^6 = 170. Binary encoding of [0, 2, 2, 2] yields 0*2^0 + 2*2^2 + 2*2^4 + 2*2^6 = 168.

[0319] The encoded data of the dataset is decoded to obtain the decoded data, and multiple cluster centers are determined based on the mapping relationship between the decoded data and the codebook.

[0320] For example, the encoded data 170 is binary decoded to obtain the intermediate data [1, 0, 1, 0, 1, 0, 1, 0]. The intermediate data is sorted from the least significant bit to the most significant bit to obtain [0, 1, 0, 1, 0, 1, 0, 1]. The adjacent bits of [0, 1, 0, 1, 0, 1, 0, 1] are merged to obtain the decoded data [01, 01, 01, 01].

[0321] The encoded data 168 is binary decoded to obtain the intermediate data [1, 0, 1, 0, 1, 0, 0, 0]. The intermediate data is sorted from the least significant bit to the most significant bit to obtain [0, 0, 0, 1, 0, 1, 0, 1]. The adjacent bits of [0, 0, 0, 1, 0, 1, 0, 1] are merged to obtain the decoded data [00, 01, 01, 01].

[0322] Multiple cluster centers are determined based on the correspondence between the data in each dimension of the decoded data and the digits in the codebook. For details, please refer to step 720.

[0323] For example, in the decoded data [01, 01, 01, 01], the data in the first dimension is 01. Based on the first bit of the data 01 in the first dimension, we search for the first bit in the second codebook of the first dimension and get 0. Based on the second bit of the data 1 in the data 01 in the first dimension, we search for the second bit in the first codebook of the first dimension and get 3. Summing the data obtained from the codebook, we get 0 + 3 = 3, thus determining the cluster center corresponding to the data in the first dimension of the decoded data as 3.

[0324] Cluster centers were determined for each dimension of the decoded data [01, 01, 01, 01], resulting in cluster centers [3, 5, 5, 6] for the decoded data [01, 01, 01, 01].

[0325] Cluster centers were determined for each dimension of the decoded data [00, 01, 01, 01], and the cluster centers for the decoded data [00, 01, 01, 01] were [5, 10, 9, 11].

[0326] Combining [3, 5, 5, 6] and [5, 10, 9, 11], we obtain the cluster centers determined by the decoded data of data V3 as [3, 5, 5, 6, 5, 10, 9, 11].

[0327] Next, refer to Figure 12 This section introduces the process of determining cluster centers based on residual data.

[0328] Figure 12 This is a flowchart illustrating a method for determining cluster centers based on residual data, as provided in this application. Figure 12 As shown, multiple data points in the dataset are clustered to obtain n clusters and n cluster centers. For example, each cluster contains one center, and each cluster contains at least one data point.

[0329] Traverse n clusters. After all clusters have been traversed, obtain the center of each cluster. Based on the obtained center of each cluster, calculate the difference between the data contained in each cluster and the center of that cluster to obtain the residual data. For the specific implementation method, please refer to step 310.

[0330] If not all n clusters have been traversed, continue traversing the untraversed clusters until all n clusters have been traversed.

[0331] Iterate through each dimension of the residual data. If not all dimensions of the residual data have been traversed, cluster the data for each dimension to determine multiple cluster centers. Refer to step 310 for the specific implementation. Once all dimensions of the residual data have been traversed, the determined multiple cluster centers are obtained.

[0332] It is understood that, in order to achieve the functions in the above embodiments, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0333] The following will combine Figure 13 This application describes an encoding apparatus provided according to the present application. The encoding apparatus is used to implement the encoding function of the retrieval system in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the apparatus is as follows: Figure 2 The retrieval system shown is still a module (such as a chip) applied to computer equipment.

[0334] like Figure 13 As shown, the encoding device 1300 includes a communication module 1310, a quantization module 1320, an encoding module 1330, and a storage module 1340.

[0335] The communication module 1310 is used to acquire a dataset containing multiple data entries.

[0336] Quantization module 1320 is used to determine multiple cluster centers based on the characteristics of multiple data points in the dataset, and to quantize the multiple data points in the dataset based on the multiple cluster centers to obtain quantized values ​​for the multiple data points. For example, quantization module 1320 is used to perform... Figure 3 Steps 310 and 320 in the process.

[0337] Encoding module 1330 is used to encode the quantized values ​​of multiple data points to obtain the encoded data of the dataset. For example, encoding module 1330 is used to perform... Figure 3 Step 330 in the process.

[0338] Optionally, the encoding module 1330 is also used to generate a codebook based on the mapping relationship and multiple cluster centers.

[0339] Storage module 1340 is used to store encoded data, codebooks, etc.

[0340] The encoding device 1300 in this embodiment is implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figure 3 In the method shown, the encoding device 1300 and its various modules are also software modules.

[0341] The encoding apparatus 1300 according to the embodiments of this application can correspondingly execute the methods described in the embodiments of this application, and the above and other operations and / or functions of each unit in the encoding apparatus 1300 are respectively for implementing Figure 3 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0342] The following will combine Figure 14 This application describes a retrieval apparatus provided according to the present application. This retrieval apparatus is used to implement the retrieval function of the retrieval system in the above-described method embodiments, and therefore also achieves the beneficial effects of the above-described method embodiments. In this embodiment, the apparatus is as follows: Figure 2 The retrieval system shown is still a module (such as a chip) applied to computer equipment.

[0343] like Figure 14 As shown, the retrieval device 1400 includes a communication module 1410, a decoding module 1420, a retrieval module 1430, and a storage module 1440.

[0344] The retrieval device 1400 is used to achieve the above. Figure 2 The method embodiment shown illustrates the function of the retrieval system 200.

[0345] Communication module 1410 is used to acquire a retrieval request, which includes query data. For example, communication module 1410 is used to execute... Figure 7 Step 710 in the process.

[0346] The decoding module 1420 is used to decode the encoded data to obtain decoded data, and to determine multiple cluster centers corresponding to the decoded data based on the mapping relationship between the decoded data and the codebook. For example, the decoding module 1420 is used to perform... Figure 7 Step 720 in the process.

[0347] The retrieval module 1430 is used to retrieve multiple cluster centers corresponding to the decoded data based on the query data, and obtain the retrieval results. For example, the retrieval module 1430 is used to execute... Figure 7 Step 730 in the process.

[0348] Storage module 1440 is used to store encoded data, codebooks, etc.

[0349] It should be understood that the retrieval device 1400 in this application embodiment is implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented using software. Figure 7 In the method shown, the retrieval device 1400 and its various modules are also software modules.

[0350] The retrieval device 1400 according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the above and other operations and / or functions of each unit in the retrieval device 1400 are respectively for implementing Figure 7 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0351] Figure 15 A schematic diagram of the structure of a computer device 1500 provided in this application. Figure 15 As shown, computer device 1500 includes processor 1510, bus 1520, memory 1530, communication interface 1540, memory 1550 (also called main memory unit), and processor 1560. Processor 1510, processor 1560, memory 1530, memory 1550, and communication interface 1540 are connected via bus 1520.

[0352] It should be understood that in this embodiment, processor 1510 is a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor is a microprocessor or any conventional processor.

[0353] Computer device 1500 also includes a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present application. For example, processor 1560 is a GPU or an NPU.

[0354] The communication interface 1540 is used to enable communication between the computer device 1500 and external devices or components.

[0355] In this application, computer device 1500 is used to implement Figure 2 When the retrieval system is functioning as shown, the communication interface 1540 is used to acquire retrieval requests, so that the processor 1510 can retrieve multiple cluster centers corresponding to the decoded data based on the query data contained in the retrieval request, and obtain retrieval results. The processor 1560 is used to determine multiple cluster centers based on the characteristics of multiple data points contained in the dataset, quantize the multiple data points contained in the dataset based on the multiple cluster centers, obtain quantized values ​​of the multiple data points, encode the quantized values ​​of the multiple data points, and obtain the encoded data of the dataset. The processor 1560 is also used to decode the encoded data of the dataset to obtain decoded data, and determine multiple cluster centers corresponding to the decoded data based on the mapping relationship between the decoded data and the codebook.

[0356] Bus 1520 includes a pathway for transferring information between the aforementioned components (such as processor 1510, memory 1550, and storage 1530). In addition to the data bus, bus 1520 also includes a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 1520 in the diagram. Bus 1520 is a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, Unified Bus (Ubus or UB), Compute Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX), etc. Bus 1520 is divided into address bus, data bus, and control bus.

[0357] As an example, computer device 1500 includes multiple processors. A processor is a multi-core (multi-CPU) processor. Here, a processor refers to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0358] It is worth noting that, Figure 15Taking computer device 1500, which includes one processor 1510 and one memory 1530, as an example, the processor 1510 and memory 1530 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment is determined according to business requirements. For example, computer device 1500 includes multiple GPUs or NPUs.

[0359] Memory 1550 can be non-transitory memory, which can be a pool of volatile memory or a pool of non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). Memory 1550 is used to store datasets or encoded data, etc.

[0360] The memory 1530 corresponds to the storage medium used in the above method embodiments for storing data sets or encoded data, such as a disk, like a mechanical hard disk or a solid-state hard disk.

[0361] The aforementioned computer device 1500 may be a general-purpose device or a special-purpose device. For example, computer device 1500 may also be a server or other device with computing capabilities.

[0362] It should be understood that the computer device 1500 according to this embodiment may correspond to the encoding device 1300 in this embodiment, and correspond to the execution according to Figure 3The corresponding subject in any of the methods, and the above and other operations and / or functions of each module in the encoding device 1300 are respectively for implementing Figure 3 The corresponding processes of each method in the code are described below. For the sake of brevity, they will not be elaborated here.

[0363] The computer device 1500 in this embodiment may also correspond to the retrieval device 1400 in this embodiment, and correspond to the execution based on Figure 7 The corresponding subject in any of the methods, and the above and other operations and / or functions of each module in the retrieval device 1400 are respectively implemented for the purpose of... Figure 7 The corresponding processes of each method in the code are described below. For the sake of brevity, they will not be elaborated here.

[0364] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in the computing device.

[0365] Some embodiments of this application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform one or more steps in the data encoding method as described in any of the above embodiments.

[0366] For example, the aforementioned computer-readable storage media include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the embodiments of this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0367] Some embodiments of this application also provide a computer program product. This computer program product includes computer program instructions carried on a non-transitory computer-readable storage medium, which, when executed on a computer, cause the computer to perform one or more steps of the data storage method as described in the above embodiments.

[0368] The beneficial effects of the computer-readable storage medium and computer program product described above are the same as those of the data storage method described in some of the above embodiments, and will not be repeated here.

[0369] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A retrieval method, characterized in that, include: Obtain a search request, which includes query data; The encoded data of the dataset is decoded to obtain decoded data. Multiple cluster centers corresponding to the decoded data are determined according to the mapping relationship between the decoded data and the codebook. The encoded data of the dataset is obtained by quantizing and encoding multiple data in the dataset according to the multiple cluster centers. The codebook is used to indicate the multiple cluster centers. Based on the query data, multiple cluster centers corresponding to the decoded data are retrieved to obtain retrieval results, which include data corresponding to at least one cluster center.

2. The method according to claim 1, characterized in that, Based on the mapping relationship between the decoded data and the codebook, multiple cluster centers corresponding to the decoded data are determined, including: Based on the digit correspondence between the decoded data and the codebook, multiple cluster centers corresponding to the decoded data are determined from the codebook.

3. The method according to claim 2, characterized in that, Conditional constraints are used to indicate the relationships between multiple cluster centers; Based on the digit correspondence between the decoded data and the codebook, multiple cluster centers corresponding to the decoded data are determined from the codebook, including: The data in the codebook corresponding to the decoded data is determined based on the digit correspondence between the decoded data and the codebook. Based on the stated constraints and the data in the codebook corresponding to the decoded data, multiple cluster centers corresponding to the decoded data are determined.

4. The method according to any one of claims 1-3, characterized in that, Based on the mapping relationship between the decoded data and the codebook, multiple cluster centers corresponding to the decoded data are determined, including: Based on the mapping relationship between the data of each dimension in the decoded data and the codebook, multiple cluster centers corresponding to the decoded data are determined.

5. The method according to claim 1, characterized in that, The decoded data obtained by decoding the encoded data of the dataset includes: The encoded data of the dataset is binary decoded to obtain the decoded data.

6. The method according to claim 1, characterized in that, The method further includes: The encoded data of the dataset is binary-decoded to obtain intermediate data; Adjacent bits of the intermediate data are merged to obtain the decoded data.

7. A data encoding method, characterized in that, include: Multiple cluster centers are determined based on the characteristics of multiple data points contained in the dataset; The quantized values ​​of the multiple data points contained in the dataset are obtained by quantizing the multiple cluster centers. The quantized values ​​of the multiple data points are encoded to obtain the encoded data of the dataset. The encoded data of the dataset is used to restore the dataset according to the codebook during retrieval. The codebook is used to indicate the multiple cluster centers.

8. The method according to claim 7, characterized in that, Encoding the quantized values ​​of the multiple data points yields the encoded data of the dataset, including: The quantized values ​​of the multiple data points are divided into multiple groups of quantized values ​​based on the number of cluster centers. The multiple groups of quantized values ​​are then encoded to obtain the encoded data of the dataset.

9. The method according to claim 7 or 8, characterized in that, The multiple cluster centers satisfy conditional constraints, which are used to indicate the relationship between the multiple cluster centers.

10. The method according to any one of claims 7-9, characterized in that, The method further includes: A codebook is generated based on the mapping relationship and the plurality of cluster centers. The mapping relationship is used to indicate the correspondence between the decoded data obtained by decoding the encoded data and the codebook.

11. The method according to any one of claims 7-10, characterized in that, Encoding the quantized values ​​of the multiple data points yields the encoded data of the dataset, including: The quantized values ​​of the multiple data points are binary encoded to obtain the encoded data of the dataset.

12. The method according to any one of claims 7-11, characterized in that, The dataset contains multiple data points that are residual data determined based on the original data in the dataset.

13. The method according to claim 12, characterized in that, The method further includes: The original data of the dataset is clustered to obtain two or more clusters; Based on the relationship between the data in each cluster and the center of the cluster, the original data of the dataset is converted into residual data.

14. A retrieval device, characterized in that, The retrieval device includes: The communication module is used to obtain retrieval requests, which include query data. The decoding module is used to decode the encoded data of the dataset to obtain decoded data, determine multiple cluster centers corresponding to the decoded data according to the mapping relationship between the decoded data and the codebook, and obtain the encoded data of the dataset by quantizing and encoding multiple data in the dataset according to the multiple cluster centers. The codebook is used to indicate the multiple cluster centers. The retrieval module is used to retrieve multiple cluster centers corresponding to the decoded data based on the query data, and obtain retrieval results, wherein the retrieval results include data corresponding to at least one cluster center.

15. An encoding device, characterized in that, The encoding device includes: The quantization module is used to determine multiple cluster centers based on the characteristics of multiple data points contained in the dataset. The quantization module is further configured to quantize multiple data points contained in the dataset based on the multiple cluster centers, and obtain quantized values ​​for the multiple data points; An encoding module is used to encode the quantized values ​​of the multiple data points to obtain the encoded data of the dataset. The encoded data of the dataset is used to restore the dataset according to the codebook during retrieval. The codebook is used to indicate the multiple cluster centers.

16. A retrieval system, characterized in that, The retrieval system includes a retrieval device and an encoding device, wherein the retrieval device is used to perform the method as described in any one of claims 1-6, and the encoding device is used to perform the method as described in any one of claims 7-13.

17. A computer device, characterized in that, The computer device includes a memory and multiple processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the multiple processors perform the operational steps of the method according to any one of claims 1-13.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, which, when executed by a computer device, implement the method as described in any one of claims 1-6 or the method as described in any one of claims 7-13.

19. A computer program product, characterized in that, The computer program product includes a computer program or instructions, which, when executed by a computer device, implement the method as described in any one of claims 1-6 or the method as described in any one of claims 7-13.

Images