Data processing method, device and system

Abstract

The present invention discloses a method, device and system for data processing. The method includes: reading a quantization index from a memory; restoring from a code table according to the quantization index to obtain a quantization vector; interleaving a request vector with a quantization vector to obtain a quantization vector dimension; calculating according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data according to the distance value. The present invention solves the technical problem that the data set reading bandwidth cannot be compressed and the system throughput cannot be improved because the traditional quantization compression in the prior art only supports symmetric quantization and the traditional optimization of the retrieval bandwidth of the quantization data set is based on the CPU / GPU system.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device and system for data processing. Background Art

[0002] In order to ensure a certain recall rate when searching large data sets, it is necessary to ensure that the proportion of the search set actually traversed to the entire data set is not too small. Therefore, in the acceleration of vector search, each search request corresponds to a large amount of search data that needs to be read, making the read bandwidth often become the bottleneck of system throughput, which is the so-called memory limit.

[0003] Vector retrieval means retrieving the k vectors closest to a given request vector X from a very large dataset S (e.g., tens of millions to billions of vectors). If no compression is used, the storage space size of the dataset is O(|S|D), where |S| is the size of the dataset and D is the dimension of each vector.

[0004] The problem with traditional quantization compression is that it only supports symmetric quantization, that is, the quantizer parameters of each subspace are consistent, and the traditional optimization of the retrieval bandwidth of the quantized data set is based on the CPU / GPU system. As a result, the defect is that it is impossible to compress the data set reading bandwidth and improve the system throughput.

[0005] Currently, no effective solution has been proposed to address the above-mentioned problem that traditional quantization compression in the prior art only supports symmetric quantization, and traditional optimization of the retrieval bandwidth of quantized data sets is based on the CPU / GPU system, resulting in the inability to compress the data set reading bandwidth and improve the system throughput. Summary of the invention

[0006] The embodiments of the present invention provide a method, device and system for data processing to at least solve the technical problem that the data set reading bandwidth cannot be compressed and the system throughput cannot be improved because the traditional quantization compression in the prior art only supports symmetric quantization and the traditional optimization of the retrieval bandwidth of the quantized data set is based on the CPU / GPU system.

[0007] According to one aspect of an embodiment of the present invention, a method for data processing is provided, comprising: reading a quantization index from a memory; restoring from a code table based on the quantization index to obtain a quantization vector; interleaving a request vector with a quantization vector to obtain a quantization vector dimension; calculating based on the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data based on the distance value.

[0008] Optionally, reading the quantization index from the memory includes: when the memory includes an off-chip memory and / or an on-chip memory, reading the quantization index from a corresponding memory.

[0009] Optionally, restoring from a code table according to a quantization index to obtain a quantization vector includes: distributing at least one quantization index to a corresponding code table; and restoring through the code table according to the quantization index to obtain a quantization vector.

[0010] Optionally, interleaving the request vector with the quantization vector to obtain the quantization vector dimension includes: querying the sub-code table in the unit memory according to the quantization index to obtain the quantization vector dimension corresponding to the request vector, wherein the code table is stored in at least one unit memory, and the output of the unit memory is arranged in the same order as the dimension of the request vector.

[0011] Optionally, calculation is performed based on the quantization vector dimension, the request vector and the quantization vector to obtain the distance value between the request vector and the quantization vector, including: calculating the distance value between each request vector and the quantization center point where the quantization vector is located based on the quantization vector dimension and the request vector; and accumulating the distance values ​​to obtain the distance value between the request vector and the quantization vector.

[0012] According to another aspect of an embodiment of the present invention, a data processing device is also provided, including: a reading module, used to read a quantization index from a memory; a restoration module, used to restore from a code table according to the quantization index to obtain a quantization vector; an interleaving module, used to interleave a request vector with a quantization vector to obtain a quantization vector dimension; a calculation module, used to calculate according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and a compression module, used to compress data according to the distance value.

[0013] Optionally, the reading module includes: a reading unit, used to read the quantization index from the corresponding memory when the memory includes an off-chip memory and / or an on-chip memory.

[0014] Optionally, the calculation module includes: a first calculation unit, used to calculate the distance value between each request vector and the quantization center point where the quantization vector is located based on the quantization vector dimension and the request vector; and a second calculation unit, used to accumulate the distance values ​​to obtain the distance value between the request vector and the quantization vector.

[0015] According to another aspect of an embodiment of the present invention, a data processing system is also provided, comprising: a first data throughput area, an interleaving device and a second data throughput area, wherein the first data throughput area comprises: a memory, an index distributor and at least one code table, wherein the index distributor is connected to the memory, and is used to read at least one quantization index and send at least one quantization index to the corresponding code table; the quantization index is restored by the code table to obtain a quantization vector; the interleaving device is connected to the first data throughput area and the second data throughput area, respectively, and is used to interleave the quantization vector with the request vector, and input the interleaving result into the second data throughput area; the second data throughput area comprises: a distance calculator and an addition tree, wherein the distance calculator is used to calculate the distance value between each request vector and the quantization center point where the quantization vector is located; the addition tree is connected to the distance calculator, and is used to accumulate the distance value to obtain the distance value between the request vector and the quantization vector, and output the distance value to the sorting circuit.

[0016] Optionally, the memory includes: an off-chip memory and / or an on-chip memory, wherein the off-chip memory and / or the on-chip memory is used to store the quantization index.

[0017] According to another aspect of an embodiment of the present invention, a storage medium is also provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute reading a quantization index from a memory; restoring from a code table based on the quantization index to obtain a quantization vector; interleaving a request vector and a quantization vector to obtain a quantization vector dimension; calculating based on the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data based on the distance value.

[0018] According to another aspect of an embodiment of the present invention, a processor is provided, and the processor is used to run a program, wherein when the program is running, the quantization index is read from the memory; the quantization index is restored from the code table to obtain the quantization vector; the request vector and the quantization vector are interleaved to obtain the quantization vector dimension; the distance value between the request vector and the quantization vector is calculated based on the quantization vector dimension, the request vector and the quantization vector; and the data is compressed based on the distance value.

[0019] In an embodiment of the present invention, an asymmetric quantization method is adopted, by reading a quantization index from a memory; restoring from a code table according to the quantization index to obtain a quantization vector; interleaving a request vector and a quantization vector to obtain a quantization vector dimension; calculating according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data according to the distance value, thereby achieving the purpose of reducing the system's demand for read bandwidth by quantizing and compressing the data set itself, caching in space, and batch processing of request vectors in time, thereby achieving the technical effect of improving the throughput of the system, and further solving the technical problem that the data set reading bandwidth cannot be compressed and the system throughput cannot be improved because traditional quantization compression in the prior art only supports symmetric quantization and the traditional optimization of the retrieval bandwidth of the quantized data set is based on the CPU / GPU system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 This is a schematic diagram of the process of quantifying the dataset of this application;

[0022] Figure 2 is a schematic diagram of a data processing system according to Embodiment 1 of the present invention;

[0023] Figure 3 It is a schematic diagram of storage allocation between an off-chip memory and an on-chip memory in a data processing system according to the first embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of interleaving adjustment in a data processing system according to Embodiment 1 of the present invention;

[0025] Figure 5 is a schematic diagram of batch processing in a data processing system according to Embodiment 1 of the present invention;

[0026] Figure 6 It is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention;

[0027] Figure 7 is a flowchart of a data processing method according to a second embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of a data processing device according to a third embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Technical terms involved in the embodiments of this application:

[0032] Vector retrieval: The process of retrieving several vectors closest to the requested vector from a vector data set;

[0033] Vector retrieval means retrieving the nearest vectors from a very large dataset S (e.g., tens of millions to billions) given a request vector X. If no compression is used, the storage space size of the dataset is O(|S|D), where |S| is the size of the dataset and D is the dimension of each vector.

[0034] Product Quantization (PQ) is a method to reduce the amount of vector retrieval calculations and storage. In PQ retrieval, the vector y in the data set is first divided into m sub-vectors, each with a dimension of D / m, and then each sub-vector is quantized separately: a certain number of quantization center points (assuming the number is n) are set in each subspace, and each sub-vector is mapped to one of the center points during quantization. When storing, the quantized data set is stored in two parts:

[0035] After quantization, each sub-vector needs a log2(n) bit index to indicate which center point it is mapped to. The quantization indexes of all sub-vectors are gathered together to form an index vector with dimension m and size mlog2(n) bits.

[0036] In addition, an additional O(nD) space is required to store the center points of all subspaces, which is called a code table.

[0037] In general, Figure 1 This is a schematic diagram of the process of quantifying the data set of this application, such as Figure 1 As shown, the quantization process of the entire data set is divided into two steps:

[0038] Codebook training: The subspace is divided based on some optimization rules, and then the codebook is trained in each subspace according to the distribution of data. A common training method is to cluster the subvectors of the data set in the subspace.

[0039] Vector quantization: Quantize each vector in the data set based on the trained codebook. Finally, aggregate the codebook and the quantization index of each vector to obtain the quantized data set.

[0040] FPGA / ASIC: underlying chip used for vector retrieval acceleration;

[0041] Product quantization: A method for vector retrieval.

[0042] Example 1

[0043] According to one aspect of an embodiment of the present invention, a data processing system is provided. Figure 2 is a schematic diagram of a data processing system according to Embodiment 1 of the present invention. Figure 2 As shown, the data processing system provided in the embodiment of the present application includes:

[0044] A first data throughput area 22, an interleaving device 24 and a second data throughput area 26, wherein the first data throughput area 22 includes: a memory, an index distributor and at least one code table, wherein the index distributor is connected to the memory, and is used to read at least one quantization index and send at least one quantization index to the corresponding code table; the code table restores the quantization index to obtain a quantization vector; the interleaving device 24 is connected to the first data throughput area 22 and the second data throughput area 26 respectively, and is used to interleave the quantization vector with the request vector, and input the interleaving result into the second data throughput area 26; the second data throughput area 26 includes: a distance calculator and an addition tree, wherein the distance calculator is used to calculate the distance value between each request vector and the quantization center point where the quantization vector is located; the addition tree is connected to the distance calculator, and is used to accumulate the distance value to obtain the distance value between the request vector and the quantization vector, and output the distance value to the sorting circuit.

[0045] Optionally, the memory includes: an off-chip memory and / or an on-chip memory, wherein the off-chip memory and / or the on-chip memory is used to store the quantization index.

[0046] Specifically, the data processing system provided in the embodiment of the present application can be applied to the retrieval of large data sets and used to accelerate vector retrieval. The data processing system provided in the embodiment of the present application will reduce the system's demand for read bandwidth from the quantitative compression of the data set itself, the spatial cache, and the batch processing of the request vector in time, and proposes a corresponding hardware acceleration system, wherein the hardware acceleration system is as follows: Figure 2 As shown, all quantization indexes are stored in the chip external memory, or if there is enough internal memory, some indexes are stored in the chip internal memory, so that the bandwidth pressure of the external memory can be greatly reduced by internal caching. Usually, the codebook occupies a limited storage space, so it is stored in the chip internal memory.

[0047] After the quantization index is read from the internal / external memory, the corresponding quantized sub-index is distributed to the corresponding code table for code table search, and the quantization index is restored to the corresponding quantization vector. Limited by the storage bandwidth, the above index reading and code table search work in the low throughput area (i.e., the first data throughput area 22 in the embodiment of the present application).

[0048] Among them, in the embodiment of the present application, the hardware acceleration system can be a data processing system based on FPGA / ASIC chips. In this hardware acceleration system, there is often a certain cache on the chip. Compared with the external memory, its capacity is small, but the access speed is very fast. When using hardware to accelerate vector retrieval, part of the quantized data set (or all quantized indexes of part of the subspace for more balanced access) can be cached on the chip. In this way, access to the data set can be obtained partly from the internal cache and partly from the external memory, thereby reducing the external memory bandwidth requirement. The storage distribution of the on-chip memory and the off-chip memory in the first data throughput area 22 is as follows: Figure 3 As shown, Figure 3 The diagram is a schematic diagram of storage allocation between an off-chip memory and an on-chip memory in a data processing system according to the first embodiment of the present invention.

[0049] like Figure 2 As shown, in the first data throughput area 22, i.e., the low throughput area, a quantization index is obtained through an off-chip memory and / or an on-chip memory, at least one quantization index is distributed to each code table for search through an index distributor, and the quantization index is restored to a corresponding quantization vector, wherein, Figure 2 Where q(0), q(1), ..., q(m-1) are quantization indices distributed by the index distributor from the off-chip memory and / or the on-chip memory, and y(0), y(1), ..., y(m-1) are restored quantization vectors.

[0050] based on Figure 2As shown in the schematic diagram of the data processing system, since the dimensional order of the request vector may be inconsistent with the dimensional order of the quantization vector restored from the code table, in order to make the two match, it is necessary to interleave the request vector dimension. That is, the interleaving device 24 provided in the embodiment of the present application obtains the input request vector X, and interleaves the request vector X with the above quantization vector, wherein the interleaving adjustment process is as follows Figure 4 As shown, Figure 4 2 is a schematic diagram of interleaving adjustment in a data processing system according to Embodiment 1 of the present invention.

[0051] Among them, a complete code table is stored in several unit memories, and the output of the unit memory is arranged in the original dimension order of the data set / request vector. The input quantization index will be cross-connected to each unit memory for sub-code table search. It can be seen that the unit memories sharing the same index constitute a complete code table. By reconfiguring the index cross connection, this architecture can support arbitrary division and aggregation of dimensions to a certain extent.

[0052] Based on the above, if Figure 2 As shown, in the high throughput area, that is, the second data throughput area 26 in the embodiment of the present application, the request vector is distanced with each quantization center point, and then the result is output to the addition tree for distance accumulation to obtain the final distance between the request vector and the quantization vector. By adopting the pipeline design mode, this part of the circuit can be operated at a higher frequency, or by duplicating the distance calculation and the addition tree circuit, this part of the function can be operated in the high throughput area. Figure 2 As shown in the figure, assuming that the distance calculation module can consume B quantized vectors per clock, but in fact only one quantized vector can be provided per clock cycle, the throughput of the entire system will be limited by the low-throughput area circuit. However, when all request vectors need to traverse all quantized data sets, batch processing can be used to improve the throughput of the entire system. That is, each time a quantized data point is read, the distance calculation is performed with B request vectors, thereby balancing the bandwidth of the two and improving the system throughput.

[0053] Among them, the batch processing method is used to improve the throughput of the entire system, including: if the retrieval of each request vector needs to traverse the entire data set, then the reading of a single quantized data point can be applied to multiple request vector data for distance calculation. Figure 5 FIG. 1 is a schematic diagram of batch processing in a data processing system according to Embodiment 1 of the present invention. Figure 5As shown, the read quantized data point y is simultaneously calculated with B request vectors, so that the system throughput can be increased to B times the original, where B is the batch size (the number of request vectors that perform distance calculations simultaneously). It is worth noting that in another implementation, the request vectors can also be serial, in which case the working frequency of the distance calculation is required to be high enough to complete the distance calculation with B request vectors before reading the next y.

[0054] The accumulated distances will be sent to the to pk sorting circuit. Each to pk sorting circuit corresponds to B request vectors. Therefore, in batch processing mode, at most to pk sorting circuits are required.

[0055] In summary, the data processing system provided in the embodiment of the present application calculates the distance between the request vector and the quantization vector through asymmetric quantization, and quantizes and compresses the obtained distance by accumulation, thereby achieving the purpose of reducing the system's demand for reading bandwidth by quantizing and compressing the data set itself, caching in space, and batch processing of request vectors in time, thereby achieving the technical effect of improving the throughput of the system, and further solving the technical problem that the traditional quantization compression in the prior art only supports symmetric quantization, and the traditional optimization of the retrieval bandwidth of the quantized data set is based on the CPU / GPU system, which makes it impossible to compress the data set reading bandwidth and improve the system throughput.

[0056] Example 2

[0057] According to an embodiment of the present invention, a method embodiment of data processing is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0058] The method embodiment provided in the second embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 6 FIG. 1 is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention. Figure 6 As shown, the computer terminal 60 may include one or more (only one is shown in the figure) processors 602 (the processor 602 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 604 for storing data, and a transmission module 606 for communication functions. It can be understood by those skilled in the art that Figure 6 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6More or fewer components as shown, or with Figure 6 Different configurations shown.

[0059] The memory 604 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the data processing method in the embodiment of the present invention. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, that is, the data processing method of the above-mentioned application program is realized. The memory 604 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 604 may further include a memory remotely arranged relative to the processor 602, and these remote memories may be connected to the computer terminal 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0060] The transmission module 606 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 60. In one example, the transmission module 606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 606 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0061] Under the above operating environment, this application provides Figure 7 The data processing method shown. Figure 7 is a flow chart of a data processing method according to a second embodiment of the present invention.

[0062] Step S702, reading a quantization index from a memory;

[0063] Step S704, restore from the code table according to the quantization index to obtain a quantization vector;

[0064] Step S706, interleaving the request vector and the quantization vector to obtain the quantization vector dimension;

[0065] Step S708, performing calculation according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector;

[0066] Step S710, compressing data according to the distance value.

[0067] Specifically, in combination with step S702 to step S710, the data processing method provided in the embodiment of the present application can be applied to the data processing system in embodiment 1, wherein, in the first data throughput area, i.e., the low throughput area, the quantization index is obtained through the off-chip memory and / or the on-chip memory, and at least one quantization index is distributed to each code table for search through the index distributor, and the quantization index is restored to the corresponding quantization vector. Since the dimensional order of the request vector may be inconsistent with the dimensional order of the quantization vector restored from the code table, in order to match the two, the request vector dimension needs to be interleaved and adjusted. In the high throughput area, i.e., the second data throughput area in the embodiment of the present application, the distance between the request vector and each quantization center point is calculated, and then the result is output to the addition tree for distance accumulation to obtain the final distance between the request vector and the quantization vector.

[0068] In an embodiment of the present invention, an asymmetric quantization method is adopted, by reading a quantization index from a memory; restoring from a code table according to the quantization index to obtain a quantization vector; interleaving a request vector and a quantization vector to obtain a quantization vector dimension; calculating according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data according to the distance value, thereby achieving the purpose of reducing the system's demand for read bandwidth by quantizing and compressing the data set itself, caching in space, and batch processing of request vectors in time, thereby achieving the technical effect of improving the throughput of the system, and further solving the technical problem that the data set reading bandwidth cannot be compressed and the system throughput cannot be improved because traditional quantization compression in the prior art only supports symmetric quantization and the traditional optimization of the retrieval bandwidth of the quantized data set is based on the CPU / GPU system.

[0069] Optionally, reading the quantization index from the memory in step S702 includes:

[0070] Step S7021: when the memory includes an off-chip memory and / or an on-chip memory, read the quantization index from the corresponding memory.

[0071] Specifically, the data processing method provided in the embodiment of the present application will reduce the system's demand for read bandwidth by quantizing and compressing the data set itself, caching in space, and batch processing of request vectors in time, and proposes a corresponding hardware acceleration system, wherein the hardware acceleration system is as follows: Figure 2 As shown, all quantization indexes are stored in the chip external memory, or if there is enough internal memory, some indexes are stored in the chip internal memory, so that the bandwidth pressure of the external memory can be greatly reduced by internal caching. Usually, the codebook occupies a limited storage space, so it is stored in the chip internal memory.

[0072] That is, according to the size of the storage space of the off-chip memory and the on-chip memory, the corresponding memory can be selected according to actual data requirements to implement the storage of the quantization index.

[0073] Optionally, in step S704, restoring from the code table according to the quantization index to obtain the quantization vector includes:

[0074] Step S7041, distribute at least one quantization index to a corresponding code table; restore through the code table according to the quantization index to obtain a quantization vector.

[0075] Specifically, a quantization index is obtained through an off-chip memory and / or an on-chip memory, at least one quantization index is distributed to each code table for search through an index distributor, and the quantization index is restored to a corresponding quantization vector, wherein: Figure 2 Where q(0), q(1), ..., q(m-1) are quantization indices distributed by the index distributor from the off-chip memory and / or the on-chip memory, and y(0), y(1), ..., y(m-1) are restored quantization vectors.

[0076] Optionally, in step S706, the request vector and the quantization vector are interleaved to obtain a quantization vector dimension including:

[0077] Step S7061, querying the sub-code table in the unit memory according to the quantization index to obtain the quantization vector dimension corresponding to the request vector, wherein the code table is stored in at least one unit memory, and the output of the unit memory is arranged in the same order as the dimension of the request vector.

[0078] Specifically, based on Figure 2 As shown in the schematic diagram of the data processing system, since the dimensional order of the request vector may be inconsistent with the dimensional order of the quantization vector restored from the code table, in order to make the two match, it is necessary to interleave the request vector dimension. That is, the interleaving device 24 provided in the embodiment of the present application obtains the input request vector X, and interleaves the request vector X with the above quantization vector, wherein the interleaving adjustment process is as follows Figure 4 As shown, Figure 4 2 is a schematic diagram of interleaving adjustment in a data processing system according to Embodiment 1 of the present invention.

[0079] Among them, a complete code table is stored in several unit memories, and the output of the unit memory is arranged in the original dimension order of the data set / request vector. The input quantization index will be cross-connected to each unit memory for sub-code table search. It can be seen that the unit memories sharing the same index constitute a complete code table. By reconfiguring the index cross connection, this architecture can support arbitrary division and aggregation of dimensions to a certain extent.

[0080] Optionally, in step S708, the distance value between the request vector and the quantization vector is obtained by performing calculation according to the quantization vector dimension, the request vector and the quantization vector, including:

[0081] Step S7081, calculating the distance value between each request vector and the quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector;

[0082] Step S7082: Accumulate the distance values ​​to obtain the distance value between the request vector and the quantized vector.

[0083] Specifically, Figure 2 As shown, in the high throughput area, that is, the second data throughput area in the embodiment of the present application, the request vector is distance calculated with each quantization center point, and then the result is output to the addition tree for distance accumulation to obtain the final distance between the request vector and the quantization vector. By adopting the pipeline design mode, this part of the circuit can work at a higher frequency, or by replicating the distance calculation and the addition tree circuit, this part of the function can work in the high throughput area. Figure 2 As shown in the figure, assuming that the distance calculation module can consume B quantized vectors per clock, but in fact only one quantized vector can be provided per clock cycle, the throughput of the entire system will be limited by the low-throughput area circuit. However, when all request vectors need to traverse all quantized data sets, batch processing can be used to improve the throughput of the entire system. That is, each time a quantized data point is read, the distance calculation is performed with B request vectors, thereby balancing the bandwidth of the two and improving the system throughput.

[0084] Among them, the batch processing method is used to improve the throughput of the entire system, including: if the retrieval of each request vector needs to traverse the entire data set, then the reading of a single quantized data point can be applied to multiple request vector data for distance calculation. Figure 5 FIG. 1 is a schematic diagram of batch processing in a data processing system according to Embodiment 1 of the present invention. Figure 5 As shown, the read quantized data point y is simultaneously calculated with B request vectors, so that the system throughput can be increased to B times the original, where B is the batch size (the number of request vectors that perform distance calculations simultaneously). It is worth noting that in another implementation, the request vectors can also be serial, in which case the working frequency of the distance calculation is required to be high enough to complete the distance calculation with B request vectors before reading the next y.

[0085] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data processing method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0087] Example 3

[0088] According to an embodiment of the present invention, a device for implementing the above data processing method is also provided. Figure 8 is a schematic diagram of a data processing device according to Embodiment 3 of the present invention. Figure 8 As shown, the device comprises:

[0089] A reading module 80 is used to read a quantization index from a memory; a restoration module 82 is used to restore from a code table according to the quantization index to obtain a quantization vector; an interleaving module 84 is used to interleave a request vector with a quantization vector to obtain a quantization vector dimension; a calculation module 86 is used to calculate according to the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; a compression module 88 is used to compress data according to the distance value.

[0090] Optionally, the reading module 80 includes: a reading unit, configured to read the quantization index from a corresponding memory when the memory includes an off-chip memory and / or an on-chip memory.

[0091] Optionally, the calculation module 86 includes: a first calculation unit, used to calculate the distance value between each request vector and the quantization center point where the quantization vector is located based on the quantization vector dimension and the request vector; and a second calculation unit, used to accumulate the distance values ​​to obtain the distance value between the request vector and the quantization vector.

[0092] Example 4

[0093] According to another aspect of an embodiment of the present invention, a storage medium is also provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute reading a quantization index from a memory; restoring from a code table based on the quantization index to obtain a quantization vector; interleaving a request vector and a quantization vector to obtain a quantization vector dimension; calculating based on the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data based on the distance value.

[0094] Example 5

[0095] According to another aspect of an embodiment of the present invention, a processor is provided, and the processor is used to run a program, wherein when the program is running, the quantization index is read from the memory; the quantization index is restored from the code table to obtain the quantization vector; the request vector and the quantization vector are interleaved to obtain the quantization vector dimension; the distance value between the request vector and the quantization vector is calculated based on the quantization vector dimension, the request vector and the quantization vector; and the data is compressed based on the distance value.

[0096] Example 6

[0097] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0098] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0099] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: reading a quantization index from a memory; restoring from a code table based on the quantization index to obtain a quantization vector; interleaving a request vector with a quantization vector to obtain a quantization vector dimension; calculating based on the quantization vector dimension, the request vector, and the quantization vector to obtain a distance value between the request vector and the quantization vector; and compressing data based on the distance value.

[0100] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: reading the quantization index from the memory includes: when the memory includes an off-chip memory and / or an on-chip memory, reading the quantization index from the corresponding memory.

[0101] Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: restoring from a code table according to a quantization index to obtain a quantization vector, including: distributing at least one quantization index to a corresponding code table; restoring through the code table according to the quantization index to obtain a quantization vector.

[0102] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: interleaving the request vector with the quantization vector to obtain the quantization vector dimension includes: querying the sub-code table in the unit memory according to the quantization index to obtain the quantization vector dimension corresponding to the request vector, wherein the code table is stored in at least one unit memory, and the output of the unit memory is arranged in the same order as the dimension of the request vector.

[0103] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing calculations based on the quantization vector dimension, the request vector and the quantization vector to obtain a distance value between the request vector and the quantization vector, including: calculating the distance value between each request vector and a quantization center point where the quantization vector is located based on the quantization vector dimension and the request vector; and accumulating the distance values ​​to obtain the distance value between the request vector and the quantization vector.

[0104] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0105] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for data processing, comprising: Read quantization index from memory; Restoring from the code table according to the quantization index to obtain a quantization vector; Interleaving the request vector with the quantization vector to obtain a quantization vector dimension; Calculate the distance between each request vector and a quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector; Accumulating the distance values ​​to obtain a distance value between the request vector and the quantized vector; Data compression is performed according to the distance value.

2. The method according to claim 1, wherein: The reading of the quantization index from the memory comprises: In the case that the memory includes an off-chip memory and / or an on-chip memory, the quantization index is read from the corresponding memory.

3. The method according to claim 1 or 2, wherein: The restoring from the code table according to the quantization index to obtain the quantization vector includes: Distributing at least one of the quantization indexes to a corresponding code table; The quantization vector is obtained by restoring the quantization index through the code table.

4. The method according to claim 1, wherein: The interleaving of the request vector and the quantization vector to obtain the quantization vector dimension comprises: The sub-code table in the unit memory is queried according to the quantization index to obtain the quantization vector dimension corresponding to the request vector, wherein the code table is stored in at least one of the unit memories, and the output of the unit memory is arranged in the same order as the dimension of the request vector.

5. A data processing device, comprising: A reading module, used for reading the quantization index from the memory; A restoration module, used for restoring from the code table according to the quantization index to obtain a quantization vector; An interleaving module, used for interleaving the request vector with the quantization vector to obtain a quantization vector dimension; A calculation module, used for calculating the distance value between each of the request vectors and the quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector; Accumulating the distance values ​​to obtain a distance value between the request vector and the quantized vector; A compression module is used to perform data compression according to the distance value.

6. The device according to claim 5, wherein: The reading module comprises: A reading unit is used to read the quantization index from a corresponding memory when the memory includes an off-chip memory and / or an on-chip memory.

7. The device according to claim 5, wherein: The calculation module comprises: A first calculation unit, configured to calculate a distance value between each of the request vectors and a quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector; The second calculation unit is used to accumulate the distance values ​​to obtain the distance value between the request vector and the quantization vector.

8. A data processing system, comprising: a first data throughput region, an interleaving device and a second data throughput region, wherein The first data throughput area includes: a memory, an index distributor and at least one code table, wherein the index distributor is connected to the memory and is used to read at least one quantization index and send the at least one quantization index to a corresponding code table; the code table restores the quantization index to obtain a quantization vector; The interleaving device is connected to the first data throughput area and the second data throughput area respectively, and is used to interleave the quantization vector and the request vector, and input the interleaving result into the second data throughput area; The second data throughput area includes: a distance calculator and an addition tree, wherein the distance calculator is used to calculate the distance value between each of the request vectors and the quantization center point where the quantization vector is located; the addition tree is connected to the distance calculator, and is used to accumulate the distance values ​​to obtain the distance values ​​between the request vectors and the quantization vectors, and output the distance values ​​to the sorting circuit.

9. The system according to claim 8, wherein: The memory includes: an off-chip memory and / or an on-chip memory, wherein the off-chip memory and / or the on-chip memory is used to store the quantization index.

10. A storage medium comprising a stored program, wherein: When the program is running, the device where the storage medium is located is controlled to execute reading of the quantization index from the memory; restoring from the code table according to the quantization index to obtain the quantization vector; interleaving the request vector with the quantization vector to obtain the quantization vector dimension; calculating the distance value between each request vector and the quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector; accumulating the distance values ​​to obtain the distance value between the request vector and the quantization vector; and performing data compression according to the distance value.

11. A processor for running a program, wherein: When the program is running, the program executes reading a quantization index from a memory; restoring from a code table according to the quantization index to obtain a quantization vector; interleaving a request vector with the quantization vector to obtain a quantization vector dimension; and calculating a distance value between each request vector and a quantization center point where the quantization vector is located according to the quantization vector dimension and the request vector; Accumulating the distance values ​​to obtain a distance value between the request vector and the quantized vector; Data compression is performed according to the distance value.

Images