Constant space data compression method, device and computer equipment without metadata

Abstract

The present application relates to a constant space data compression method, device and computer equipment that does not require metadata. The method represents high-dimensional data as a vector, and uses the array offset of each element of the vector in the vector storage area as a coordinate index; maps the corresponding vector to a bucket array according to the coordinate index; uses a cumulative function to update the vector elements of each bucket; when querying, the query vector coordinates are processed in the same way as the compression process, the same bucket array is selected, and the bucket array is processed by an aggregation function to obtain the query result. The method supports compressing vectors of arbitrary length into constant space storage through in-situ mapping, updating, and aggregation, and can decouple the relationship between data size and storage space; there is no need to maintain metadata, and the space occupancy of the compressed data structure can be flexibly set according to the computing environment, and is widely applicable to computing environments such as data centers, edge computing, smart phones, and the Internet of Things.

Description

Technical Field

[0001] The present application relates to the technical field of data compression, and in particular to a constant space data compression method, device and computer equipment without metadata. Background Art

[0002] With the rapid development of machine learning and deep learning, machine learning systems have become the infrastructure of the intelligent era. The storage and calculation of big data are the core requirements of machine learning systems. At present, the data of machine learning systems are usually high-order and high-dimensional data types such as matrices and tensors. The scale is huge, which brings challenges to storage space and access efficiency.

[0003] Data compression in machine learning systems is an effective means to improve the storage and computing of big data. Storing compressed data can not only reduce storage pressure, but also select only the required data for processing when needed. It has high flexibility and has therefore received widespread attention in data centers, edge computing, smart Internet of Things and other fields.

[0004] However, current data compression methods are not suitable for machine learning system environments and have deficiencies in storage space and access methods. Currently, lossy compression methods such as vector sparsification and low-order decomposition are mainly used. However, sparsification achieves the purpose of reducing space overhead by discarding a large amount of original data distribution features, and vector coordinate information must be maintained. Low-order decomposition requires the maintenance of metadata and compression values. The metadata and original data are linearly related, and the compression value requires the allocation of additional space. Therefore, existing compression methods cannot effectively reduce space. Summary of the invention

[0005] Based on this, it is necessary to provide a constant space data compression method, device and computer equipment without metadata to address the above technical problems.

[0006] A constant space data compression method without metadata, the method comprising:

[0007] The acquired high-dimensional data is represented as a vector, and the array offset of each element of the vector in the vector storage area is used as the coordinate index.

[0008] A preset method is used to process the coordinate index, and the vector corresponding to the coordinate index is mapped to a bucket array through a modulo operation on the processing result; the bucket array includes multiple buckets, and each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate a hash value, or using a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector.

[0009] The accumulation function is used to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket.

[0010] When querying, the corresponding preset method is used to process the query vector coordinates, and the same bucket array is selected by performing the same modulo operation on the obtained processing results.

[0011] Process the values ​​of the selected bucket array through the aggregation function to obtain the query results.

[0012] In one embodiment, a preset method is used to process the coordinate index, and a vector corresponding to the coordinate index is mapped to a bucket array by a modulo operation on the obtained processing result, including:

[0013] For each element of the vector, a random hash function is used to calculate the hash value according to the coordinate index, and through the modulo operation, 1 or k bucket arrays are selected to map the vector to the bucket array; where k is an integer greater than 0.

[0014] Alternatively, for each element of the vector, a vector storage area layout function is used to directly retrieve the index of the array according to the storage area layout of the vector, and through a modulo operation, 1 or k bucket arrays are selected to map the vector to the bucket array.

[0015] In one of the embodiments, the accumulation function is: an algebraic sum or a weighted algebraic sum.

[0016] In one embodiment, the aggregation function is: minimum value, median value, maximum value or mean value.

[0017] In one embodiment, the accumulation function is an intelligent accumulation function; the intelligent accumulation function is implemented by a Transformer-based encoder; the encoder includes an embedding layer, an encoding layer, and a compression layer; the encoding layer includes multiple Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network.

[0018] Use the accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket, including:

[0019] The input vector X is transformed through the embedding layer, encoding layer, and compression layer, and then stored in the encoding domain of the bucket; the encoding domain of the bucket is:

[0020] ;

[0021] ;

[0022] ;

[0023] in, is the bucket code field, is the encoding feature, and are the weights and biases of the compression layer; where is the embedding feature, is the input vector X, ReLU is the activation function, and are the weights and biases of the embedding layer, The encoding layer.

[0024] Transform each element of the input vector X through the encoder and update the encoding domain of the bucket.

[0025] In one embodiment, the aggregation function is an intelligent aggregation function; the intelligent aggregation function is implemented by a decoder; the decoder includes a decoding embedding layer, a decoding layer and a decompression layer; the decoding layer includes multiple Transformer decoder layers, and each Transformer decoder layer includes a multi-head self-attention mechanism and a feedforward neural network.

[0026] Process the values ​​of the selected bucket array through the aggregation function to obtain the query results, including:

[0027] Select the barrel encoding domain, transform through the decoding embedding layer, decoding layer, and decompression layer, and get the decoded representation as:

[0028] ;

[0029] ;

[0030] ;

[0031] in, To decode the representation, Output features for the decoding layer, and are the weights and biases of the decompression layer, To decode the embedding layer output features, is the bucket code field, and are the weights and biases of the decoder embedding layer, The decoding layer.

[0032] The decoded representation is organized into a vector form to obtain the query result; the order of each element of the vector in the query result is consistent with the order sequence of insertion into the bucket.

[0033] In one embodiment, when the dimension of a multi-dimensional vector is 1, it collapses to 1 count value.

[0034] When the dimension of a multidimensional vector is greater than 1, it represents the fine-grained information of the vector inserted into the bucket; the number of buckets represents the target size that needs to be compressed.

[0035] A constant space data compression device without metadata, the device comprising:

[0036] The data representation module is used to represent the acquired high-dimensional data as a vector, and use the array offset of each element of the vector in the vector storage area as a coordinate index.

[0037] The compression module is used to process according to the coordinate index using a preset method, and map the vector corresponding to the coordinate index to a bucket array through a modulo operation on the obtained processing result; the bucket array includes multiple buckets, each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate the hash value, or using a vector storage area layout function to directly retrieve the index where the array is located according to the storage area layout of the vector; using an accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket.

[0038] The query module is used to process the query vector coordinates using the corresponding preset method during the query, select the same bucket array through the same modulo operation on the processing results, and process the values ​​of the selected bucket array through the aggregation function to obtain the query results.

[0039] In one of the embodiments, the compression module is also used to calculate a hash value for each element of the vector according to the coordinate index using a random hash function, and select 1 or k bucket arrays through a modulo operation to map the vector to the bucket array; k is an integer greater than 0; or, for each element of the vector, use a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector, and select 1 or k bucket arrays through a modulo operation to map the vector to the bucket array.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] The acquired high-dimensional data is represented as a vector, and the array offset of each element of the vector in the vector storage area is used as the coordinate index.

[0042] A preset method is used to process the coordinate index, and the vector corresponding to the coordinate index is mapped to a bucket array through a modulo operation on the processing result; the bucket array includes multiple buckets, and each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate a hash value, or using a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector.

[0043] The accumulation function is used to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket.

[0044] When querying, the corresponding preset method is used to process the query vector coordinates, and the same bucket array is selected through the same modulo operation on the obtained processing results.

[0045] Process the values ​​of the selected bucket array through the aggregation function to obtain the query results.

[0046] The above-mentioned constant space data compression method, device and computer equipment without metadata, the method includes: representing the acquired high-dimensional data as a vector, using the array offset of each element of the vector in the vector storage area as a coordinate index; mapping the corresponding vector to the bucket array according to the coordinate index; using the accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket; when querying, the query vector coordinates are processed in the same way as the compression process, and the same bucket array is selected; the value of the selected bucket array is processed through an aggregation function to obtain the query result. This method supports compressing vectors of arbitrary length into constant space storage through in-situ mapping, updating, and aggregation, and can decouple the relationship between data size and storage space; this method does not need to maintain metadata, and the space occupancy of the compressed data structure can be flexibly set according to the computing environment, and is widely applicable to network computing environments such as data centers, edge computing, smart phones, and the Internet of Things, and has broad adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of a constant space data compression method without metadata in one embodiment;

[0048] Figure 2 is a flow chart of a constant space data compression method without metadata in one embodiment;

[0049] Figure 3 is a structural block diagram of a constant space data compression device without metadata in one embodiment;

[0050] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 As shown, a constant space data compression method without metadata is provided, the method comprising the following steps:

[0053] Step 100: Represent the acquired high-dimensional data as a vector, and use the array offset of each element of the vector in the vector storage area as a coordinate index.

[0054] Specifically, the machine learning system includes training and reasoning processes. Training and reasoning are performed according to the model hierarchy, and the calculation process of each layer involves compressed storage and calculation of input data, model weight matrix, and intermediate activation data. The acquired high-dimensional data is the high-dimensional data of input data, model weight matrix, and intermediate activation data in the training or reasoning process of the machine learning system.

[0055] For high-dimensional data such as matrices and tensors, they are uniformly represented as vectors. Each element of a vector is a tuple (coordinate index, value). The coordinate index is an intrinsic property of the vector and does not need to be maintained separately.

[0056] Step 102: Process the coordinate index using a preset method, and map the vector corresponding to the coordinate index to a bucket array through a modulo operation on the processing result; the bucket array includes multiple buckets, each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate a hash value, or using a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector.

[0057] Specifically, the vector storage area layout function refers to a layout in a priority manner according to a certain dimension of the coordinates.

[0058] The number of buckets represents the target size that needs to be compressed. Reducing the number of buckets directly improves the compression effect; conversely, it reduces the compression effect.

[0059] First, for each element of the vector, a random hash function is selected, and the hash value is calculated according to the coordinate index. Through the modulo operation, one or k bucket arrays are selected; or, for each element of the vector, according to the coordinate index, a vector storage area layout function is used (layout is performed in a priority manner according to a certain dimension of the coordinate) to directly retrieve the index of the array according to the storage area layout of the vector, and one or k bucket arrays are selected through the modulo operation.

[0060] The fixed-length multidimensional vector contained in each bucket is called the compressed domain.

[0061] Step 104: Use an accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array into the selected bucket.

[0062] Specifically, for the mapped vector elements, the vector values ​​are accumulated into the selected buckets; simple accumulation functions include: SUM (algebraic sum), Signed SUM (weighted algebraic sum), and complex accumulation functions are accumulated through models. Selecting different accumulation functions affects the accuracy. Accumulation functions based on intelligent models can be more faithful, but the required calculations are increased.

[0063] Step 106: When querying, a corresponding preset method is used to process the query vector coordinates, and the same bucket array is selected through the same modulo operation on the obtained processing results.

[0064] Specifically, the compressed data structure supports in-situ queries, that is, users can directly query the value of any position in the vector on the bucket array.

[0065] Mapping: When querying, select the same random hash function or vector storage area layout function as the compression process, calculate the hash value or directly retrieve the array index according to the vector storage area layout, and select the same bucket array through the same modulus operation.

[0066] The corresponding preset method means that the preset method used in the query process is the same as that used in the compression process. That is, if the compression process uses the method of calculating the hash value, the random hash function selected in the query process is the same as the random hash function selected in the compression process; if the vector storage area layout function is used in the compression process, the index of the array is directly retrieved according to the storage area layout of the vector, then the same vector storage area layout function is used in the query process.

[0067] Step 108: Process the values ​​of the selected bucket array through an aggregate function to obtain a query result.

[0068] Specifically, aggregation: Return the value of the bucket through an aggregation function. Simple aggregation functions include: MIN (minimum value), MEDIAN (middle value), MAX (maximum value), MEAN (mean value). More intelligent aggregation functions use a decoder neural network structure to make queries more accurate. Choosing different aggregation functions affects accuracy and computational efficiency.

[0069] In the above constant space data compression method without metadata, the method includes: representing the acquired high-dimensional data as a vector, using the array offset of each element of the vector in the vector storage area as a coordinate index; mapping the corresponding vector to the bucket array according to the coordinate index; using the accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket; when querying, the query vector coordinates are processed in the same way as the compression process, and the same bucket array is selected; the value of the selected bucket array is processed through an aggregation function to obtain the query result. This method supports compressing vectors of arbitrary length into constant space storage through in-situ mapping, updating, and aggregation, and can decouple the relationship between data size and storage space; this method does not need to maintain metadata, and the space occupancy of the compressed data structure can be flexibly set according to the computing environment, and is widely applicable to network computing environments such as data centers, edge computing, smart phones, and the Internet of Things, and has broad adaptability.

[0070] In one embodiment, step 102 includes: for each element of the vector, using a random hash function to calculate a hash value according to the coordinate index, and selecting 1 or k bucket arrays through a modulo operation to map the vector to the bucket array; where k is an integer greater than 0; or, for each element of the vector, using a vector storage area layout function, directly retrieving the index of the array according to the storage area layout of the vector, and selecting 1 or k bucket arrays through a modulo operation to map the vector to the bucket array.

[0071] In one embodiment, the accumulation function in step 104 is: algebraic sum or weighted algebraic sum.

[0072] Specifically, algebraic and SUM: x<-x+v, for i in mapping bucket;

[0073] Weighted algebraic sum Signed SUM: x<-x + sign(vector element index coordinate associated with i)*v, for iin mapping bucket;

[0074] Smart accumulation function: x<-Enc(x) , for i in mapping bucket.

[0075] In one embodiment, the aggregation function in step 108 is: minimum value, median value, maximum value or mean value.

[0076] In one embodiment, the accumulation function is an intelligent accumulation function; the intelligent accumulation function is implemented by a Transformer-based encoder; the encoder includes an embedding layer, an encoding layer, and a compression layer; the encoding layer includes multiple Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network; step 104 includes: transforming the input vector X through the embedding layer, the encoding layer, and the compression layer, and then storing it in the encoding domain of the bucket; the encoding domain of the bucket is:

[0077] ;

[0078] ;

[0079] ;

[0080] in, is the encoding domain of the barrel, ReLU is the activation function, is the encoding feature, and are the weights and biases of the compression layer; where is the embedding feature, is the input vector X, and are the weights and biases of the embedding layer, The encoding layer.

[0081] Transform each element of the input vector X through the encoder and update the encoding domain of the bucket.

[0082] Specifically, accumulation is the process of converting a vector sequence into a compressed representation, corresponding to the encoding stage.

[0083] The accumulation function and aggregation function are implemented using the autoencoder.

[0084] Input: For cumulative functions, the input corresponds to a sequence of vector values ​​mapped to buckets; for aggregate functions, the input corresponds to multiple predictions of the same value. Let input be X.

[0085] The model structure of the autoencoder includes an encoder and a decoder. The encoder converts X into Enc(X) through encoding operations; the decoder reads the encoded Enc(X) and converts it into Dec(Enc(X)) through decoding operations. The decoding result Dec(Enc(X)) is as close to the original sequence X as possible.

[0086] The encoder is a multi-layer neural network structure. Each layer selects MLP and TransformerEncoder by default, and the encoding structure can also be customized. Among them, MLP includes two levels of fully connected linear layers and activation layers (the default is ReLU activation function). TransformerEncoder is a Transformer encoder, including a multi-head self-attention layer MHA and a forward layer FFN. MHA is responsible for self-attention calculation of the sequence, and improves the generalization performance of self-attention through multiple heads; FFN includes 2 layers of linear layers and activation layers. The following uses the Transformer encoder layer to explain the structure of the encoder. The encoder includes: embedding layer, encoding layer and compression layer.

[0087] (1) Embedding layer: A linear activation layer is used to convert the input into the encoding feature space (i.e., convert the input x into the latent space): , where ReLU is the activation function, and are the weights and biases of the embedding layer.

[0088] (2) Encoding layer: used for context feature conversion after embedding. It consists of multiple Transformer encoder layers. Each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The specific representation of the Transformer encoder layer is:

[0089] The stacking of multiple Transformer encoders is implemented using nn.TransformerEncoder. A Transformer encoder is usually composed of multiple nn.TransformerEncoderLayer stacks, each of which contains a self-attention mechanism and a feedforward neural network. Specifically:

[0090] nn.TransformerEncoder( nn.TransformerEncoderLayer( d_model=hidden_dim, nhead=num_heads, dim_feedforward=hidden_dim * 4, dropout=dropout_val, activation='relu'), num_layers=num_layers_val);

[0091] Among them, d_model=hidden_dim: specifies the feature dimension of input and output, that is, the dimension of hidden layer, the default is 64; nhead=num_heads: specifies the number of heads in multi-head attention mechanism, the default is 8; dim_feedforward=hidden_dim * 4: specifies the dimension of feedforward neural network, usually 4 times of hidden layer dimension; dropout=dropout_val: specifies dropout probability to prevent overfitting, the default is 0.1; activation='relu': sets the activation function of feedforward neural network, the default is ReLU activation function ; num_layers = num_layers_val: specifies the number of encoder layers contained in the Transformer encoder, the default is 5.

[0092] (3) Compression layer: A linear activation layer is used to convert the encoding layer into the encoding domain of the bucket. , where ReLU is the activation function, and are the weights and biases of the embedding layer.

[0093] In one of the embodiments, the aggregation function is an intelligent aggregation function (the intelligent aggregation function is an aggregation function implemented using an autoencoder, intelligent aggregation function: x < Dec(Enc(X)), jointly solving the encoding Enc(X) of the associated buckets); the intelligent aggregation function is implemented through a decoder; the decoder includes a decoding embedding layer, a decoding layer, and a decompression layer; the decoding layer includes multiple Transformer decoder layers, and each Transformer decoder layer includes a multi-head self-attention mechanism and a feed-forward neural network. Step 108 includes: selecting a bucket encoding domain, and through conversion by the decoding embedding layer, the decoding layer, and the decompression layer, the decoded representation is obtained as:

[0094] ;

[0095] ;

[0096] ;

[0097] wherein, is the decoded representation, is the output feature of the decoding layer, and are the weights and biases of the decompression layer, is the output feature of the decoding embedding layer, is the encoding domain of the bucket (i.e., the output feature of the compression layer), ReLU is the activation function, and are the weights and biases of the decoding embedding layer, is the decoding layer.

[0098] Arrange the decoded representation into a vector form to obtain the query result; where the order of the elements of the vector in the query result is consistent with the order sequence inserted into the bucket.

[0099] Specifically, aggregation is to decode the encoding result back to the original value, corresponding to the decoding stage.

[0100] The decoder is a multi-layer neural network structure. By default, MLP and TransformerEncoder are selected, and the encoding structure can also be customized. Among them, MLP and TransformerEncoder are defined in the same way as the encoder and can also be adaptively adjusted.

[0101] (1) Decoding embedding layer: Adopt a linear activation layer to convert the encoding into the decoding feature space: , where ReLU is the activation function, and are the weights and biases of the embedding layer.

[0102] (2) Decoding layer: responsible for decoding the embedded context feature transformation, consisting of multiple Transformer decoder layers. Each Transformer decoder layer includes a multi-head self-attention mechanism and a feedforward neural network. .

[0103] The stacking of multiple Transformer decoder layers is implemented using nn.TransformerDecoder. Specifically:

[0104] nn.TransformerDecoder(nn.TransformerDecoderLayer(d_model=hidden_dim,nhead=num_heads, dim_feedforward=hidden_dim * 4, dropout=dropout_val,activation='relu'),num_layers= num_layers_val)

[0105] Among them, d_model=hidden_dim: specifies the feature dimension of input and output, that is, the dimension of hidden layer, the default is 64; nhead=num_heads: specifies the number of heads in multi-head attention mechanism, the default is 8; dim_feedforward=hidden_dim * 4: specifies the dimension of feedforward neural network, usually 4 times of hidden layer dimension; dropout=dropout_val: specifies dropout probability to prevent overfitting, the default is 0.1; activation='relu': sets the activation function of feedforward neural network, the default is ReLU activation function ; num_layers = num_layers_val: specifies the number of encoder layers contained in the Transformer encoder, the default is 5.

[0106] (3) Decompression layer: A linear layer is used to convert the decoded embedding into the original sequence, that is: ,in and are the weights and biases of the decompression layer, respectively.

[0107] Specifically, this method does not need to maintain the metadata of the dataset, but only needs to configure the hyperparameters and the model parameters of the encoder and decoder. The system's hyperparameters include the dimension of the encoding feature space, the number of network layers, the activation function type, the encoding compression dimension, and the training rounds. The system's hyperparameters are obtained through ablation experiments, and the default parameter configuration is the dimension of the encoding feature space = 64, the number of network layers = 3, the activation function type = ReLU, the encoding compression dimension = 3, and the training rounds = 400.

[0108] The autoencoder consisting of an encoder and a decoder is constructed through pre-training. The specific process includes: first, generating representative data samples: samples are obtained by sampling real data sets, and data sets can be synthesized from typical data distributions to improve the distribution and generalization level of data; second, constructing the partitioning of the data set, training the model through the training set, testing the convergence of the model through the validation set, and finally verifying the effect of the data set through the test set. Third, constructing a loss function based on the mean square error MSE: predicted value-true value. Fourth, iterative training, recording the best model parameters to the file system.

[0109] In one of the embodiments, when the dimension of the multidimensional vector is 1, it collapses to 1 count value; when the dimension of the multidimensional vector is greater than 1, it represents fine-grained information of the vector inserted into the bucket; the number of buckets represents the target size that needs to be compressed.

[0110] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0111] In one embodiment, Figure 3 As shown, a constant space data compression device without metadata is provided, including: a data representation module, a compression module and a query module, wherein:

[0112] The data representation module is used to represent the acquired high-dimensional data as a vector, and use the array offset of each element of the vector in the vector storage area as a coordinate index.

[0113] The compression module is used to process according to the coordinate index using a preset method, and map the vector corresponding to the coordinate index to a bucket array through a modulo operation on the obtained processing result; the bucket array includes multiple buckets, each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate the hash value, or using a vector storage area layout function to directly retrieve the index where the array is located according to the storage area layout of the vector; using an accumulation function to accumulate the values ​​of the vector elements of each bucket in the bucket array to the selected bucket.

[0114] The query module is used to process the query vector coordinates using the corresponding preset method during the query, select the same bucket array through the same modulo operation on the processing results, and process the values ​​of the selected bucket array through the aggregation function to obtain the query results.

[0115] In one of the embodiments, the compression module is also used to calculate a hash value for each element of the vector according to the coordinate index using a random hash function, and select 1 or k bucket arrays through a modulo operation to map the vector to the bucket array; k is an integer greater than 0; or, for each element of the vector, use a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector, and select 1 or k bucket arrays through a modulo operation to map the vector to the bucket array.

[0116] In one of the embodiments, the accumulation function in the compression module is: algebraic sum or weighted algebraic sum.

[0117] In one embodiment, the aggregation function in the query module is: minimum value, median value, maximum value or mean value.

[0118] In one embodiment, the accumulation function is an intelligent accumulation function; the intelligent accumulation function is implemented by a Transformer-based encoder; the encoder includes an embedding layer, an encoding layer, and a compression layer; the encoding layer includes multiple Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network; the compression module is further used to transform the input vector X through the embedding layer, the encoding layer, and the compression layer, and then store it in the encoding domain of the bucket; the encoding domain of the bucket is:

[0119] ;

[0120] ;

[0121] ;

[0122] in, is the encoding domain of the barrel, ReLU is the activation function, is the encoding feature, and are the weights and biases of the compression layer; where is the embedding feature, is the input vector X, and are the weights and biases of the embedding layer, The encoding layer.

[0123] Transform each element of the input vector X through the encoder and update the encoding domain of the bucket.

[0124] In one embodiment, the aggregation function is an intelligent aggregation function; the intelligent aggregation function is implemented by a decoder; the decoder includes a decoding embedding layer, a decoding layer, and a decompression layer; the decoding layer includes multiple Transformer decoder layers, each Transformer decoder layer includes a multi-head self-attention mechanism and a feedforward neural network. The query module is also used to select the barrel encoding domain, and the decoding representation is obtained by converting the decoding embedding layer, the decoding layer, and the decompression layer:

[0125] ;

[0126] ;

[0127] ;

[0128] in, To decode the representation, Output features for the decoding layer, and are the weights and biases of the decompression layer, To decode the embedding layer output features, is the encoding domain of the barrel, ReLU is the activation function, and are the weights and biases of the decoder embedding layer, The decoding layer.

[0129] The decoded representation is organized into a vector form to obtain the query result; the order of each element of the vector in the query result is consistent with the order sequence of insertion into the bucket.

[0130] In one of the embodiments, when the dimension of the multidimensional vector is 1, it collapses to 1 count value; when the dimension of the multidimensional vector is greater than 1, it represents fine-grained information of the vector inserted into the bucket; the number of buckets represents the target size that needs to be compressed.

[0131] For the specific definition of the constant space data compression device without metadata, please refer to the definition of the constant space data compression method without metadata mentioned above, which will not be repeated here. Each module in the above-mentioned constant space data compression device without metadata can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0132] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a constant space data compression method without metadata is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0133] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A constant space data compression method without metadata, characterized in that: The method comprises: The acquired high-dimensional data is represented as a vector, and the array offset of each element of the vector in the vector storage area is used as a coordinate index; Processing is performed according to the coordinate index using a preset method, and the vector corresponding to the coordinate index is mapped to a bucket array through a modulo operation on the obtained processing result; the bucket array includes a plurality of buckets, each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate a hash value, or using a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector; Accumulate the value of the vector element of each bucket in the bucket array into the selected bucket using an accumulation function; When querying, the corresponding preset method is used to process the query vector coordinates, and the same bucket array is selected by performing the same modulo operation on the obtained processing results; Process the values ​​of the selected bucket array through the aggregation function to obtain the query results.

2. The constant space data compression method without metadata according to claim 1, characterized in that: According to the coordinate index, a preset method is used to process the obtained processing result, and the vector corresponding to the coordinate index is mapped to the bucket array through a modulo operation, including: For each element of the vector, a random hash function is used to calculate a hash value according to the coordinate index, and one or k bucket arrays are selected through a modulo operation to map the vector to the bucket array; where k is an integer greater than 0; Alternatively, for each element of the vector, a vector storage area layout function is used to directly retrieve the index of the array according to the storage area layout of the vector, and through a modulo operation, 1 or k bucket arrays are selected to map the vector to the bucket array.

3. The constant space data compression method without metadata according to claim 1, characterized in that: The cumulative function is: algebraic sum or weighted algebraic sum.

4. The constant space data compression method without metadata according to claim 1, characterized in that: The aggregation function is: minimum value, median value, maximum value or mean value.

5. The constant space data compression method without metadata according to claim 1, characterized in that: The accumulation function is an intelligent accumulation function; the intelligent accumulation function is implemented by a Transformer-based encoder; the encoder includes an embedding layer, an encoding layer, and a compression layer; the encoding layer includes multiple Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network; The accumulation function is used to accumulate the value of the vector element of each bucket in the bucket array to the selected bucket, including: The input vector X is transformed through the embedding layer, the encoding layer, and the compression layer, and then stored in the encoding domain of the bucket; the encoding domain of the bucket is: ； ； ； in, is the encoding domain of the barrel, ReLU is the activation function, is the encoding feature, and are the weights and biases of the compression layer; where is the embedding feature, is the input vector X, and are the weights and biases of the embedding layer, is the coding layer; Transform each element of the input vector X through the encoder and update the encoding domain of the bucket.

6. The constant space data compression method without metadata according to claim 5, characterized in that: The aggregation function is an intelligent aggregation function; the intelligent aggregation function is implemented by a decoder; the decoder includes a decoding embedding layer, a decoding layer and a decompression layer; the decoding layer includes a plurality of Transformer decoder layers, each Transformer decoder layer includes a multi-head self-attention mechanism and a feedforward neural network; Process the values ​​of the selected bucket array through the aggregation function to obtain the query results, including: Select the barrel encoding domain, transform through the decoding embedding layer, decoding layer, and decompression layer, and get the decoded representation as: ； ； ； in, To decode the representation, Output features for the decoding layer, and are the weights and biases of the decompression layer, To decode the embedding layer output features, is the bucket encoding domain, and are the weights and biases of the decoder embedding layer, is the decoding layer; The decoded representation is organized into a vector form to obtain a query result; wherein the order of each element of the vector in the query result is consistent with the order sequence of insertion into the bucket.

7. The constant space data compression method without metadata according to claim 1, characterized in that: When the dimension of a multidimensional vector is 1, it collapses to 1 count value; When the dimension of a multidimensional vector is greater than 1, it represents the fine-grained information of the vector inserted into the bucket; the number of buckets represents the target size that needs to be compressed.

8. A constant space data compression device without metadata, characterized in that: The device comprises: A data representation module, used to represent the acquired high-dimensional data as a vector, and use the array offset of each element of the vector in the vector storage area as a coordinate index; A compression module is used to process the coordinate index using a preset method, and map the vector corresponding to the coordinate index to a bucket array through a modulo operation on the obtained processing result; the bucket array includes multiple buckets, and each bucket includes a multidimensional vector of a fixed length; the preset method is: using a random hash function to calculate a hash value, or using a vector storage area layout function to directly retrieve the index where the array is located according to the storage area layout of the vector; using an accumulation function to accumulate the value of the vector element of each bucket in the bucket array to a selected bucket; The query module is used to process the query vector coordinates using the corresponding preset method during the query, select the same bucket array through the same modulus operation for the obtained processing results, and process the values ​​of the selected bucket array through an aggregation function to obtain the query results.

9. The constant space data compression device without metadata according to claim 8, characterized in that: The compression module is also used to calculate the hash value of each element of the vector according to the coordinate index using a random hash function, select 1 or k bucket arrays through a modulo operation, and map the vector to the bucket array; k is an integer greater than 0; or, for each element of the vector, use a vector storage area layout function to directly retrieve the index of the array according to the storage area layout of the vector, select 1 or k bucket arrays through a modulo operation, and map the vector to the bucket array.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the constant space data compression method without metadata described in any one of claims 1 to 7 is implemented.

Images