Method and system for matrix compression based on product quantization

Abstract

Methods and systems for compressing a matrix are described. A matrix having a plurality of rows formed from a respective plurality of vectors is partitioned into a plurality of sub-matrices, each sub-matrix containing a respective set of sub-vectors from one or more consecutive columns of the matrix. For each given sub-matrix, the sub-vectors are clustered into a plurality of clusters. For each given cluster, a centroid and a variance are computed and stored based on the sub-vectors belonging to the given cluster. A mapping of each vector to a respective cluster in each sub-matrix is stored. The stored centroids, the stored variances, and the stored mappings form a compressed data set for reconstructing the matrix.

Description

Technical Field

[0001] This disclosure relates to methods and systems for compressing matrices, including methods and systems for compressing matrices using product quantization, and more particularly, for natural language processing applications. Background Technology

[0002] Product quantization (PQ) is a vector quantization technique that decomposes a high-dimensional vector space into low-dimensional subspaces (or clusters) and quantizes each cluster (e.g., by using the centroid of each cluster as a representation of all elements of the cluster). PQ has been used to provide a faster approximate nearest neighbor search. PQ is also applied in areas such as information retrieval and embedding matrix compression in natural language processing (NLP). In NLP, an embedding matrix is ​​a matrix containing multiple embedding vectors, where each embedding vector is a vector representation of the corresponding word in a corpus. Embedding matrices can be used to map input data (e.g., one-hot vector representations of words in a corpus) to embedding vectors, enabling machine learning algorithms to learn NLP tasks, such as translation. The ability to compress embedding matrices allows for more efficient use of memory and processing resources. In this disclosure, compression of the embedding matrix means using fewer bits of data to represent the original embedding matrix and recovering the embedding matrix from it (possibly with some acceptable information loss).

[0003] However, existing PQ techniques for compressing embedding matrices do not account for the uneven distribution of tokens in the corpus vocabulary, which can lead to unsatisfactory compression ratios. PQ uses cluster centroids as a fixed representation of all elements within a cluster, which may result in the loss of information about individual elements within the cluster.

[0004] Providing a method for compressing the embedding matrix that better preserves the information from the original embedding matrix would be useful. Summary of the Invention

[0005] This disclosure provides exemplary methods and systems for PQ-based matrix compression. Some drawbacks of conventional PQ compression techniques, particularly for NLP applications, are addressed through the examples described herein.

[0006] Various examples are described, illustrating techniques for preserving information about the variance of elements within a cluster in compressed data. One stochastic compression method is described, in which the centroid and variance of each cluster are stored, and the reconstruction of each cluster can be performed by sampling from a Gaussian distribution having the centroid and variance of the cluster.

[0007] In some examples, the importance of certain vectors in the matrix relative to other vectors is considered. A weighted K-means clustering method is described, where the elements of a cluster are weighted according to their relative importance. The weights can be calculated based on the frequency of each element in the source dataset (e.g., the frequency of each word in a corpus) or based on the Euclidean norm of the vector corresponding to each element (e.g., where each element is a subvector of the vector in the matrix).

[0008] The examples described in this paper may help address the shortcomings of conventional PQ compression techniques, particularly for NLP applications. For instance, the examples described can be used to perform non-uniform quantization, which enables compression that considers the relative importance of different embedding vectors based on appropriate weights. Weighting each embedding vector using the Euclidean norm may be more useful for NLP applications than simply weighting based on word frequencies in the corpus.

[0009] Although described in the context of embedding matrices, this may be useful for training neural networks to perform NLP tasks, but the examples described in this paper can be used to compress any matrix, which can be used for any neural network.

[0010] In some exemplary aspects, this disclosure describes a computing system including: a memory; and a processing device in communication with the memory. The processing device is configured to execute instructions to cause the computing system to: partition a matrix having multiple rows formed by corresponding multiple vectors into multiple submatrices, each submatrix containing subvectors from a corresponding group of one or more consecutive columns of the matrix; for each given submatrix, cluster the subvectors of the given submatrix into multiple clusters; for each given cluster, compute and store centroids and variances based on the subvectors belonging to the given cluster; and store a mapping from each vector to a corresponding cluster in each submatrix. The stored centroids, the stored variances, and the stored mappings form a compressed dataset for reconstructing the matrix.

[0011] In any exemplary aspect of the computing system, the matrix may be an embedding matrix, and the plurality of vectors may be a plurality of embedding vectors, each embedding vector being a vector representation of a corresponding subword in the vocabulary of the corpus.

[0012] In any exemplary aspect of the computing system, the matrix may be a layer of a neural network, and the plurality of vectors may be trainable parameters of the neural network.

[0013] In any exemplary aspect of the computing system, the processing device may be configured to execute the instructions to cause the computing system to cluster the subvectors using K-means clustering.

[0014] In any exemplary aspect of the computing system, the processing device may be configured to execute the instructions to cause the computing system to cluster the subvectors using weighted K-means clustering. The centroid of each given cluster may be calculated using element-weighted values ​​to weight each subvector of the given cluster.

[0015] In any exemplary aspect of the computing system, each vector of the matrix may represent a corresponding subword in the source dataset, and the element weights of each corresponding subvector of the given cluster may be defined based on the frequency of the corresponding subword in the source dataset.

[0016] In any exemplary aspect of the computing system, the element weights of each corresponding subvector of the given cluster may be defined based on the Euclidean norm of the corresponding vector in the matrix.

[0017] In any exemplary aspect of the computing system, the processing device may be used to execute the instructions to cause the computing system to: transfer the compressed dataset to another device, the compressed dataset being used as input for performing a machine learning task.

[0018] In some exemplary aspects, this disclosure describes a computing system including: a memory; and a processing device communicating with the memory. The processing device is configured to execute instructions to cause the computing system to: obtain a compressed dataset for reconstructing a matrix, the compressed dataset including a centroid set and a variance set, each centroid associated with a corresponding cluster and each variance associated with a corresponding cluster, the compressed dataset also including a mapping of each subvector of the matrix to a corresponding cluster; for each given cluster, generating reconstructed centroids by sampling from a distribution generated using the centroids and variances associated with the given cluster; reconstructing each given subvector using the reconstructed centroids generated for related clusters, the related clusters of the given subvectors being identified using the mapping; and reconstructing the matrix by concatenating the reconstructed subvectors.

[0019] In any exemplary aspect of the computing system, the distribution may be a multivariate Gaussian distribution.

[0020] In any exemplary aspect of the computing system, the compressed dataset may be obtained in communication from another device.

[0021] In any exemplary aspect of the computing system, the matrix may be an embedded matrix.

[0022] In some exemplary aspects, this disclosure describes a method for compressing a matrix, the method comprising: dividing a matrix having a plurality of rows formed by corresponding plurality of vectors into a plurality of submatrices, each submatrice containing subvectors from a corresponding group of one or more consecutive columns of the matrix; for each given submatrice, clustering the subvectors of the given submatrices into a plurality of clusters; for each given cluster, calculating and storing centroids and variances based on the subvectors belonging to the given cluster; and storing a mapping from each vector to a corresponding cluster in each submatric. The stored centroids, the stored variances, and the stored mappings form a compressed dataset for reconstructing the matrix.

[0023] In any exemplary aspect of the method, the matrix may be an embedding matrix, and the plurality of vectors may be a plurality of embedding vectors, each embedding vector being a vector representation of a corresponding subword in the vocabulary of the corpus.

[0024] In any exemplary aspect of the method, the matrix may be a layer of a neural network, and the plurality of vectors may be trainable parameters of the neural network.

[0025] In any exemplary aspect of the method, the subvectors may be clustered using K-means clustering.

[0026] In any exemplary aspect of the method, the subvectors may be clustered using weighted K-means clustering, and the centroid of each given cluster may be calculated using element weights to weight each subvector of the given cluster.

[0027] In any exemplary aspect of the method, each vector of the matrix may represent a corresponding subword in the source dataset, and the element weights of each corresponding subvector of the given cluster may be defined based on the frequency of the corresponding subword in the source dataset.

[0028] In any exemplary aspect of the method, the element weights of each corresponding subvector of the given cluster may be defined based on the Euclidean norm of the corresponding vector in the matrix.

[0029] In any exemplary aspect of the method, the method may include: transmitting the compressed dataset to another device, the compressed dataset being used as input for performing a machine learning task.

[0030] In some exemplary aspects, this disclosure describes a method comprising: obtaining a compressed dataset for reconstructing a matrix, the compressed dataset comprising a centroid set and a variance set, each centroid associated with a corresponding cluster and each variance associated with a corresponding cluster, the compressed dataset further comprising a mapping of each subvector of the matrix to a corresponding cluster; for each given cluster, generating reconstructed centroids by sampling from a distribution generated using the centroids and variances associated with the given cluster; reconstructing each given subvector using the reconstructed centroids generated for associated clusters, the associated clusters of the given subvectors being identified using the mapping; and reconstructing the matrix by concatenating the reconstructed subvectors.

[0031] In any exemplary aspect of the method, the distribution may be a multivariate Gaussian distribution.

[0032] In any exemplary aspect of the method, the compressed dataset may be obtained in communication from another device.

[0033] In any exemplary aspect of the method, the matrix may be an embedding matrix.

[0034] In some exemplary aspects, this disclosure describes a computer-readable medium that may include instructions for causing a computing system to perform any of the exemplary aspects of the methods described above. Attached Figure Description

[0035] The exemplary embodiments of this application will now be illustrated by way of example in the accompanying drawings, in which:

[0036] Figure 1 This is a block diagram of an exemplary computing system that can be used to implement the exemplary embodiments described herein;

[0037] Figure 2 An example is shown on how to process subwords in a corpus into embedding vectors of an embedding matrix;

[0038] Figure 3 An example of how clusters can be formed in a PQ-based compression technique is shown;

[0039] Figure 4 This is a flowchart of an exemplary disclosed method for compressing matrices;

[0040] Figure 5 This is a flowchart of an exemplary public method for reconstructing a matrix;

[0041] Figure 6 It shows Figure 4 and Figure 5 An exemplary implementation of the compression and reconstruction method.

[0042] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0043] In the exemplary embodiments disclosed herein, methods and systems capable of more effectively compressing embedding matrices are described, particularly for natural language processing (NLP) and speech-related applications. For machine learning algorithms (e.g., neural networks) to accurately perform learned NLP tasks (e.g., machine language translation, machine text generation, machine text analysis, machine text filtering, machine answer generation, etc.), it is important to provide the machine learning algorithms with data (e.g., training data during the training phase or input data during the inference phase) that provides accurate and sufficient information (i.e., “high-quality” data). However, the size of the data that can be input to the machine learning algorithm performing the NLP task (e.g., in terms of the number of words in the vocabulary) may be limited by the memory and / or processing resources of the computing device implementing the machine learning algorithm (e.g., a handheld or mobile computing device). Data compression can be performed to accommodate these resource limitations. A trade-off typically exists between the amount of compression and the quality of the recovered data reconstructed from the compressed data. In various examples, this disclosure describes methods and systems for compressing embedding matrices that can achieve a better compression-to-quality ratio.

[0044] To aid in understanding this disclosure, we will first discuss Figure 1 .

[0045] Figure 1 This is a block diagram illustrating a simplified exemplary implementation of a computing system 100 suitable for implementing the embodiments described herein. Examples of this disclosure can be implemented in other computing systems that may include components different from those discussed below. For example, in some examples, computing system 100 may be an artificial intelligence (AI) chip, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA) that includes circuitry for compressing and embedding matrices according to the examples described herein.

[0046] although Figure 1 A single instance of each component is shown, but there may be multiple instances of each component in the computing system 100.

[0047] For example, computing system 100 may be a server-side device including one or more servers located in a data center, or it may be a cloud computing system, wherein the embedding matrix can be compressed according to the examples described herein. In some examples, computing system 100 may be a user device, such as a client device / terminal, user equipment / device (UE), mobile station (STA), smartphone, laptop, computer, tablet, smart device, machine-type communication device, smart (or connected) vehicle, or consumer electronics device, and other possible devices. In some examples, computing system 100 may be an edge computing device.

[0048] The computing system 100 may include one or more processing devices 114, such as processors, microprocessors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated logic circuits, dedicated artificial intelligence processing units, tensor processing units, neural processing units, hardware accelerators, or combinations thereof. The computing system 100 may also include one or more optional input / output (I / O) interfaces 116, which may allow connection to one or more optional input devices 118 and / or optional output devices 120.

[0049] In the example shown, one or more input devices 118 (e.g., keyboard, mouse, microphone, touchscreen, and / or keypad) and one or more output devices 120 (e.g., display, speaker, and / or printer) are shown as optional and external to the server. In other exemplary embodiments, there may be no input devices 118 and output devices 120, in which case one or more I / O interfaces 116 may not be required.

[0050] The computing system 100 may include one or more network interfaces 122 for wired or wireless communication with other computing devices in the network. The one or more network interfaces 122 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication.

[0051] The computing system 100 may also include one or more storage units 124, which may include high-capacity storage units such as solid-state drives, hard disk drives, disk drives and / or optical disk drives.

[0052] The computing system 100 may include one or more memories 128, which may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transitory memories 128 may store instructions for execution by one or more processing devices 114, such as performing the exemplary embodiments described in this disclosure. The one or more memories 128 may include other software instructions, such as software instructions for implementing an operating system and other applications / functions.

[0053] In some exemplary embodiments, one or more storage units 124 (or one or more memories 128) may include software instructions for execution by processing device 114 to implement module 150 for product-quantization-based compression (e.g., as a preprocessor for compression), as discussed further below. In some exemplary embodiments, one or more memories 128 may alternatively or additionally include software instructions for execution by processing device 114 to implement module 160 for product-quantization-based decompression, as discussed further below. In some exemplary embodiments, computing system 100 may additionally or alternatively execute instructions from external memory (e.g., an external drive connected to a server via wired or wireless communication), or may provide executable instructions from transient or non-transitory computer-readable media. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage.

[0054] To help understand this disclosure, some terms used in NLP will now be discussed.

[0055] Figure 2 This section illustrates a simplified example of how subwords from a corpus can be processed into embedding vectors and subsequently used to form an embedding matrix. A corpus is a collection of text documents representing a linguistic domain (typically a single language). A corpus consists of multiple words, and these words are often related to each other based on their relative positions within a word sequence (e.g., within a sentence). A corpus can also include non-words such as symbols (e.g., “?”, “!”, and other punctuation marks), spaces, or numeric characters.

[0056] Tokenization is an NLP technique that separates a corpus into units called subwords. Subwords are typically words, while non-words are discarded. Figure 2 The example shows the subword 202 "dog". Subword 202 can be encoded as a sparse vector representation, such as a one-hot vector 204. In one-hot vector 204, there is only one non-zero entry (typically with a value of "1") at the index corresponding to subword 202, and all other entries are zero. One-hot vector 204 can be further processed into a dense vector representation of subword 202, called embedding vector 206 (or simply embedding). Embedding vector 206 is a vector representation of subword 202 that is denser than one-hot vector 204 (i.e., has more non-zero values) and allows semantically related subwords to be closer to each other in the vector space (i.e., the set of embedding vectors). Unsupervised representation learning (e.g., an autoencoder) can be used to learn the embedding vector 206 for a given subword 202 (e.g., learn the weights of each entry or dimension in embedding vector 206). Figure 2 In the diagram, embedding vector 206 is represented as a 1×6 vector with different shades to indicate different weights for each dimension in embedding vector 206.

[0057] The entire corpus can be processed similarly to generate multiple embedding vectors 206, each with the same number of dimensions (also referred to as having the same number of features or the same number of entries). Embedding vectors 206 can each form rows of an embedding matrix 208. Thus, embedding matrix 208 represents a vector space (i.e., the set of all embedding vectors) or latent code space that represents the vocabulary contained in the corpus and the semantic relations within those words. While in this example, embedding vectors 206 are shown as having a length of 6, and the corresponding embedding matrix 208 has 6 columns, embedding vectors 206 can have any length and are typically longer than 6. Therefore, embedding matrix 208 can similarly have any number of columns and is typically more than 6 columns. The number of rows in embedding matrix 208 corresponds to the number of embedding vectors 206 generated from the subwords 202 contained in the corpus. As can be understood, for large corpora with large vocabularys, embedding matrix 208 can be very large (e.g., having more than 50,000 rows and more than 200 columns).

[0058] While containing useful information about the corpus, the large embedding matrix 208 may be impractical for devices with limited memory and / or processing resources (e.g., handheld or mobile devices). Product quantization (PQ) is an existing technique used for the compression of embedding matrices. Some discussion of existing PQ techniques is provided here.

[0059] Consider the embedding matrix E∈R |V|×nHere, V is the vocabulary set contained in the corpus, |V| is the vocabulary size (i.e., the number of distinct subwords represented in the embedding matrix), and n is the embedding vector size (i.e., the length of each embedding vector). The first step in PQ is to partition the embedding matrix E into a known number of column groups g. Each group can be called a submatrix of the embedding matrix. The g submatrices of the embedding matrix E can be represented as G. i Where i is an integer, 1 ≤ i ≤ g and Therefore, the embedding matrix can be represented as a concatenation of submatrices as follows: E = [G1, G2, ... G g ]. Each partition G i It consists of sub-vectors from each row of the embedding matrix. Each sub-vector has a length. Use K-means clustering to divide each partition G i Cluster into c clusters to form c additional partitions within that partition.

[0060] Figure 3 It shows that the size is E∈R 4×6 An illustration of the PQ technique on embedding matrix 302. For simplicity, only one row of embedding matrix 302 (using different shading) indicates the dimensional weights; however, it should be understood that each row of embedding matrix 302 contains a set representing the dimensional weights of the corresponding sub-words. In this example, embedding matrix 302 is first partitioned (indicated by dashed lines) into three sub-matrices 304a, 304b, and 304c (often referred to as sub-matrices 304). Each sub-matrix consists of sub-vectors from each embedding vector (i.e., each embedding vector corresponding to each row of embedding matrix 302). Each sub-vector has a length of 2. Clustering is then performed (e.g., using K-means clustering) to cluster the rows of each sub-matrix 304 into two clusters 306 (different clusters 306 are represented by boxes of different line quality, where boxes of the same line quality indicate elements belonging to the same cluster 306). Thus, six clusters 306 are obtained (i.e., two clusters 306 are obtained for each of the three sub-matrices 304). It should be noted that a given cluster 306 may include non-adjacent rows of submatrix 304 (e.g., as indicated in submatrix 304b and submatrix 304c), different clusters 306 may have different sizes (i.e., different numbers of elements), and submatrix 304 may have different numbers of clusters.

[0061] After performing clustering, the centroid of each cluster 306 in each submatrix 304 is calculated and stored. The calculated centroid has the same dimension as the subvectors belonging to cluster 306. That is, for vectors with dimension... The cluster 306 of subvectors, whose centroids are also calculated, also have dimensions. exist Figure 3In the example, the calculated centroid of each cluster 306 has 2 dimensions. For a given submatrix 304, the centroids are calculated for all c clusters 306, and the centroids are stored in the centroid vector. In the matrix, j is the index of the cluster in the i-th submatrix 304. Therefore, there exist g centroid submatrices that can be stored together in the centroid matrix. In, among them, To reconstruct the embedding matrix E, the quantization matrix Q is also stored, where Q∈N. |V|×g The quantization matrix Q maps each embedding vector to each submatrix G. i A mapping of the number of specific clusters in the dataset. For example, in... Figure 3 In the matrix, the embedding vector 310 can be mapped to cluster c1 in submatrix 304a, cluster c2 in submatrix 304b, and cluster c3 in submatrix 304c. The embedding vector 310 can be reconstructed by using the centroid of c1 for the first two dimensions, the centroid of c2 for the middle two dimensions, and the centroid of c3 for the last two dimensions.

[0062] Therefore, matrix E can be reconstructed (as an approximation) using only the quantization matrix Q and the centroid matrix U. The compression ratio of this compression (assuming k bits are used to represent the floating-point value of each weight in the embedding vector) is given by the following equation:

[0063]

[0064] As understood from the above discussion, some information is lost during compression. While some information loss may be acceptable and expected when compressing embedding matrices, using conventional PQ techniques can lead to information loss that negatively impacts the performance of trained NLP models that learn compressed embedding matrices to perform NLP tasks. For example, conventional PQ techniques do not account for the uneven distribution of subwords in a corpus vocabulary, where some subwords may appear more frequently or have higher importance compared to others. Another drawback of conventional PQ techniques is that the centroid of each cluster is computed as a fixed representation of all elements in the cluster. However, there may be differences between element values ​​within a cluster, and this information is lost in conventional PQ compression. The result of these drawbacks is that the compression ratio may be lower than expected.

[0065] Another existing embedding matrix compression technique is called GroupReduce, which is based on singular value decomposition (SVD). GroupReduce first groups the words in the embedding matrix E based on the square root of their frequencies in the corpus, and then applies a weighted SVD to each group. This enables embedding matrix compression. Finally, the last step of GroupReduce refines the word group assignments by reducing reconstruction errors. However, as an SVD-based technique, GroupReduce performs worse in terms of compression quality ratio compared to conventional PQ techniques. GroupReduce relies on corpus-level word frequencies to define the relative importance of embedding vectors, which limits its applicability to compressing only the embedding matrix in NLP neural networks. For example, GroupReduce cannot be applied to compress other inner or hidden layers of the neural network.

[0066] This disclosure describes a PQ-based compression technique that has been found to have better performance than SVD and helps overcome some of the shortcomings of conventional PQ techniques.

[0067] The exemplary methods and systems described in this paper provide a means of capturing and preserving information about variance that is lost in conventional PQ compression. In neural networks, the computation performed by each layer can be represented by y = f(Wx), where x is the input vector, y is the output vector, f is the activation function, and W is a matrix containing elements of parameters learned through backpropagation. Similar to other types of matrices, matrices in neural networks contain variance information. The examples disclosed herein are able to incorporate such information after compression.

[0068] The exemplary methods and systems described herein also provide a means of capturing and preserving information about the relative importance of embedding vectors. In NLP applications, such as recommender systems, machine translation, automatic speech recognition, and many others, the subwords of a vocabulary are not uniformly distributed. This non-uniform distribution should be taken into account when compressing the embedding matrix of an NLP application to achieve (e.g., performance compared to the same NLP task using uncompressed data as input) more accurate performance of the NLP task. The non-uniform distribution of subwords means that not all embedding vectors are equally important. Therefore, it may be beneficial to compress the embedding vectors of those subwords that are more important to the task (i.e., using more data bits) with less compression, resulting in less information loss. On the other hand, the embedding vectors of less important subwords can be compressed more (i.e., using fewer data bits). Thus, a balance between compression and performance can be achieved. This disclosure also describes a method for determining the relative importance of vectors, which can be general enough to be applied to any matrix in a neural network, not limited to embedding vectors and embedding matrices.

[0069] Figure 4 This is a flowchart illustrating an exemplary method 400 for performing matrix compression. Method 400 can be... Figure 1 The computational system 100 executes the compression, for example, using instructions stored in the PQ-based compression module 150. Although referred to as PQ-based compression, it should be understood that the disclosed method 400 is not equivalent to conventional PQ compression techniques. As will be understood after the following discussion, the disclosed method 400 captures a different set of information and decompresses compressed data (i.e., reconstructs the compressed matrix) in a manner different from conventional PQ compression techniques. Specifically, method 400 stores random information about each cluster (e.g., mean and variance information) and uses this random information to reconstruct cluster elements by sampling from a Gaussian distribution. The following discussion is provided in the context of embedded matrix compression. However, it should be understood that this is provided only as illustration, and the compression techniques described below can be used for the compression of any matrix.

[0070] Method 400 begins at 402 with obtaining the embedding matrix E. Obtaining the embedding matrix E may include receiving it from another computing system (e.g., from an external database storing the embedding matrix), or may include retrieving it from the memory of computing system 100 (e.g., where the embedding matrix E was previously generated by computing system 100 itself from the vocabulary of a corpus). The embedding matrix E is defined as a matrix. Where V is the vocabulary set, and n is the dimension of each embedding vector.

[0071] At position 404, the embedding matrix E is decomposed or partitioned into g submatrices, each submatrix being one or more columns of the embedding matrix E. The partitioning can be uniform, such that all submatrices are of equal size (i.e., the number of columns and rows are equal), and each submatrix is ​​formed by one or more consecutive columns of the corresponding embedding matrix E. Each submatrix can be represented as G. i Where i is an integer, 1 ≤ i ≤ g. A subset of matrices can be represented as [G1, ..., G...]. g ],in,

[0072] At position 406, each submatrix G i The subvectors in 1≤i≤g are clustered into c clusters. In some examples, in each submatrix G i The number of clusters formed in the matrix may not be equal across all submatrices. Any suitable clustering algorithm can be used to perform this clustering. For example, the K-means clustering algorithm can be used. In some examples, the weighted K-means clustering algorithm can be used, as discussed further below.

[0073] At position 408, compute the centroid of each given cluster and the variance of all subvectors belonging to that cluster. Store the centroids and variances. It should be noted that the centroids and variances computed for a given cluster have the same dimension as the subvectors belonging to that cluster. For example, when the subvectors have a dimension... At the same time, the centroid and variance also have dimensions. The centroid calculated for all clusters over all submatrices can be represented as Where j is the index of the cluster in the i-th submatrix (1≤j≤c and 1≤i≤g). Furthermore, the variance calculated for all clusters across all submatrices can be expressed as: Here, j is the index of the cluster in the i-th submatrix (1≤j≤c and 1≤i≤g). That is, the centroid and variance calculated for a given cluster in a given submatrix can be identified by the index pair (j, i). The centroid can be stored together with the variance or stored separately.

[0074] At position 410, a mapping from each word in the vocabulary set V to the corresponding cluster in each submatrix is ​​also generated and stored. For a given word, the stored mapping indicates the cluster indices of all subvectors (indicating which cluster the centroid and variance should be selected from) to reconstruct each subvector, and they are concatenated to reconstruct the embedding vector of the given word. This mapping can be stored in a quantization matrix Q, where Q∈N. |V|×g .

[0075] It should be understood that the steps of method 400 can be performed in a different order than shown. For example, in step 406, the mapping can be generated and stored at any point during or after clustering.

[0076] The result of method 400 stores compressed data, including a centroid set indexed by the number of clusters (j) and the number of submatrices (i); a variance set indexed by the number of clusters (j) and the number of submatrices (i); and a mapping from each word to an index pair set, indicating the relevant cluster in each submatrix. As will be discussed further below, the embedding matrix can be reconstructed as a concatenation of subvectors reconstructed using this compressed data.

[0077] The compressed data generated by method 400 can be stored in computing system 100 for later processing (e.g., further used by a trained NLP model to perform a learned NLP task). Alternatively or additionally, the compressed data can be transferred from computing system 100 to another device (e.g., a handheld or mobile device, or other end-user device) for performing the NLP task. It should be understood that the compressed embedding matrix generated by method 400 is compressed such that the memory requirements for storing the compressed data and / or the transmission resources required for transmitting the compressed data are lower than the memory requirements and / or transmission resources required for storing and / or transmitting the original embedding matrix.

[0078] Figure 5This is a flowchart illustrating an exemplary method 500 for decompressing compressed data generated by method 400. Decompressing compressed data can also be referred to as reconstructing the compression matrix. Method 500 can be... Figure 1 The computing system 100 executes, for example, using instructions stored in the module 160 for PQ-based decompression. It should be noted that the same device (e.g., computing system 100) can execute both method 400 and method 500, in which case the device can store instructions for both the PQ-based compression module 150 and the PQ-based decompression module 160. Alternatively, method 400 and method 500 can be executed by different devices (each of which can be computing system 100), in which case the compression device can store only the instructions for the PQ-based compression module 150, and the decompression device can store only the instructions for the PQ-based decompression module 160.

[0079] The following discussion is provided in the context of embedded matrix reconstruction. However, it should be understood that this is provided only as an example, and the reconstruction techniques described below can be used to reconstruct any matrix using method 400 compression.

[0080] At 502, compressed data is obtained, including a centroid set indexed by the number of clusters (j) and the number of submatrices (i); a variance set indexed by the number of clusters (j) and the number of submatrices (i); and a mapping from each word to an index pair set, indicating the relevant cluster in each submatrix. Obtaining compressed data may include receiving compressed data from another computing system (e.g., from another computing system performing method 400), or may include retrieving compressed data from the memory of the same computing system 100 (e.g., in the case where both method 400 and method 500 are performed on the same computing system 100).

[0081] At position 504, to reconstruct each cluster, a continuous distribution (e.g., a multivariate Gaussian distribution) is generated using the correlation centroid and correlation variance. For example, each cluster can be indexed by an index pair (j, i), thus indicating a given cluster (j) in a given submatrix (i). The correlation centroid is then the centroid in the set of centroids indexed to this index pair, i.e., u ji The relevant variance is the variance in the covariance set from the index to this index pair, i.e., v ji Then, the multivariate Gaussian distribution of a given cluster at index pair (j, i) can be generated as follows:

[0082]

[0083] Then, it can be done from GD ji A sampled vector is used to reconstruct a given cluster. The sampled vector is used as the reconstructed centroid of the cluster. Although the multivariate Gaussian distribution is used as an example here, other continuous distributions can be used instead.

[0084] At position 506, for each word in the vocabulary set V, a mapping is used to identify the relevant clusters for reconstructing the embedding vectors representing the word. Each relevant cluster corresponds to a corresponding sub-vector of the embedding vector. For example, the mapping could be a quantization matrix Q that maps each word in the vocabulary to a cluster index, where each cluster index indicates the index pair (j, i) of the relevant cluster of the sub-vector of the embedding vector. Then, the centroid of the reconstructed relevant clusters is... It can be used to reconstruct each subvector.

[0085] At position 508, the embedding vector can be reconstructed by concatenating all the reconstructed subvectors in the appropriate order. Similarly, the entire embedding matrix can be reconstructed by concatenating the centroids corresponding to the reconstructions of each subvector in the appropriate order.

[0086] It should be understood that the steps of method 500 can be performed in a different order than those shown, and the exact concatenation order can be changed. For example, the embedding matrix can be reconstructed by first reconstructing each embedding vector (i.e., each row of the embedding matrix) by concatenating the subvectors of each embedding vector, and then concatenating the embedding vectors together to form the embedding matrix. Alternatively, the embedding matrix can be reconstructed by first reconstructing each submatrix by concatenating the subvectors belonging to the same submatrix, and then concatenating the submatrixes to form the embedding matrix.

[0087] The reconstructed embedding matrix can be used to replace the weights of the embedding matrix learned by a trained NLP model, for example, to perform NLP tasks (e.g., translation tasks). Furthermore, the reconstruction method 500 can also be incorporated into machine learning algorithms, for example, by modifying the architecture of a neural network so that a trained neural network performs reconstruction as a learning task.

[0088] Figure 6 This is a simplified graphical illustration of matrix compression performed using method 400 and decompression (or matrix reconstruction) performed using method 500. It should be understood that... Figure 6 The compression and decompression shown can be performed by separate devices, as discussed earlier.

[0089] Figure 6 The compression process following clustering is shown (e.g., after step 406 of method 400), and the decompression process is shown up to the cascaded reconstruction of the subvectors of the clusters (e.g., up to step 508 of method 500). In this simplified example, compression and reconstruction of a single cluster are shown. However, it should be understood that the compression and reconstruction of a single cluster is a demonstration of the compression and reconstruction of the larger matrix to which the cluster belongs.

[0090] exist Figure 6In the diagram, the j-th cluster of the i-th submatrix is ​​shown, denoted as c. ji In this example, cluster c ji It is formed by two sub-vectors 312a and 312b, each with two dimensions. Different shading indicates the weights of different dimensions in each sub-vector 312a and 312b. Cluster c ji The center of mass (denoted as u) ji ) and variance (denoted as v) ji The values ​​are calculated and stored in the centroid set 322 and the variance set 324, respectively, and are assigned to the cluster c. ji The index pairs (j, i) are indexed (e.g., corresponding to step 408 of method 400). Specifically, since they belong to cluster c ji Subvectors 312a and 312b have two dimensions, therefore the centroid u ji and variance v ji It also has two dimensions. For example, the centroid u ji It can be represented as:

[0091] u ji =(μ1,μ2)

[0092] And the variance v ji It can be represented as:

[0093] v ji =(σ1,σ2)

[0094] Each subvector 312a, 312b and its corresponding cluster c ji The relationships are also stored in mapping 326 (e.g., corresponding to step 410 of method 400). After processing all clusters across the entire matrix, the centroid set 322, the variance set 324, and the mapping 326 together form a set of compressed data 320 of the matrix. This compressed data 320 can be stored for future use by the same computing system and / or can be transferred to another computing system, as discussed above.

[0095] Figure 6 This demonstrates the reconstruction of a single cluster c using compressed data 320. ji Using index pairs (j, i), cluster c is identified from the centroid set 322 and the variance set 324, respectively. ji center of mass u ji and variance v ji Center of mass u ji and variance v ji This is used to generate a multivariate Gaussian distribution (in this example, a Gaussian distribution in two dimensions). The multivariate Gaussian distribution is sampled, and the sampled vector is used as the reconstructed centroid, denoted as... To reconstruct cluster v ji (For example, corresponding to step 504 of method 500). Different colors are used to indicate the centroid of the reconstruction. The weights of different sampling dimensions. It should be noted that the centroid is reconstructed. It is not necessarily equal to the centroid u calculated in the original calculation. ji Using mapping 326, identify the cluster corresponding to c. ji The subvectors, and can be used to reconstruct the centroid. The subvectors are reconstructed (e.g., corresponding to step 506 of method 500). The reconstructed subvectors can then be concatenated to reconstruct the cluster (and similarly reconstruct the entire matrix) (e.g., corresponding to step 508 of method 500).

[0096] As previously described, the clustering technique used to cluster the subvectors in each submatrix (e.g., at step 406 of method 400) can be K-means clustering. In some examples, weighted K-means clustering can be used. In the examples described below, the element weights are defined in a way that captures the relative importance of elements within a cluster. For example, each subvector belonging to a given cluster in a submatrix is ​​considered an element of the given cluster. The weighting of the elements then refers to the weight assigned to each subvector of the given cluster to control the relative influence of each subvector on the centroid of the computation of the given cluster. It should be understood that although the weighted K-means clustering disclosed herein is described in the context of the disclosed compression technique, the disclosed weighted K-means clustering technique can be used in other contexts; for example, the weighted K-means clustering disclosed herein can be used for clustering in any PQ-based technique (including conventional PQ compression). For example, the definition of the element weights disclosed below can be applied to any algorithm that incorporates frequency information in a PQ-based method by applying weights to perform static or dynamic clustering.

[0097] In the following discussion, weighted K-means clustering can be formulated as an expectation-maximization (EM) procedure. An example procedure for clustering a vector set x into c clusters is described below, where x... i Let μ represent the i-th vector in set x, and μ k Denotes the centroid of the k-th cluster:

[0098]

[0099] In the expected step (E-step) of the EM procedure, each element is assigned to the nearest mean (or centroid) of the cluster.

[0100] The expected update steps can be expressed as:

[0101]

[0102] In the maximization step (M-step) of the EM procedure, the mean (or centroid) of each cluster is recalculated based on the elements assigned to each cluster. The maximization step update can be expressed as:

[0103]

[0104] In the equation above, μ k Let z represent the mean of the k-th cluster. i Store the cluster mapping from the i-th element to the k-th cluster, N k This represents the number of elements in the k-th cluster. It should be noted that the cluster mean is weighted by the element weight α(i) of the i-th element. In this disclosure, element weights are defined to capture the relative importance of the i-th element within the cluster.

[0105] For example, element weights can be defined as the square root of the frequency of a given word in a corpus. This can be expressed as: Among them, f i It represents the frequency of the i-th word in the corpus. For example, this definition of element weights might be useful in the context of embedding matrices. In the context of other matrices, for example, the element weights of a given vector of a matrix could be based on the frequencies of corresponding words in the source dataset.

[0106] In another example that might help compress the matrix rather than embed it, the element weights can be defined as the Euclidean norm of the matrix rows. For example, for an embedding matrix E, the element weights can be defined as α(i) = ||E|. i ||2, which represents the Euclidean norm of the i-th row of matrix E. As mentioned above, this calculation can be performed similarly for any other matrix, and is not limited to the embedded matrix E.

[0107] Generally speaking, the Euclidean norm may be a good proxy for weighing the relative importance of row vectors in a matrix in neural network applications, because important trainable parameters in neural networks tend to change more frequently and are more likely to have higher norm values.

[0108] Clusters can be formed in such a way that elements with higher relative importance have a greater influence on the calculated cluster centroids by applying element weights (e.g., based on frequency or Euclidean norm). Using element weights defined by frequency or Euclidean norm, weighted K-means clustering techniques can be incorporated into the compression method disclosed herein. For example, weighted K-means clustering can be used at step 406 of method 400. Therefore, the disclosed weighted K-means clustering technique achieves non-uniform quantization, which allows for greater focus on different vectors based on their relative importance (as represented by their relative element weights). Specifically, using Euclidean norm to define element weights allows the disclosed weighted K-means clustering technique to be used independently of word frequency in the corpus. This makes the weighted K-means technique more widely applicable to different matrices, including hidden layers in neural networks.

[0109] In various example simulations, it has been found that the compression and reconstruction techniques described in this paper, whether using conventional K-means clustering or the disclosed weighted K-means clustering, achieve better compression quality ratios than existing techniques for compressing embedding matrices. For example, based on the size of the compressed data and the bilingual evaluation understudy (BLEU) score, neural networks trained to perform machine translation (e.g., English-to-French machine translation, English-to-German machine translation) achieve better compression quality ratios when using data compressed using the disclosed techniques compared to conventional PQ compression and GroupReduce techniques.

[0110] In various examples, this disclosure describes methods and systems capable of stochastic modeling of clusters (e.g., by including variance information in compressed data), which can provide more useful information for the input of machine learning algorithms.

[0111] This disclosure also describes different functions for defining element weights, which can be used in weighted K-means clustering or other weighted clustering techniques.

[0112] Although examples have already been given in the context of embedding matrices for NLP tasks, the examples in this disclosure can be applied to compressing any matrix, including those used in deep neural networks.

[0113] Although this disclosure describes methods and processes by way of steps performed in a certain order, one or more steps in the methods and processes may be omitted or modified as appropriate. Where appropriate, one or more steps may be performed in an order other than that described.

[0114] Although this disclosure describes at least part of the methodological aspects, those skilled in the art will understand that this disclosure also relates to various components for performing at least some aspects and features of the method, whether by hardware components, software, or any combination thereof. Accordingly, the technical solutions of this disclosure can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon that cause a processing device (e.g., a personal computer, server, or network device) to perform exemplary embodiments of the methods disclosed herein. Machine-executable instructions can be in the form of code sequences, configuration information, or other data that, when executed, cause a machine (e.g., a processor or other processing device) to perform the steps in the methods according to exemplary embodiments of this disclosure.

[0115] This disclosure may be implemented in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Features selected from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations will be understood within the scope of this disclosure.

[0116] All values ​​and sub-ranges within the scope of the disclosure are also disclosed. Furthermore, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices, and components may be modified to include more or fewer of such elements / components. For example, although any element / component disclosed may be referenced as a single quantity, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

Claims

1. A computing system, characterized in that, The system includes: Memory; A processing device, communicating with the memory, is configured to execute instructions to enable the computing system to: A matrix having multiple rows formed by corresponding multiple vectors is divided into multiple submatrices, each submatrix containing subvectors from one or more consecutive columns of the matrix; For each given submatrix, the subvectors of the given submatrix are clustered into multiple clusters; For each given cluster, the centroid and variance are calculated and stored based on the subvectors belonging to the given cluster; Store the mapping from each vector to the corresponding cluster in each submatrix; The stored centroid, the stored variance, and the stored mapping form a compressed dataset for reconstructing the matrix; Wherein, the matrix is ​​an embedding matrix, and the plurality of vectors are plurality of embedding vectors, each embedding vector being a vector representation of a corresponding subword in the vocabulary of the corpus.

2. The computing system according to claim 1, characterized in that, The processing device is used to execute the instructions to cause the computing system to cluster the subvectors using K-means clustering.

3. The computing system according to claim 1, characterized in that, The processing device is configured to execute the instructions to cause the computing system to cluster the subvectors using weighted K-means clustering, wherein the centroid of each given cluster is calculated using element weights to weight each subvector of the given cluster.

4. The computing system according to claim 3, characterized in that, Each vector in the matrix represents a corresponding sub-word in the source dataset, and the element weights of each corresponding sub-vector of the given cluster are defined based on the frequency of the corresponding sub-word in the source dataset.

5. The computing system according to claim 3, characterized in that, The element weights of each corresponding subvector of the given cluster are defined based on the Euclidean norm of the corresponding vector in the matrix.

6. The computing system according to any one of claims 1 to 5, characterized in that, The processing device is used to execute the instructions to enable the computing system: The compressed dataset is transmitted to another device, where it will be used as input for performing machine learning tasks.

7. A method for compressing matrices, characterized in that, The method includes: A matrix having multiple rows formed by corresponding multiple vectors is divided into multiple submatrices, each submatrix containing subvectors from one or more consecutive columns of the matrix; For each given submatrix, the subvectors of the given submatrix are clustered into multiple clusters; For each given cluster, the centroid and variance are calculated and stored based on the subvectors belonging to the given cluster; Store the mapping from each vector to the corresponding cluster in each submatrix; The stored centroid, the stored variance, and the stored mapping form a compressed dataset for reconstructing the matrix; Wherein, the matrix is ​​an embedding matrix, and the plurality of vectors are plurality of embedding vectors, each embedding vector being a vector representation of a corresponding subword in the vocabulary of the corpus.

8. The method according to claim 7, characterized in that, The subvectors are clustered using K-means clustering.

9. The method according to claim 7, characterized in that, The subvectors are clustered using weighted K-means clustering, and the centroid of each given cluster is calculated using element weights to weight each subvector of the given cluster.

10. The method according to claim 9, characterized in that, Each vector in the matrix represents a corresponding sub-word in the source dataset, and the element weights of each corresponding sub-vector of the given cluster are defined based on the frequency of the corresponding sub-word in the source dataset.

11. The method according to claim 9, characterized in that, The element weights of each corresponding subvector of the given cluster are defined based on the Euclidean norm of the corresponding vector in the matrix.

12. The method according to any one of claims 7 to 11, characterized in that, The method further includes: The compressed dataset is transmitted to another device, where it will be used as input for performing machine learning tasks.

13. A computer-readable medium, characterized in that, The medium includes instructions that, when executed by a processing device of a computing device, cause the computing device to perform the method according to any one of claims 7 to 12.

14. A computing system, characterized in that, The system includes: Memory; A processing device, communicating with the memory, is configured to execute instructions to enable the computing system to: Obtain a compressed dataset for reconstructing the matrix, the compressed dataset comprising a centroid set and a variance set, each centroid being associated with a corresponding cluster and each variance being associated with a corresponding cluster, the compressed dataset also comprising a mapping of each subvector of the matrix to the corresponding cluster; For each given cluster, reconstructed centroids are generated by sampling from a distribution that is generated using the centroids and variance associated with the given cluster; Each given subvector is reconstructed using the reconstructed centroids generated for the relevant clusters, the relevant clusters of which are identified using the mapping; The matrix is ​​reconstructed by concatenating the reconstructed subvectors; The matrix is ​​an embedding matrix, which is formed by a plurality of vectors, and the plurality of vectors are multiple embedding vectors, each of which is a vector representation of a corresponding subword in the vocabulary of the corpus.

15. The computing system according to claim 14, characterized in that, The distribution is a multivariate Gaussian distribution.

16. The computing system according to claim 14, characterized in that, The compressed dataset was obtained in communication from another device.

17. A method for reconstructing a compressed matrix, characterized in that, The method includes: Obtain a compressed dataset for reconstructing the matrix, the compressed dataset comprising a centroid set and a variance set, each centroid being associated with a corresponding cluster and each variance being associated with a corresponding cluster, the compressed dataset also comprising a mapping of each subvector of the matrix to the corresponding cluster; For each given cluster, reconstructed centroids are generated by sampling from a distribution that is generated using the centroids and variance associated with the given cluster; Each given subvector is reconstructed using the reconstructed centroids generated for the relevant clusters, the relevant clusters of which are identified using the mapping; The matrix is ​​reconstructed by concatenating the reconstructed subvectors; The matrix is ​​an embedding matrix, which is formed by a plurality of vectors, and the plurality of vectors are multiple embedding vectors, each of which is a vector representation of a corresponding subword in the vocabulary of the corpus.

18. The method according to claim 17, characterized in that, The distribution is a multivariate Gaussian distribution.

19. The method according to claim 17, characterized in that, The compressed dataset was obtained in communication from another device.

20. A computer-readable medium, characterized in that, The medium includes instructions that, when executed by a processing device of a computing device, cause the computing device to perform the method according to any one of claims 17 to 19.

Images