Data Compression with Local Entropy Encoding

Abstract

Methods, systems, and apparatuses for compressing and decompressing data, including computer programs encoded on computer storage media. In one aspect, an encoder neural network processes data to generate an output that includes a representation of the data as an ordered set of code symbols. The ordered set of code symbols is entropy encoded using one or more code symbol probability distributions. A compressed representation of the data is determined based on the entropy encoded representation of the set of code symbols and data indicating the code symbol probability distribution used to entropy encode the set of code symbols. In another aspect, the compressed representation of the data is decoded to determine the set of code symbols representing the data. The set of code symbols is processed by a decoder neural network to determine a reconstruction of the data.

Description

Technical Field

[0001] This specification relates to data compression. Background Art

[0002] Compressed data refers to determining a representation of data that occupies less space in a memory. The compressed data can be stored (e.g., in a logical data storage area or a physical data storage device), transmitted to a destination via a communication network (e.g., the Internet), or used in any other way. Generally, the data can be (approximately or precisely) reconstructed from the compressed representation of the data. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which performs data compression and data decompression.

[0004] According to a first aspect, a system for compressing data is provided. The system includes an encoder neural network configured to receive data to be compressed and process the data according to current values of a set of encoder neural network parameters. The encoder neural network generates an output including a representation of the data as an ordered set of code symbols.

[0005] The system further includes a compression subsystem configured to receive the output of the encoder neural network, the output of the encoder neural network including a representation of the data as an ordered set of code symbols. Identify a partition of the set of code symbols into one or more code symbol subsets. For each code symbol subset, identify a code symbol probability distribution of the code symbol subset based on the code symbol subset and from a dictionary of multiple code symbol probability distributions. Generate an entropy-coded representation of each code symbol subset by entropy-coding the code symbol subset using the identified code symbol probability distribution for the code symbol subset. Determine a compressed representation of each code symbol subset, the compressed representation including: (i) the entropy-coded representation of the code symbol subset, and (ii) data indicating the code symbol probability distribution used for entropy-coding the code symbol subset. Determine a compressed representation of the data based on the compressed representations of the code symbol subsets.

[0006] In some embodiments, the data to be compressed includes an image.

[0007] In some embodiments, the encoder neural network is a convolutional neural network.

[0008] In some embodiments, the system includes a learning subsystem configured to process data in a set of training data through an encoder neural network and according to current values of a set of encoder neural network parameters. A representation of the data in the set of training data is generated as an ordered set of code symbols. For each ordered set of code symbols, a partition of the ordered set of code symbols into one or more subsets of code symbols is identified. Based on the distribution of code symbols in the subsets of code symbols of the ordered set of code symbols, a dictionary of a plurality of code symbol probability distributions is determined.

[0009] In some embodiments, determining the dictionary of code symbol probability distributions includes: for each subset of code symbols of the ordered set of code symbols, determining a corresponding code symbol probability distribution. For each subset of code symbols, the code symbol probability distribution reflects the occurrence frequency of the code symbols in the subset of code symbols. Clustering the determined code symbol probability distributions. The dictionary of code symbol probability distributions is determined based on the cluster centers of the clustered code symbol probability distributions.

[0010] In some embodiments, clustering the code symbol probability distributions includes determining a distance between code symbol probability distributions based on a divergence metric.

[0011] In some embodiments, identifying the code symbol probability distribution of a subset of code symbols includes: for each code symbol probability distribution in the dictionary, determining a corresponding length of an entropy-coded representation of the subset of code symbols based on the code symbol probability distribution. The code symbol probability distribution of the subset of code symbols is identified as the code symbol probability distribution from the dictionary having the minimum corresponding length of the entropy-coded representation.

[0012] In some embodiments, determining the length of the entropy-coded representation of a subset of code symbols based on the code symbol probability distribution includes determining, according to the code symbol probability distribution, the sum of the logarithms of the probabilities of the code symbols on each code symbol in the subset of code symbols.

[0013] In some embodiments, one or more subsets of code symbols are determined as candidate subsets of code symbols. A custom code symbol probability distribution is determined based on the candidate subsets of code symbols. A savings value is determined based on a reduction in the length of the entropy-coded representation of the candidate subsets of code symbols, the reduction in the length of the entropy-coded representation of the candidate subsets of code symbols being achieved by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution instead of the code symbol probability distribution from the dictionary. A cost value is determined based on the data length required to represent the custom code symbol probability distribution. In response to determining that the savings value is greater than the cost value, an entropy-coded representation of the code symbols of the candidate subsets of code symbols is generated by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution.

[0014] In some embodiments, identifying a partition of the set of code symbols into one or more subsets of code symbols includes identifying a partition of the set of code symbols into a plurality of different proper subsets.

[0015] According to a second aspect, a system for decompressing data is provided. The system is configured to receive a compressed representation of the data, the compressed representation of the data comprising a compressed representation of each of one or more code symbol subsets representing an ordered set of code symbols for the data. The compressed representation of each code symbol subset comprises: (i) an entropy-encoded representation of the code symbol subset, and (ii) data indicating a code symbol probability distribution used to entropy encode the code symbol subset. Each code symbol subset is determined by decoding the entropy-encoded representation of the code symbol subset based on the code symbol probability distribution used to entropy encode the code symbol subset. Based on the code symbol subsets, a representation of the data is determined as an ordered set of code symbols.

[0016] The system also includes a decoder neural network. The decoder neural network is configured to receive a representation of the data as an ordered set of code symbols. The ordered set of code symbols is processed by the decoder neural network based on current values of a set of decoder neural network parameters to generate an output defining a reconstruction of the data.

[0017] In some embodiments, the data indicative of a code symbol probability distribution for entropy encoding the subset of code symbols comprises data indicative of a code symbol probability distribution from a dictionary of code symbol probability distributions.

[0018] In some embodiments, the data indicative of a code symbol probability distribution for entropy encoding a subset of code symbols includes data defining a respective numerical probability value for each code symbol in a set of possible code symbols.

[0019] According to a third aspect, there is provided a non-transitory computer-readable storage medium storing instructions executable by a data processing apparatus and, when so executed, causing the data processing apparatus to perform the operations of the aforementioned system.

[0020] According to a fourth aspect, there is provided a computer-implemented method of performing the operations of the aforementioned system.

[0021] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0022] The encoding systems described in this specification can compress data more efficiently than some conventional systems by generating compressed representations of data that occupy less memory than compressed representations of the data generated by conventional systems.

[0023] For example, the coding system described in this specification can partition an ordered set of code symbols representing data into subsets and perform entropy coding on each subset with a corresponding local code symbol probability distribution. In contrast, some conventional systems perform entropy coding on an ordered set of code symbols representing data with a global code symbol probability distribution, which may not represent the actual occurrence frequencies of different code symbols in the ordered set of code symbols. By performing entropy coding on different subsets of the ordered set of code symbols with different local code symbol probability distributions instead of a single global code symbol probability distribution, the coding system described in this specification can compress data more efficiently than some conventional systems.

[0024] As another example, different from some conventional systems, the coding system described in this specification can perform entropy coding using a custom code symbol probability distribution determined from the data being compressed (rather than learned from training data) when doing so will result in more efficient entropy coding. Thus, the coding system described in this specification can compress data more efficiently than some conventional systems that only use a general code symbol probability distribution for entropy coding, which is not determined from the data being compressed. This can be particularly beneficial in the case of using an encoder or decoder neural network, where the representation of the data can be obtained in a non-linear manner that is not derived from a strict analysis and can vary according to the training state of the neural network.

[0025] By compressing data more efficiently than conventional systems, the coding system described in this specification can achieve more efficient data transmission (e.g., by reducing the communication network bandwidth required to transmit data) and more efficient data storage (e.g., by reducing the amount of memory required to store data). Additionally, through the disclosed methods, efficiency can be improved without compromising data quality and / or authenticity. These are technological improvements in the field of data compression.

[0026] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram of an example coding system.

[0028] Figure 2 is a block diagram of an example decoding system.

[0029] Figure 3 is a flowchart of an example process for determining a compressed representation of data.

[0030] Figure 4 A flowchart of an example process for determining a code symbol probability distribution for a dictionary from a set of training data.

[0031] Figure 5 A flowchart of an example process for identifying a custom code symbol probability distribution for one or more subsets of code symbols.

[0032] Figure 6 A flowchart of an example process for determining a reconstruction of data from a compressed representation of the data.

[0033] Like reference numerals and names in different figures indicate like elements. DETAILED DESCRIPTION

[0034] The techniques described in this specification relate to an encoding system that performs data compression. The data to be compressed can be image data, video data, or any other type of data. The encoding system generates a representation of the data (e.g., via an encoder neural network) as an ordered set of code symbols (e.g., integer values). The encoding system identifies a partitioning of the ordered set of code symbols representing the data into one or more subsets of code symbols. For example, the ordered set of code symbols can be a matrix, and the partitioning can be “tiling” the matrix into regularly spaced rectangular subsets. The encoding system can perform entropy encoding (e.g., via Huffman encoding) on each subset of code symbols using a corresponding code symbol probability distribution from a dictionary of code symbol probability distributions learned from a set of training data. The encoding system can determine a compressed representation of the data based on: (i) the entropy-coded representation of the subset of code symbols, and (ii) data indicating the corresponding code symbol probability distribution used to perform entropy encoding on the subset of code symbols.

[0035] In some cases, the encoding system can determine that more efficient (i.e., smaller) compressed representations of the data can be achieved by performing entropy encoding on one or more subsets of code symbols using a custom code symbol probability distribution learned directly from the data to be compressed. In these cases, the encoding system can perform entropy encoding on one or more of the subsets of code symbols using the custom code symbol probability distribution.

[0036] This specification also describes a decoding system that is configured to process the compressed representation of the data generated by the encoding system to generate a (approximate or exact) reconstruction of the data. These features and other features will be described in more detail later.

[0037] Figure 1is a block diagram of an example encoding system 100. The encoding system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, where the systems, components, and techniques described below are implemented.

[0038] The encoding system 100 processes data 102 to generate a compressed representation 104 of the data 102 as output. The compressed representation 104 can occupy less memory than the data 102 (e.g., in a logical data storage area or a physical data storage device). The data 102 can be image data, audio data, video data, or any other form of data. Generally, the compressed representation 104 contains information characterizing the data 102 such that a decoding system 200 can process the compressed representation 104 to generate a (near or exact) reconstruction of the data 102 (as further described with reference to Figure 2 ).

[0039] The encoding system 100 determines a representation of the data 102 as an ordered set 106 of code symbols. The code symbols are members of a set of discrete possible code symbols. For example, the possible code symbols can be numerical symbols (e.g., integer values), alphabetical symbols (e.g., from the English alphabet), or symbols from any other discrete set. The ordered set 106 of code symbols can be represented as a vector of code symbols, a (multi-dimensional) matrix of code symbols (e.g., as shown at 108), or in any other suitable manner.

[0040] The encoding system 100 can determine the set 106 of code symbols in any suitable way. For example, the encoding system 100 can determine the set 106 of code symbols as the coefficients of a representation of the data 102 with respect to a set of basis functions (e.g., Fourier basis). As another example, the encoding system 100 can determine the set 106 of code symbols by providing a digital representation of the data 102 as input to an encoder neural network 110. The encoder neural network 110 is configured to process the input according to the current values of the encoder neural network parameters to generate the set 106 of code symbols as output (as will be further described with reference to Figure 3 ). The encoder neural network 110 can be a feedforward neural network, a recurrent neural network, or any other suitable neural network implementation. The encoder neural network 110 can include fully connected layers, convolutional layers, or any suitable neural network layers.

[0041] In a particular example, the data 102 can be an image, and the encoder neural network 110 can be a convolutional neural network. In this example, the set 106 of code symbols can be a three-dimensional (3D) matrix C of code symbols (e.g., as shown at 108). The 3D matrix C of code symbols can be indexed by x, y, and z coordinates. In this specification, the two-dimensional (2D) set C(·,·,z) of code symbols corresponding to any particular z coordinate is referred to as a channel of the set 106 of code symbols. That is, the channel of the 3D matrix C corresponding to the coordinate z is given by:

[0042] C(·,·,z) = {C(x,y,z): for each x coordinate and each y coordinate} (1)

[0043] The encoding system 100 provides the set 106 of code symbols to the compression system 112, which processes the set 106 of code symbols to generate a compressed representation 104 of the data 102.

[0044] The compression system of 112 provides the set 106 of code symbols as input to a partitioning engine 114. The partitioning engine 114 is configured to identify a partitioning of the set 106 of code symbols into one or more subsets 116 of code symbols. That is, the partitioning engine 114 partitions the set 106 of code symbols into one or more non-overlapping parts, each non-overlapping part being referred to as a subset 116 of code symbols.

[0045] In a particular example, when the set 106 of code symbols is a 3D matrix of code symbols (e.g., as shown at 108), the partitioning engine 114 can identify the subsets 116 of code symbols by partitioning each channel of the 3D matrix of code symbols into regularly spaced rectangular blocks. As shown at 118, each channel of the 3D matrix of code symbols can be partitioned into 2×2 subsets 116 of code symbols (or, more generally, subsets 116 of code symbols of any suitable dimension). Other n×n subsets 116 of code symbols, such as 4×4, can also be used.

[0046] The compression system 112 provides each subset 116 of code symbols to an entropy encoding engine 120. For each subset 116 of code symbols, the entropy encoding engine 120 identifies a corresponding code symbol probability distribution (e.g., one of the three code symbol probability distributions depicted in 122) from a dictionary 124 of code symbol probability distributions (i.e., a set of code symbol probability distributions). A code symbol probability distribution is data that indicates the respective numerical probability values for each code symbol in a set of discrete possible code symbols. As will be referenced Figure 3Further described, for a given subset 116 of code symbols, the entropy encoding engine 120 can identify a code symbol probability distribution from the dictionary 124 that enables the given subset 116 of code symbols to be most effectively entropy encoded.

[0047] For each subset 116 of code symbols, after identifying the code symbol probability distribution of the subset 116 of code symbols from the dictionary 124, the entropy encoding engine 120 generates an entropy encoded representation 126 of the subset 116 of code symbols using the identified code symbol probability distribution. The entropy encoding engine 120 can generate the corresponding entropy encoded representation 126 of each subset 116 of code symbols by any suitable entropy encoding algorithm (such as Huffman coding or arithmetic coding). An entropy encoding algorithm is a method for lossless compression of a set of code symbols conditioned on a probability distribution over the set of code symbols. The entropy encoded representation 126 can be represented in any suitable format. For example, the entropy encoded representation 126 can be represented numerically as binary data.

[0048] After generating the entropy encoded representation 126 of each subset 116 of code symbols, the compression system 112 determines a compressed representation of the data 102. The compressed representation 104 of the data 102 can include the compressed representation of each subset 116 of code symbols. The compressed representation of each subset 116 of code symbols can include: (i) the entropy encoded representation 126 of the subset 116 of code symbols, and (ii) data indicating the code symbol probability distribution used by the entropy encoding engine 120 to entropy encode the subset 116 of code symbols. For example, if the code symbol probability distributions in the dictionary 124 are indexed by natural numbers (e.g., 1, 2, 3, …), the data indicating which code symbol probability distribution was used to entropy encode the subset 116 of code symbols can be the index of the appropriate code symbol probability distribution. The compressed representation 104 can be represented in any suitable format. For example, the compressed representation 104 can be represented numerically as binary data.

[0049] The encoding system 100 includes a learning system 128 that determines the code symbol probability distributions in the dictionary 124 from a set of training data 130. Typically, the training data 130 includes data of the same type as the data 102 to be compressed. For example, if the data 102 is image data, then the training data 130 can include multiple images. As will be referenced Figure 4Further described, the learning system 128 can process the training data 130 through the encoder neural network 110 to generate a representation of the training data as an ordered set of code symbols. After determining the ordered set of code symbols representing the training data 130, the learning system 128 can partition the ordered set of code symbols into code symbol subsets and determine a set of code symbol probability distributions representing the occurrence frequencies of the code symbols in the code symbol subsets. Next, the learning system 128 can cluster the set of code symbol probability distributions to determine a fixed number of representative code symbol probability distributions to be included in the dictionary 124.

[0050] The code symbol probability distributions in the dictionary 124 are determined from the training data 130 and represent the occurrence frequencies of the code symbols in the exemplary code symbol subsets 116. In some cases, the occurrence frequencies of the code symbols in one or more of the code symbol subsets 116 of the set of code symbols 106 representing the data 102 are not well represented by the code symbol probability distributions in the dictionary 124. In these cases, as will be referenced Figure 5 Further described, the encoding system 100 can determine one or more custom code symbol probability distributions 117 based on the code symbol subsets 116 representing the data 102. The entropy encoding engine 120 can use the custom code symbol probability distributions 117 to perform entropy encoding on one or more of the code symbol subsets 116.

[0051] After the encoding system 100 determines the compressed representation 104 of the data 102, the compressed representation 104 can be stored (e.g., in a logical data storage area or a physical data storage device), transmitted over a communication network (e.g., the Internet) to a destination, or used in any other way. The data 102 can be (approximately or precisely) reconstructed by the decoding system 200 based on the compressed representation 104, as will be referenced Figure 2 Further described.

[0052] Figure 2 is a block diagram of an exemplary decoding system 200. The decoding system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations, where the systems, components, and techniques described below are implemented.

[0053] The decoding system 200 processes the compressed representation 104 of the data 102 (e.g., as referenced Figure 1described as being generated by the coding system 100) to generate reconstructed data 202 as output, which is an (approximate or exact) reconstruction of the data 102. As previously described, the compressed representation 104 can include a compressed representation of each code symbol subset 116 of a set 106 of code symbols representing the data 102. The compressed representation of the code symbol subset 116 can include: (i) an entropy-coded representation 126 of the code symbol subset 116, and (ii) data indicating the code symbol probability distribution used for entropy-coding the code symbol subset 116.

[0054] The coding system 100 and the decoding system 200 share a dictionary 124 of code symbol probability distributions. When the code symbol probability distribution used for entropy-coding the code symbol subset 116 is a custom code symbol probability distribution (as referenced Figure 1 \ as described), the compressed representation 104 includes data defining the custom code symbol probability distribution. For example, the compressed representation 104 can include a vector having components indicating the respective numerical probability values of each code symbol in a set of discrete possible code symbols according to the custom code symbol probability distribution.

[0055] The decoding system 200 provides the entropy-coded representation 126 of the code symbol subset 116 to the entropy decoding engine 204. The entropy decoding engine 204 is configured to determine the code symbol subset 116 by decoding each entropy-coded representation 126 of the code symbol subset 116 based on the code symbol probability distribution used for entropy-coding the code symbol subset 116. For example, conditional on the code symbol probability distribution used for entropy-coding the code symbol subset 116, the entropy decoding engine 204 can determine the mapping between portions of the entropy-coded representation 126 of the code symbol subset 116 and the corresponding code symbols. The entropy decoding engine 204 can use this mapping to determine the code symbol subset 116 from the entropy-coded representation 126 of the code symbol subset 116.

[0056] The decoding system 200 determines an ordered set 106 of code symbols representing the data 102 from the code symbol subset 116. For example, the decoding system 200 can concatenate the code symbol subsets 116 to determine the ordered set 106 of code symbols representing the data 102.

[0057] The decoding system 200 determines reconstructed data 202 based on an ordered set 106 of code symbols, where the reconstructed data 202 is an (approximate or exact) reconstruction of the data 102. For example, the encoding system 100 may have determined the ordered set 106 of code symbols as the coefficients of a representation of the data 102 with respect to a set of basis functions (e.g., Fourier basis). In this example, the decoding system 200 may determine the reconstructed data 202 as a linear combination of the basis functions and the coefficients given by the set 106 of code symbols. As another example, if the encoding system 100 determines the ordered set 106 of code symbols as the output of an encoder neural network 110, the decoding system 200 may determine the reconstructed data 202 by providing the set 106 of code symbols as an input to a decoder neural network 206. The decoder neural network 206 is configured to process the set 106 of code symbols according to the current values of the decoder neural network parameters to generate an output that defines the reconstructed data 202. In a particular example, if the encoder neural network 110 is a convolutional neural network, the decoder neural network 206 may be a corresponding deconvolutional neural network.

[0058] Figure 3 is a flowchart of an example process for determining a compressed representation of data. For convenience, the process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, an encoding system, such as Figure 1 the encoding system 100 of, programmed appropriately according to this specification, may perform the process 300.

[0059] The system receives data to be compressed (302). The data can be image data, audio data, video data, or any other form of data. For example, the data can be an I-frame, P-frame, or B-frame from video data. An I-frame refers to an image representing a frame in a video. A P-frame refers to data representing the residual (i.e., difference) between an image representing a frame in a video and a prediction of the image representing that frame based on a previous frame in the video. A B-frame refers to data representing the residual between an image representing a frame in a video and a prediction of the image representing that frame based on a previous frame and a subsequent frame. As another example, the data can be feature data derived from an image or video. As another example, the data can be 3D mesh data. As another example, the data can be light field data (i.e., photographic data from multiple cameras).

[0060] The system generates a representation of the data as an ordered set of code symbols (304). A code symbol is a member of a set of discrete possible code symbols. For example, the possible code symbols can be numerical symbols (e.g., integer values), alphabetic symbols (e.g., from the English alphabet), or symbols from any other discrete set. The ordered set of code symbols can be represented as a vector of code symbols, a (multi-dimensional) matrix of code symbols, or in any other suitable manner.

[0061] The system can generate a set of code symbols in any suitable manner. For example, the system can determine the coefficients of a set of basis functions (e.g., Fourier basis functions) representing data, and generate a set of code symbols by quantizing the coefficients to integer values within a bounded range.

[0062] As another example, the system can generate a set of code symbols by providing a digital representation of the data as an input to an encoder neural network. For example, if the data is image data, the system can provide the digital representation of the image data as a matrix of digital values representing pixel colors / intensities to the encoder neural network.

[0063] The encoder neural network processes the input according to the current values of the encoder neural network parameters to generate an output that defines a set of code symbols. The system can determine the set of code symbols by quantizing the output of the encoder neural network to integer values within a bounded range. The encoder neural network can be a feedforward neural network, a recurrent neural network, or any other suitable neural network implementation. The encoder neural network can include fully connected layers, convolutional layers, or any suitable neural network layers.

[0064] The system identifies a partitioning (306) of the set of code symbols into one or more subsets of code symbols. For example, when the set of code symbols is a 3D matrix of code symbols, the system can identify subsets of code symbols by partitioning each channel of the 3D matrix of code symbols into regularly spaced rectangular blocks.

[0065] For each subset of code symbols, the system identifies the corresponding code symbol probability distribution (308) from a dictionary of code symbol probability distributions. A code symbol probability distribution is data that indicates the corresponding numerical probability values for each code symbol in a set of discrete possible code symbols. Refer to Figure 4 for further description of the dictionary of code symbol probability distributions.

[0066] For a given subset of code symbols, the system can identify the corresponding code symbol probability distribution by determining a corresponding numerical score for each code symbol probability distribution in the dictionary. For a given code symbol probability distribution in the dictionary, the score can be the corresponding length (e.g., measured in bits) of the entropy-coded representation of the subset of code symbols determined using the code symbol probability distribution. For example, for each code symbol probability distribution p in the dictionary, the system can determine the score as the optimal length L of the entropy-coded representation of the subset of code symbols determined using the code symbol probability distribution p, given by:

[0067]

[0068] where n is the number of code symbols in the subset of code symbols, and p(cj ) is the probability of the j-th code symbol in the code symbol subset according to the code symbol probability distribution p.

[0069] The system can identify the code symbol probability distribution of a given code symbol subset based on the determined score of the code symbol probability distribution in the dictionary. For example, for a given code symbol subset, the system can identify the code symbol probability distribution in the dictionary with the lowest corresponding score. That is, in this example, the system can identify the code symbol probability distribution in the dictionary that enables the most efficient entropy coding of the code symbol subset.

[0070] Optionally, for one or more code symbol subsets, the system identifies a custom code symbol probability distribution (310) based on the occurrence frequency of the code symbols in the code symbol subset. Refer to Figure 5 describes an example process for identifying a custom code symbol probability distribution for one or more code symbol subsets.

[0071] The system generates an entropy-coded representation (312) of each code symbol subset. More specifically, for each code symbol subset, the system determines the entropy-coded representation of the code symbol subset by entropy-coding the code symbol subset using the code symbol probability distribution identified for the code symbol subset (e.g., as described in 308 and 310). The code symbol probability distribution used to entropy-code the code symbol subset can be the code symbol probability distribution from the dictionary, or in some cases, a custom code symbol probability distribution (e.g., as described in reference 310). The system can entropy-code the code symbol subset using any suitable entropy-coding algorithm (such as Huffman coding, arithmetic coding, or range coding). The entropy-coded representation of the code symbol subset can be represented in any suitable format. For example, the entropy-coded representation can be represented numerically as binary data.

[0072] The system determines a compressed representation (314) of each code symbol subset. The compressed representation of each code symbol subset can include: (i) the entropy-coded representation of the code symbol subset (e.g., as described in reference 312), and (ii) data indicating which code symbol probability distribution was used to entropy-code the code symbol subset. For example, if the code symbol probability distribution in the dictionary is indexed by natural numbers (e.g., 1, 2, 3,...), the data indicating which code symbol probability distribution was used to entropy-code the code symbol subset can be the index of the code symbol subset in the dictionary. As another example, if the code symbol probability distribution used to entropy-code the code symbol subset is a custom code symbol probability distribution, the data indicating the custom code symbol probability distribution can include the corresponding custom probability values for each code symbol in a set of discrete possible code symbols.

[0073] The system determines a compressed representation of data (316). The system can determine the compressed representation of data based on: (i) the compressed representation of each subset of code symbols (e.g., as described in reference 314), and (ii) the representation of any custom code symbol probability distribution for entropy encoding any subset of code symbols within a subset of code symbols. The compressed representation of data can be represented in any suitable format. For example, the compressed representation of data can be represented numerically as binary data.

[0074] Figure 4 is a flowchart of an example process for determining a code symbol probability distribution from a set of training data. For convenience, process 400 will be described as being performed by a system of one or more computers located at one or more locations. For example, an encoding system, such as Figure 1 encoding system 100 of, programmed appropriately according to this specification, can perform process 400.

[0075] The system processes data included in a set of training data to generate an ordered set of code symbols representing the training data (402). The training data can be image data (e.g., including multiple different images), audio data (e.g., including multiple different audio segments), or any other form of data. An example of generating an ordered set of code symbols to represent data is described in reference 304.

[0076] The system identifies a corresponding partition of the set of each code symbol representing the training data into one or more subsets of code symbols (404). An example of identifying a partition of a set of code symbols into one or more subsets of code symbols is described in reference 306.

[0077] For each subset of code symbols of the set of each code symbol representing the training data, the system determines a corresponding code symbol probability distribution representing the frequency of occurrence of the code symbols in the subset of code symbols (406). For example, if a set of possible code symbols is: {0, 1, 2, 3}, and the subset of code symbols is given by (0, 3, 1, 2, 2, 0, 1, 3), then the system can determine the code symbol probability distribution of the subset of code symbols: (0.25, 0.25, 0.25, 0.25).

[0078] The system clusters a set of code symbol probability distributions representing the occurrence frequencies of code symbols in a subset of code symbols determined from training data (408). Clustering the set of code symbol probability distributions means grouping the code symbol probability distributions such that the code symbol probability distributions in the same group are more similar than those in different groups. The system can use any suitable clustering algorithm (such as, the k-means clustering algorithm, the expectation maximization clustering algorithm, or the neural network clustering algorithm). In a particular example, the system can use the k-means clustering algorithm, where the distance between probability distributions is calculated by a divergence metric (e.g., the Kulback-Leibler divergence metric).

[0079] The system determines a dictionary of code symbol probability distributions (i.e., a set of code symbol probability distributions) based on the clustered code symbol probability distributions (e.g., as determined in 408) (410). For example, for each group of code symbol probability distributions determined by clustering, the system can determine the probability distribution defined by the cluster center as the code symbol probability distribution to be included in the dictionary. For a given group of code symbol probability distributions determined by clustering, the probability distribution defined by the cluster center can be the average of the code symbol probability distributions included in that group.

[0080] Figure 5 is a flowchart of an example process for identifying a custom code symbol probability distribution for one or more subsets of code symbols. For convenience, process 500 will be described as being performed by a system of one or more computers located at one or more locations. For example, an encoding system 100, such as Figure 1 the encoding system appropriately programmed according to this specification, can perform process 500. Generally, identifying a custom code symbol probability distribution can be performed without partitioning the set of code symbols into multiple subsets of code symbols.

[0081] The system determines one or more candidate subsets of code symbols from the subset of code symbols representing the data to be compressed (e.g., as described with reference to 306). Each candidate subset of code symbols is a candidate to be entropy encoded relative to the custom code symbol probability distribution. For example, the system can determine the candidate subsets of code symbols as those subsets of code symbols that are determined to not be well represented by the dictionary of code symbol probability distributions determined from training data (e.g., as described with reference to Figure 4 ). In this example, if the following relationship is satisfied, the system can determine that the subset of code symbols is not well represented by the dictionary of code symbol probability distributions:

[0082] |L dic -L true |>δ·L dic (3)

[0083] where δ is a positive value (e.g., 0.5%, or any other value between 0 and 1), L dic is given by:

[0084]

[0085] where n is the number of code symbols in the code symbol subset, and p dic (c j ) is the probability of the j-th code symbol in the code symbol subset according to the code symbol probability distribution recognized from the dictionary (i.e., as described in reference 308), and L true is given by:

[0086]

[0087] where p true (c j ) is the probability of the j-th code symbol in the code symbol subset according to the code symbol probability distribution reflecting the actual occurrence frequency of the code symbols in the code symbol subset.

[0088] The system determines a custom code symbol probability distribution (5**04**) based on a candidate code symbol subset (e.g., as determined in 5**02**). For example, the system can determine the custom code symbol probability distribution as an average of the code symbol probability distributions reflecting the actual occurrence frequencies of the code symbols in the candidate code symbol subset (5**04**).

[0089] The system determines a savings value achieved by entropy encoding the candidate code symbol subset using the custom code symbol probability distribution instead of the code symbol probability distribution from the dictionary (5**06**). For example, the system can determine the savings value based on a reduction in the length of the entropy-encoded representation of the candidate code symbol subset (e.g., measured in bits), the reduction in the length of the entropy-encoded representation of the candidate code symbol subset being achieved by entropy encoding the candidate code symbol subset using the custom code symbol probability distribution instead of the code symbol probability distribution from the dictionary.

[0090] The system determines a cost value representing the custom code symbol probability distribution (5**08**). For example, the system can determine the cost value based on the length of the data required to represent the custom code symbol probability distribution (e.g., measured in bits).

[0091] The system determines whether the savings value is greater than the cost value (510). In response to determining that the savings value is less than the cost value, the system may determine that the custom code symbol probability distribution should not be used for entropy encoding the candidate code symbol subset (e.g., as described in 312) (512). Instead, in response to determining that the savings value is greater than the cost value, the system may identify the custom code symbol probability distribution of the candidate code symbol subset (514). In this case, the candidate code symbol subset is entropy encoded using the custom code symbol probability distribution rather than the code symbol probability distribution from the dictionary (as described in reference 312).

[0092] Figure 6 is a flowchart of an example process for determining the reconstruction of data from a compressed representation of the data. For convenience, process 600 will be described as being performed by a system of one or more computers located at one or more locations. For example, a decoding system, such as Figure 2 the decoding system 200 of, can perform process 600.

[0093] The system receives a compressed representation of the data (e.g., data compressed by an encoding system, as described in reference Figure 3 )(602). The compressed representation of the data includes a compressed representation of each code symbol subset representing one or more code symbol subsets of the data (e.g., as described in reference 306). The compressed representation of each code symbol subset can include: (i) an entropy encoded representation of the code symbol subset, and (ii) data indicating the code symbol probability distribution used for entropy encoding the code symbol subset (e.g., as described in reference 314). When the code symbol probability distribution used for entropy encoding the code symbol subset is a custom code symbol probability distribution (e.g., as described in reference Figure 5 ), the compressed representation includes data defining the custom code symbol probability distribution. For example, the compressed representation can include a vector having corresponding numerical probability values for each code symbol in a set of discrete possible code symbols according to the custom code symbol probability distribution.

[0094] For each compressed representation of a code symbol subset, the system determines the code symbol subset based on the code symbol probability distribution used for entropy encoding the code symbol subset (604). For example, conditioned on the code symbol probability distribution used for entropy encoding the code symbol subset, the system can determine the mapping between portions of the entropy encoded representation of the code symbol subset and the corresponding code symbols. The system can use this mapping to determine the code symbol subset from the compressed representation of the code symbol subset.

[0095] The system determines an ordered set of code symbols representing data from a subset of code symbols (e.g., as described in reference 304) (606). For example, the system can concatenate the subset of code symbols to determine the ordered set of code symbols representing data.

[0096] The system determines an (approximate or exact) reconstruction of the data based on the ordered set of code symbols (608). For example, if the ordered set of code symbols is the coefficients of a representation of the data with respect to a set of basis functions (e.g., Fourier basis), the system can determine the reconstruction of the data as a linear combination of the basis functions and the coefficients given by the set of code symbols. As another example, if the ordered set of code symbols is the output of an encoder neural network, the system can determine the reconstruction of the data by providing the set of code symbols as input to a decoder neural network. The decoder neural network is configured to process the set of code symbols according to the current values of the decoder neural network parameters to generate an output defining the reconstruction of the data. In a particular example, if the encoder neural network is a convolutional neural network, the decoder neural network can be a corresponding deconvolutional neural network.

[0097] This specification uses the term "configured" in connection with systems and computer program components. For a system consisting of one or more computers, being configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination of them installed on it, which in operation causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when run by a data processing apparatus, cause the apparatus to perform the operation or action.

[0098] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus.

[0099] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code for creating a runtime environment for a computer program, such as, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0100] A computer program, which may also be referred to as or described as a program, software, a software application, an application, a module, a software module, a script, or code, can be written in any form of programming language, including a compiled or interpreted language, or a declarative or procedural language; and it can be deployed in any form, including as a stand-alone program or as a module, a component, a subroutine, or other unit suitable for use in a computing environment. A program may or may not correspond to a file in a file system. A program may be stored in a part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, stored in a single file dedicated to the program in question, or stored in multiple cooperating files, such as files that hold one or more modules, subroutines, or portions of code. A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0101] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same or multiple computers.

[0102] The processes and logical flows described in this specification can be performed by one or more programmable computers that run one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuitry, such as, for example, an FPGA or an ASIC, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0103] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor or both, or on any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing the instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0104] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0105] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or an LCD (liquid crystal display); and a keyboard and a pointing device through which the user may provide input to the computer, such as a mouse or a trackball. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user device in response to a request received from the web browser. Further, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and in turn receiving a response message from the user.

[0106] The data processing apparatus for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing common and computationally intensive portions of machine learning training or production, i.e., inference, workload.

[0107] The machine learning model may be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0108] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes backend components, such as a data server; or includes middleware components, such as an application server; or includes frontend components, such as a client computer having a graphical user interface, a web browser, or an application through which the user may interact with the implementations of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the Internet.

[0109] A computing system may include a client and a server. The client and the server are typically far apart from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and has a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interactions, can be received at the server from the device.

[0110] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what is claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from that combination, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0111] Similarly, although operations are depicted in the figures and recited in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Additionally, the separation of the various system modules and components in the above embodiments should not be understood as required in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0112] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the figures do not necessarily need the particular order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

[0113] According to a first further aspect of the present disclosure, there is provided a system for compressing data, the system comprising: an encoder neural network configured to perform operations that include: receiving data to be compressed; processing the data according to current values of a set of encoder neural network parameters to generate an output comprising a representation of the data as an ordered set of code symbols; a compression subsystem configured to perform operations that include: receiving the output of the encoder neural network, the output of the encoder neural network comprising a representation of the data as an ordered set of code symbols; identifying a partitioning of the set of code symbols into one or more code symbol subsets; for each code symbol subset: identifying, from a dictionary of multiple code symbol probability distributions and based on the code symbol subset, a code symbol probability distribution for the code symbol subset; generating an entropy-coded representation of the code symbol subset by entropy-coding the code symbol subset using the code symbol probability distribution identified for the code symbol subset; determining a compressed representation of the code symbol subset, the compressed representation comprising: (i) the entropy-coded representation of the code symbol subset, and (ii) data indicating the code symbol probability distribution used for entropy-coding the code symbol subset; determining a compressed representation of the data based on the compressed representations of the code symbol subsets.

[0114] According to a second further aspect of the present disclosure, there is provided a computer-implemented method for compressing data, the method comprising: receiving data to be compressed; processing the data through an encoder neural network and according to current values of a set of encoder neural network parameters to generate an output comprising a representation of the data as an ordered set of code symbols; identifying a partitioning of the set of code symbols into one or more code symbol subsets; for each code symbol subset: identifying, from a dictionary of multiple code symbol probability distributions and based on the code symbol subset, a code symbol probability distribution for the code symbol subset; generating an entropy-coded representation of the code symbol subset by entropy-coding the code symbol subset using the code symbol probability distribution identified for the code symbol subset; determining a compressed representation of the code symbol subset, the compressed representation comprising: (i) the entropy-coded representation of the code symbol subset, and (ii) data indicating the code symbol probability distribution used for entropy-coding the code symbol subset; determining a compressed representation of the data based on the compressed representations of the code symbol subsets.

[0115] Optionally, in the first and / or second further aspect, the data to be compressed comprises an image.

[0116] Optionally, in the first and / or second further aspect, the encoder neural network is a convolutional neural network.

[0117] Optionally, the system of the first aspect further includes a learning subsystem configured to perform operations including: processing data in a set of training data through an encoder neural network and according to a current value of a set of encoder neural network parameters to generate a representation output of the data in the set of training data including the data as an ordered set of code symbols; for each ordered set of code symbols, identifying a partition of the ordered set of code symbols into one or more subsets of code symbols; and determining a dictionary of multiple code symbol probability distributions based on the distribution of code symbols in the subsets of code symbols of the ordered set of code symbols.

[0118] Optionally, the method of the second aspect further includes processing data in a set of training data through an encoder neural network and according to a current value of a set of encoder neural network parameters to generate an output representing the data in the set of training data as an ordered set of code symbols; for each ordered set of code symbols, identifying a partition of the ordered set of code symbols into one or more subsets of code symbols; and determining a dictionary of multiple code symbol probability distributions based on the distribution of code symbols in the subsets of code symbols of the ordered set of code symbols.

[0119] Optionally, in the first and / or second further aspects, determining the dictionary of code symbol probability distributions includes: for each subset of code symbols of the ordered set of code symbols, determining a corresponding code symbol probability distribution reflecting the occurrence frequency of code symbols in the subset of code symbols; clustering the determined code symbol probability distributions; and determining the dictionary of code symbol probability distributions based on the cluster centers of the clustered code symbol probability distributions.

[0120] Optionally, in the first and / or second further aspects, clustering the code symbol probability distributions includes determining the distance between code symbol probability distributions based on a divergence metric.

[0121] Optionally, in the first and / or second further aspects, identifying the code symbol probability distribution of a subset of code symbols includes: for each code symbol probability distribution in the dictionary, determining a corresponding length of the entropy-coded representation of the subset of code symbols based on the code symbol probability distribution; and identifying the code symbol probability distribution of the subset of code symbols as the code symbol probability distribution from the dictionary having the minimum corresponding length of the entropy-coded representation.

[0122] Optionally, in the first and / or second further aspects, determining the length of the entropy-coded representation of a subset of code symbols based on the code symbol probability distribution includes determining, according to the code symbol probability distribution, the sum of the logarithms of the probabilities of the code symbols on each code symbol in the subset of code symbols.

[0123] Optionally, operations performed by the compression subsystem of the first further aspect further include: determining one or more subsets of code symbols as candidate subsets of code symbols; determining a custom code symbol probability distribution based on the candidate subsets of code symbols; determining a savings value based on a reduction in the length of the entropy-coded representation of the candidate subsets of code symbols, the reduction in the length of the entropy-coded representation of the candidate subsets of code symbols being achieved by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution rather than the code symbol probability distribution from the dictionary; determining a cost value based on the data length required to represent the custom code symbol probability distribution; and in response to determining that the savings value is greater than the cost value, generating an entropy-coded representation of the code symbols of the candidate subsets of code symbols by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution.

[0124] Optionally, the method of the second further aspect further includes: determining one or more subsets of code symbols as candidate subsets of code symbols; determining a custom code symbol probability distribution based on the candidate subsets of code symbols; determining a savings value based on a reduction in the length of the entropy-coded representation of the candidate subsets of code symbols, the reduction in the length of the entropy-coded representation of the candidate subsets of code symbols being achieved by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution rather than the code symbol probability distribution from the dictionary; determining a cost value based on the data length required to represent the custom code symbol probability distribution; and in response to determining that the savings value is greater than the cost value, generating an entropy-coded representation of the code symbols of the candidate subsets of code symbols by entropy-coding the candidate subsets of code symbols using the custom code symbol probability distribution.

[0125] Optionally, in the first and / or second additional aspects, identifying the partitioning of the set of code symbols into one or more subsets of code symbols includes identifying the partitioning of the set of code symbols into multiple distinct proper subsets.

[0126] According to a third further aspect, the present disclosure provides a system for decompressing data, the system comprising: a decompression subsystem configured to perform operations including: receiving a compressed representation of data, the compressed representation of the data including a compressed representation of each of one or more subsets of code symbols representing an ordered set of code symbols of the data, wherein the compressed representation of a subset of code symbols includes: (i) an entropy-coded representation of the subset of code symbols, and (ii) data indicating a code symbol probability distribution used for entropy-coding the subset of code symbols; determining each subset of code symbols by decoding the entropy-coded representation of the subset of code symbols based on the code symbol probability distribution used for entropy-coding the subset of code symbols; determining a representation of the data as an ordered set of code symbols based on the subsets of code symbols; and a decoder neural network configured to perform operations including: receiving the representation of the data as an ordered set of code symbols determined by the decompression subsystem; processing the ordered set of code symbols according to a current value of a set of decoder neural network parameters to generate an output defining a reconstruction of the data.

[0127] According to a fourth further aspect, the present disclosure provides a computer-implemented method, comprising: receiving, by a decompression subsystem, a compressed representation of data, the compressed representation of the data including a compressed representation of each of one or more subsets of code symbols representing an ordered set of code symbols of the data, wherein the compressed representation of a subset of code symbols includes: (i) an entropy-coded representation of the subset of code symbols, and (ii) data indicating a code symbol probability distribution used for entropy-coding the subset of code symbols; determining, by the decompression subsystem, each subset of code symbols by decoding the entropy-coded representation of the subset of code symbols based on the code symbol probability distribution used for entropy-coding the subset of code symbols; determining, by the decompression subsystem, a representation of the data as an ordered set of code symbols based on the subsets of code symbols; and processing, by the decoder neural network, the ordered set of code symbols according to a current value of a set of decoder neural network parameters to generate an output defining a reconstruction of the data.

[0128] Optionally, in the third and / or fourth further aspects, the data indicating the code symbol probability distribution used for entropy-coding the subset of code symbols includes data indicating a code symbol probability distribution from a dictionary of code symbol probability distributions.

[0129] Optionally, in the third and / or fourth further aspects, the data indicating the code symbol probability distribution used for entropy-coding the subset of code symbols includes data defining a respective numerical probability value for each code symbol in a set of possible code symbols.

[0130] It will be appreciated that the optional features of the first and second further aspects may also be applied to the third and fourth further aspects.

[0131] According to a fifth further aspect, the present disclosure further provides a non-transitory computer-readable storage medium storing instructions executable by a data processing apparatus, and which, when so executed, causes the data processing apparatus to perform the operations according to the second or fourth further aspects.

Claims

1. A system for compressing data, wherein, The system includes one or more computers and one or more storage media, where the one or more storage media store: A dictionary of probability distributions of multiple code symbols, where each probability distribution of code symbols includes data indicating the corresponding numerical probability value of each code symbol in a discrete set of possible code symbols; and Instructions that, when executed by the one or more computers, cause the one or more computers to implement: An encoder neural network configured to perform operations including: Receiving data to be compressed; Processing the data according to the current values of the encoder neural network parameter set to generate an output that includes a representation of the data as an ordered set of code symbols that are members of the discrete set of possible code symbols; and A compression subsystem configured to perform operations including: Receiving the output of the encoder neural network, where the output of the encoder neural network includes a representation of the data as an ordered set of code symbols that are members of the discrete set of possible code symbols; Identifying a partition of the set of code symbols into multiple subsets of code symbols; For each subset of code symbols: Identifying the probability distribution of code symbols of the subset of code symbols from the dictionary of probability distributions of multiple code symbols, where identifying the probability distribution of code symbols of the subset of code symbols includes: For each probability distribution of code symbols in the dictionary, determining the corresponding length of the entropy-coded representation of the subset of code symbols based on the probability distribution of code symbols; and Identifying the probability distribution of code symbols of the subset of code symbols as the probability distribution of code symbols from the dictionary having the minimum corresponding length of the entropy-coded representation; Generating an entropy-coded representation of the subset of code symbols by entropy-coding the subset of code symbols using the probability distribution of code symbols identified for the subset of code symbols; Determining a compressed representation of the subset of code symbols, where the compressed representation includes: (i) the entropy-coded representation of the subset of code symbols, and (ii) data indicating the probability distribution of code symbols used for entropy-coding the subset of code symbols; and Determining a compressed representation of the data based on the compressed representation of the subset of code symbols.

2. The system according to claim 1, wherein The data to be compressed includes an image.

3. The system according to claim 1, wherein, The encoder neural network is a convolutional neural network.

4. The system according to claim 1, further comprising a learning subsystem configured to perform operations including: Processing data in a training dataset by the encoder neural network and according to the current values of the encoder neural network parameter set to generate an output that includes a representation of the data in the training dataset as an ordered set of code symbols; For each ordered set of code symbols, identifying a partition of the ordered set of code symbols into one or more subsets of code symbols; And Determining the dictionary of probability distributions of multiple code symbols based on the distribution of code symbols in the subsets of code symbols of the ordered set of code symbols.

5. The system according to claim 4, wherein Determining the dictionary of probability distributions of multiple code symbols includes: For each in a subset of code symbols of an ordered set of code symbols, determine a corresponding code symbol probability distribution reflecting the frequency of occurrence of the code symbols in the subset of code symbols; Cluster the determined code symbol probability distributions; and Determine a dictionary of the plurality of code symbol probability distributions based on the cluster centers of the clustered code symbol probability distributions.

6. The system according to claim 5, wherein, Clustering the code symbol probability distributions includes determining the distance between code symbol probability distributions based on a divergence metric.

7. The system according to claim 1, wherein, Based on the code symbol probability distribution, determining the length of the entropy-coded representation of the subset of code symbols includes: Determining, according to the code symbol probability distribution, the sum of the logarithms of the probabilities of the code symbols on each code symbol in the subset of code symbols.

8. The system according to claim 1, wherein: The operations performed by the compression subsystem further include: Determining one or more subsets of code symbols in the set of discrete possible code symbols as candidate subsets of code symbols; Determining a custom code symbol probability distribution based on the candidate subset of code symbols; Determining a savings value based on a reduction in the length of the entropy-coded representation of the candidate subset of code symbols, the reduction in the length of the entropy-coded representation of the candidate subset of code symbols being achieved by entropy-coding the candidate subset of code symbols using the custom code symbol probability distribution instead of the code symbol probability distribution from the dictionary; Determining a cost value based on the length of the data required to represent the custom code symbol probability distribution; and In response to determining that the savings value is greater than the cost value, generating an entropy-coded representation of the code symbols of the candidate subset of code symbols by entropy-coding the candidate subset of code symbols using the custom code symbol probability distribution.

9. The system according to any one of claims 1-8, wherein, Identifying a partition of a set of code symbols into a plurality of subsets of code symbols includes identifying a partition of the set of code symbols into a plurality of different proper subsets.

10. A system for decompressing data, wherein, The system includes one or more computers and one or more storage media, wherein the one or more storage media store: A dictionary of a plurality of code symbol probability distributions, wherein each code symbol probability distribution includes data indicating a corresponding numerical probability value for each code symbol in a set of discrete possible code symbols; and Instructions that, when executed by the one or more computers, cause the one or more computers to implement: A decompression subsystem configured to perform operations including: Receiving a compressed representation of data, the compressed representation of data including a compressed representation of each of a plurality of subsets of code symbols representing an ordered set of code symbols that are members of the set of discrete possible code symbols, wherein the compressed representation of a subset of code symbols includes: (i) an entropy-coded representation of the subset of code symbols, and (ii) data indicating the code symbol probability distribution from the dictionary of code symbol probability distributions used for entropy-coding the subset of code symbols; Determining each subset of code symbols by decoding the entropy-coded representation of the subset of code symbols based on the code symbol probability distribution used for entropy-coding the subset of code symbols; Based on the subset of code symbols, determining the representation of the data as an ordered set of code symbols; and A decoder neural network configured to perform operations including: Receiving a representation of data as an ordered set of code symbols determined by a decompression subsystem; and Processing the ordered set of code symbols according to a current value of a set of decoder neural network parameters to generate an output defining a reconstruction of the data; Wherein each code symbol probability distribution in the dictionary determines a corresponding length of an entropy-coded representation of a subset of code symbols based on a code symbol probability distribution for entropy-encoding the subset of code symbols; and The code symbol probability distribution of the subset of code symbols identifies the code symbol probability distribution from the dictionary having the smallest corresponding length of the entropy-coded representation.

11. The system according to claim 10, wherein Data indicating a code symbol probability distribution for entropy-encoding the subset of code symbols includes data defining corresponding numerical probability values for each code symbol in a set of possible code symbols.

12. A computer-implemented method for compressing data, the method including: Storing a dictionary of a plurality of code symbol probability distributions, wherein each code symbol probability distribution includes data indicating corresponding numerical probability values for each code symbol in a discrete set of possible code symbols Receiving data to be compressed; Processing the data by an encoder neural network and according to a current value of a set of encoder neural network parameters to generate an output including a representation of the data as an ordered set of code symbols that are members of the discrete set of possible code symbols; Identifying a partitioning of the set of code symbols into a plurality of subsets of code symbols; For each of the plurality of subsets of code symbols: Identifying from the dictionary of the plurality of code symbol probability distributions a code symbol probability distribution of the subset of code symbols, wherein identifying the code symbol probability distribution of the subset of code symbols includes: For each code symbol probability distribution in the dictionary, determining a corresponding length of an entropy-coded representation of the subset of code symbols based on the code symbol probability distribution; and Identifying the code symbol probability distribution of the subset of code symbols as the code symbol probability distribution from the dictionary having the smallest corresponding length of the entropy-coded representation; Generating an entropy-coded representation of the subset of code symbols by entropy-encoding the subset of code symbols using the code symbol probability distribution identified for the subset of code symbols; Determining a compressed representation of the subset of code symbols, the compressed representation including: (i) the entropy-coded representation of the subset of code symbols, and (ii) data indicating the code symbol probability distribution used for entropy-encoding the subset of code symbols; Determining a compressed representation of the data based on the compressed representation of the subset of code symbols.

13. The method according to claim 12, wherein, The data to be compressed includes an image.

14. The method according to claim 12, wherein, The encoder neural network is a convolutional neural network.

15. A non-transitory computer-readable storage medium storing instructions executable by a data processing device, which when executed, cause the data processing device to perform operations including: Storing a dictionary of a plurality of code symbol probability distributions, wherein each code symbol probability distribution includes data indicating corresponding numerical probability values for each code symbol in a discrete set of possible code symbols; and Receiving data to be compressed; Process data by an encoder neural network and according to a current value of a set of encoder neural network parameters to generate an output, the output including a representation of the data as an ordered set of code symbols that are members of a discrete set of possible code symbols; Identify a partitioning of the set of code symbols into a plurality of code symbol subsets; For each code symbol subset: Identify a code symbol probability distribution of the code symbol subset from a dictionary of a plurality of code symbol probability distributions, wherein identifying the code symbol probability distribution of the code symbol subset includes: For each code symbol probability distribution in the dictionary, determine a corresponding length of an entropy-coded representation of the code symbol subset based on the code symbol probability distribution; and Identify the code symbol probability distribution of the code symbol subset as the code symbol probability distribution from the dictionary having the minimum corresponding length of the entropy-coded representation; Generate an entropy-coded representation of the code symbol subset by entropy-encoding the code symbol subset using the code symbol probability distribution identified for the code symbol subset; Determine a compressed representation of the code symbol subset, the compressed representation including: (i) the entropy-coded representation of the code symbol subset, and (ii) data indicating the code symbol probability distribution used to entropy-encode the code symbol subset; Determine a compressed representation of the data based on the compressed representation of the code symbol subset.

16. The non-transitory computer-readable storage medium according to claim 15, wherein, The data to be compressed includes an image.

17. The non-transitory computer-readable storage medium according to claim 15, wherein, The encoder neural network is a convolutional neural network.

Images