Lossless data compression and reconstruction using language model neural networks

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for receiving a data item of a first modality that is not text and generating a compressed data item Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for receiving a compressed data item and generating a lossless reconstruction of the data item of a first modality that is not text. Compressing data items and decompressing compressed data items both include the use of language model neural networks to uncover the complex structure within data items. For compressing data items, the use of the language model neural network allows compression systems to achieve better compression rates than traditional methods across a range of data modalities that are not text.

Description

[0001]DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application LOSSLESS DATA COMPRESSION AND RECONSTRUCTION USING LANGUAGE MODEL NEURAL NETWORKS CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to U.S. Provisional Application No.63 / 538,040, filed on September 12, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND This specification relates to compressing and reconstructing data items using machine learning models. As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights. SUMMARY This specification describes a compression system implemented as computer programs on one or more computers in one or more locations that compresses data items using a language model neural network. This specification also describes a decompression system implemented as computer programs on one or more computers in one or more locations that decompresses a compressed data item (also referred to as a compressed representation of a data item) to generate a reconstruction of the original data item. Generally, the compression and decompression systems may be co-located or remotely located, and compressed data items generated by the compression system can be provided to the decompression system in any of a variety of ways. For example, the compressed data item may be stored (e.g., in a physical data storage device or logical data storage area), and then subsequently retrieved from storage and provided to the decompression system. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application As another example, the compressed data item may be transmitted over a communications network (e.g., the Internet) to a destination, where it is subsequently retrieved and provided to the decompression system. The data items can generally have any appropriate modality that is different from text, i.e., that is not text, and, in particular, the compression system can use the same language model neural network to compress data items of multiple, non-text modalities without needing to re-train the neural network. Broadly, in implementations the system processes a data item to obtain a sequence of tokens that represent the item. The data item, which is not text, is nonetheless represented using text tokens (as an example, the text tokens can be ASCII characters). The language model neural network is used to process the sequence of text tokens, in particular to generate a probability distribution for each of the text tokens in the sequence. These probabilities can then be used to encode the text tokens, e.g. using an entropy coding technique such as so-called “arithmetic coding” (a certain type of coding that is a particular form of entropy coding). These are examples of coding techniques that construct a code, e.g. a binary code word representing the sequence, that has a length (in bits) that takes into account the probabilities of the encoded text tokens, in particular using shorter codes for more frequent, higher probability sequences. Thus, a probabilistic model (deterministic in the sense that the same output is always generated for the same input) can be used for lossless compression of a sequence of text tokens. It is very surprising that a model trained on a corpus of text is also effective at compressing other types of data such as images and audio, can be better (in terms of compression rate) than specialized image and audio compressors, and can serve as a general purpose compressor. The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The computational cost of generating, transmitting, storing, and processing raw data, i.e., non-compressed data item, is directly proportional to the size of the raw data. By using data compression technology, the cost of manipulating data items can be reduced significantly, and the performance of technologies that leverage data items can be boosted. Traditional compression and decompression systems, e.g., gzip, LZMA2, PNG, FLAC, and so on, rely on algorithms that identify and replace repeated patterns with references (for compression) and reverses the process (for decompression). Despite the general applicability of these methods to different data modalities, e.g., text, image, audio, DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application etc., and their favorable time complexity, these methods have limited compression rates (compressed memory size / raw memory size) due to a lack of context awareness, i.e., lack of in-context learning ability, i.e., lack of ability to detect complex patterns within the content of the data item being compressed. That is, traditional data compression methods treat data items to be compressed as simple sequences without identifying the potentially complex structure present. This specification, on the other hand, describes compression and decompression systems that use language model neural networks to uncover the complex structure within data items to achieve better compression rates than traditional methods, especially if the modality of the training data and the data item to be compressed match. But even when the training data and the data item to be compressed are of different modalities, the components of the large language model neural network, e.g., attention blocks, can identify complex patterns within the data item, resulting in impressive compression rates. Thus, by representing non-text data as a sequence of text tokens, the compression system can use a large language model to achieve impressive compression performance on data of any modality. In addition, a language model neural network belonging to a compression system can be trained specifically on the data item that is to be compressed so that the compression system can achieve even greater compression rates, referred to as compressing in an online fashion, as opposed to an offline fashion, i.e., using pre-trained large language model neural networks for compression and decompression of data items. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. According to a first aspect there is provided a method performed by one or more computers. The method includes receiving a data item of a first modality, where the data item of the first modality includes a plurality of numerical values. Then, generating a sequence of text tokens from a vocabulary of text tokens for a language model neural network that represents the data item of the first modality, where the first modality is not text. Then lastly, generating a lossless compression of the data item of the first modality by processing an input that includes the input sequence of text tokens using the language model neural network. In some implementations, the language model neural network has been pre-trained on text data. In addition, in some implementations, after the pre-training, the language model neural network has not been fine-tuned on any data of the first modality. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application In some implementations, the language model neural network has not been trained on data of the first modality. In some implementations, data of the first modality makes up less than five percent of a total amount of training data on which the language model neural network has been trained. In other implementations, data of the first modality makes up less than one percent of a total amount of training data on which the language model neural network has been trained. In some cases, generating a lossless compression of the data item of the first modality by processing an input that includes the input sequence of text tokens using the language model neural network includes: processing the input that includes the input sequence of text tokens using the language model neural network to generate a respective probability distribution for each of the text tokens in the input sequence, and encoding the input sequence of text tokens using the respective probability distributions for the text tokens in the input sequence to generate a sequence of bits, where the lossless compression includes the sequence of bits. For such cases, the encoding can be entropy encoding or arithmetic encoding. In some cases, the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and has a first number of representable numerical values that is greater than a total number of text tokens in the vocabulary of text tokens. For such cases, generating a sequence of text tokens from the vocabulary of text tokens includes, for each numerical value, identifying a first subset of the plurality of bits representing the numerical value and a second subset of the plurality of bits representing the numerical value and mapping the second subset of the plurality of bits to a corresponding text token in the vocabulary. In other cases, the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and has a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens. For such cases, generating a sequence of text tokens from the vocabulary of text tokens includes mapping each numerical value to a corresponding text token in the vocabulary. In some implementations, when the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and has a first number of representable numerical values that is greater than a total number of text DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application tokens in the vocabulary of text tokens, the lossless compression includes the sequence of bits and the first subset of bits for the plurality of numerical values. According to a second aspect there is provided a method performed by one or more computers. The method includes receiving a data item of a first modality, where the data item of the first modality includes a plurality of numerical values, and losslessly compressing the data item of the first modality using a language model neural network, where the language model neural network has been pre-trained on text and where the first modality is not text. In some cases of the first or second aspect, the data item includes an image; the data item includes a video; and the data item includes an audio signal. In some cases of the first or second aspect, the language model neural network is configured to process an input sequence to generate as output a probability distribution over a vocabulary of text tokens, where the probability assigned to each text token by the probability distribution represents a likelihood that the text token follows a last input in the input sequence. According to a third aspect there is provided a method performed by one or more computers. The method includes receiving a compressed representation of a data item of a first modality, where the data item of the first modality includes a plurality of numerical values and generating a lossless reconstruction of the data item of the first modality includes generating, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network. In some implementations of the third aspect, generating, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network includes generating, using the language model neural network, probability distributions for entropy decoding the compressed representation. In some cases of the third aspect, the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and has a first number of representable numerical values that is greater than a total number of text tokens in the vocabulary of text tokens. For such cases, generating a lossless reconstruction of the data item of the first modality includes mapping each text token in the sequence to a respective second subset of the plurality of bits representing a corresponding one of the numerical values. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application In some cases of the third aspect, the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and that has a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens. For such cases, generating a lossless reconstruction of the data item of the first modality includes mapping each text token in the lossless reconstruction of the sequence to a corresponding numerical value. In some implementations of the third aspect, when the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and has a first number of representable numerical values that is greater than a total number of text tokens in the vocabulary of text tokens, the compressed representation includes a respective first subset of the plurality of bits representing each of the numerical values. Also, generating a lossless reconstruction of the data item of the first modality includes, for each numerical value, combining the respective first and second subsets of the plurality of bits representing each numerical value. In some cases of the third aspect, the data item includes an image; the data item includes a video; and the data item includes an audio signal. In some cases of the third aspect, the language model neural network is configured to process an input sequence to generate as output a probability distribution over a vocabulary of text tokens, where the probability assigned to each text token by the probability distribution represents a likelihood that the text token follows a last input in the input sequence. According to a fourth aspect there is provided the methods of the first or second or third aspect performed by one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method. According to a fifth aspect there is provided the methods of the first or second or third aspect performed by one or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 shows an example compression system. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application FIG.2 shows an example decompression system. FIG.3 is a flow diagram of an example process for compressing a data item. FIG.4 is a flow diagram of an example process for decompressing a compressed data item. FIG.5 is a visualization of an example of arithmetic encoding a sequence of text tokens ‘AIXI’ and arithmetic decoding of compressed data item ‘0101010’B using a language model neural network. FIG.6 shows an example of the performance of the described techniques. DETAILED DESCRIPTION In general, a compression system as described herein generates a lossless compression of a data item by processing an input sequence of text tokens representing the data item, using a language model neural network. In implementations this generates a respective probability distribution for each of the text tokens in the input sequence, used to encode the input sequence of text tokens to generate a sequence of bits. The lossless compression of the input sequence of text tokens includes the sequence of bits. The data item is of a modality that is not text, i.e. it does not include text but instead may include any other sort of data, e.g. audio data or image data, but is nonetheless very effectively compressed using the language model neural network. There are various techniques that can be used to perform lossless encoding using a probabilistic model, e.g. Huffman coding (Huffman, “A method for the construction of minimum-redundancy codes”, Proceedings of the IRE, 1952), arithmetic coding (Pasco, “Source coding algorithms for fast data compression”, PhD thesis and IEEE Trans. Inf. Theory, 1977; Rissanen, “Generalized kraft inequality and arithmetic coding”, IBM J. Res. Dev., 1976), and ANS (asymmetrical numerical systems) coding (Duda, “Asymmetric numeral systems”, arXiv:0902.0271, 2009). Notionally the coding can assign a binary code of a particular length to each possible input sequence of text tokens, based on the probability distribution for each of the text tokens in the input sequence, aiming to minimize expected number of bits per sequence. Arithmetic coding can be particularly useful as it is known to construct codes of almost optimal length. An example of an arithmetic coding process is visualized in FIG.5, described further later. Broadly, the real interval ^^ ൌ ^0,1^ is iteratively partitioned according to the probability distribution for each of the text tokens in the input sequence, until the probability of the input sequence is fully contained in ^^ (here ^^ is different to the character DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application “I” represented in FIG.5). In the illustrated example ^^0 → ^0,0.5^, ^^01 → ^0.25,0.5^ and so forth. The probability of “A” is 0.45 and the range of the column for “A” is 0 to 1. In the column for “B” the range for the column has been expanded to 0 to 0.45, as illustrated by the arrows, and the probability of “I” given “A” is 0.6, which results in a combined probability of ^0.09,0.36^ (explained in detail later). This is not fully contained in ^0.25,0.5^ so is labelled “b0?” in the figure, and the iterative process continues. Decompression is straightforward and involves partitioning the interval I = [0,1) according to the probabilities of all text tokens in the vocabulary as determined by the language model neural network and determining which contains the probability corresponding to the compressed data item to be decoded. This continues iteratively until the input sequence of text tokens is recovered, and this is then mapped back to the data item. FIG.1 shows an example compression system 100. The compression system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The compression system 100 is a system that receives a data item 102 and generates a compressed data item 112. That is, the system 100 receives a data item 102 of a first modality that is not text and that includes a plurality of numerical values; generates a sequence of text tokens 106 from a vocabulary of text tokens for a language model neural network 110 that represents the data item 102 of the first modality; and generates a lossless compressed data item 112 of the first modality by processing an input that includes the input sequence of text tokens 106 using the language model neural network 110. Generally, a data item 102 of a given modality is represented as a plurality of numerical values and the modality of the data item 102 can be text, image, audio and so on. For example, the data items 102 can include images, i.e., data items of the “image” modality. In this example, the numerical values can represent the intensity values of the pixels of the image, e.g., RGB values or greyscale values. As another example, the data items 102 can include videos, i.e., data items of the “video” modality. In this example, the numerical values can represent the intensity values of the pixels of the video frames in the video. As yet another example, the data items 102 can include audio signals, i.e., data items of the “audio” modality. In this example, the numerical values can represent audio samples DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application of the audio data, e.g., amplitude values of an audio waveform or compressed amplitude values of the audio waveform. As another example, the data items 102 can be three-dimensional images that are made up of voxels of the image in a three-dimensional coordinate space, e.g., a three- dimensional grid, where the numerical values represent one or more intensity values of each of the voxel. Examples of such images include computer tomography (CT) image, a magnetic resonance imaging (MRI) image, an ultrasound image, an X-ray image, a mammogram image, a fluoroscopy image, or a positron-emission tomography (PET) image. As another example, the data items 102 can be point clouds, made up of multiple points in a three-dimensional coordinate space, where the numerical values represent at least a respective intensity each respective point. The values can optionally represent other components of a point in a point cloud, e.g., return, second return, elongation, and so on. For example, the data item can be a point cloud captured by a laser sensor, e.g., a LiDAR sensor. As can be seen from the above examples, the data items 102 can be any appropriate signal sensed by one or more sensors, e.g., one or more sensors configured to sense a real- world environment. The language model neural network 110 is a neural network that is configured to process an input, e.g. a sequence of text tokens, to generate an output that includes a probability distribution over a set of text tokens in vocabulary of text tokens, with the probability for each token representing the likelihood that the text token immediately follows the input. The vocabulary of text tokens can include any appropriate tokens that appear in natural language text, e.g., ASCII characters, words, word pieces, or differently distributed n-grams. For example, the vocabulary of text tokens can be fixed or can have been generated by applying an appropriate tokenizer, e.g., a byte pair encoding tokenizer or the SentencePiece tokenizer to a corpus of text. That is, as a more particular example, the text tokens can comprise ASCII coded characters (e.g. with an alphabet size of 128 or 256), although these do not require a tokenizer as such. For example, the language model neural network 110 can be an auto-regressive language model neural network. The language model neural network 110 is referred to as an auto-regressive neural network because the neural network auto-regressively generates an output sequence of tokens by generating each particular token in the output sequence conditioned on a current DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have for already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence (a “context sequence”). For example, the current input sequence when generating a token at any given position in the output sequence can include the context sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the context sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the context and the current output sequence can be separated by one or more predetermined tokens within the current input sequence. More specifically, to generate a particular token at a particular position within a candidate output sequence, the neural network 110 can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective logit or a respective probability (which, as used here, includes a log probability), to each text token in the vocabulary of text tokens. The neural network 110 can then select, as the particular token, a text token from the vocabulary using the score distribution. For example, the neural network 110 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution. As a particular example, the language model neural network 110 can be an auto- regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution. The neural network 110 can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. Generally, however, the Transformer-based neural network includes a sequence of attention blocks, and, during the processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates at least the hidden state for the last token in given input sequence at least in part by applying self-attention to generate a respective output hidden state for the last token. The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block. In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution. Generally, prior to using the neural network 110, the system 100 or another training system pre-trains the language model neural network 110 on a language modeling task, e.g., a task that requires predicting, given a current sequence of text tokens, the next token that follows the current sequence in the training data. As a particular example, the language model neural network 110 can be pre-trained on a maximum-likelihood objective on a large dataset of text, e.g., text that is publicly available from the Internet or another text corpus. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application There are many such databases; examples are the enwik8 and enwik9 datasets of the Hutter Prize, which include the first 10଼and 10ଽbytes respectively of an English Wikipedia dump. In such cases, the language model neural network 110 has not been trained on data of the first modality (a non-text modality). That is, the language model neural network 110 can be pre-trained on text, such as books, websites, essays, and not be pre-trained on data of a non-text modality, such as images, videos, audio samples, and so on. But despite being pre-trained on only text data, the language model neural network 110 can be used to compress non-text modality data represented as text tokens. Additionally, in some cases, after pre-training on text data, the language model neural network 110 is also not fine-tuned on any data of the first modality. That is, the language model neural network 110 is never pre-trained or fine-tuned on non-text modality data, but it can still be used to compress non- text modality data represented as text tokens. In some other implementations, the language model neural network has been trained on data of the first modality. That is, the system encodes the numerical values of the data item of the first modality as text tokens and includes it in the training data for the language model neural network. For such implementations the data of the first modality can make up less than five, or less than one percent of a total amount of training data on which the language model neural network trains on. For example, if the first modality data is image data, the system can preprocess the image data into a sequence of text tokens, e.g., representing the plurality of numerical values that represents pixel values at particular image coordinates as text, e.g., (frame height = 10, frame width = 20, pixel intensity = 100) can be represented through a sequence of text tokens as ‘(10,20,100)’, and the language model neural network can be trained to auto- regressively predict the sequence of text that encodes the image data. As another example, if the first modality data is a video data, the system can preprocess the video data into a sequence of text tokens, e.g., representing the plurality of numerical values that represents the video, i.e., 4D tensor where the dimensions contain information on the image frame index, image frame x coordinate, image frame y coordinate, and image grayscale intensity, as text, e.g., (frame index = 123, frame height = 10, frame width = 20, pixel intensity = 100) can be represented through a sequence of text tokens as “(123,10,20,100)”, and the language model neural network can be trained to auto- regressively predict the sequence of text that encodes the video data. As another example, if the first modality data is audio data, e.g., audio samples, e.g., amplitude values of an audio waveform, the system can preprocess the audio data into a DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application sequence of text tokens, e.g., representing the plurality of numerical values that represents the sequence of amplitude values of an audio waveform as text, e.g., (time step = 123, amplitude= 10) can be represented through a sequence of text tokens as “(123,10)”, and the language model neural network can be trained to auto-regressively predict the sequence of text that encodes the audio data. Although the numerical values included in the data item in these previous three examples are written in decimal format for ease of reading, the numerical values included in the data item can be any numerical format, e.g., binary. In some implementations, to compress a data item 102 of a first modality, e.g., an image, a video, an audio signal, and so on, that has a plurality of numerical values, the compression system 100 generates a sequence of text tokens 106 from a vocabulary of text tokens for the language model neural network 110. Generally, the sequence of text tokens 106 represents the data item 102 of the first modality, e.g., represents some or all of the bits that are used to represent the numerical values of the data item 102. Further details of generating a sequence of text tokens 106 from a vocabulary of text tokens for a language model neural network 110 that represents the data item 102 of the first modality are described below with reference to FIG.3. The system 100 then generates a lossless compression of the data item 102 of the first modality by processing an input that includes the input sequence of text tokens 106 using the language model neural network 110. The input can also optionally include one or more other text tokens, e.g., as a prompt for the language model neural network. For example, the system 100 can process the input that includes the input sequence of text tokens 106 using the language model neural network 110 to generate a respective probability distribution for each of the text tokens in the input sequence and then encode, e.g., using arithmetic or entropy encoding, the input sequence of text tokens 106 using the respective probability distributions for the text tokens in the input sequence to generate a sequence of bits, i.e., the compressed data item 112. Thus, the lossless compression, i.e., the compressed data item 112, includes the bitstream, i.e., sequence of bits, generated by encoding the input sequence of text tokens 106 using the respective probability distributions for the text tokens. Further details of generating a lossless compression of the data item 102 of the first modality by processing an input that includes the input sequence of text tokens 106 using the language model neural network 110 are described below with reference to FIG.3. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application FIG.2 shows an example decompression system 200. The decompression system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The compression system 100 and decompression system 200 can be co-located or remotely located, and compressed data item 112 generated by the compression system 100 can be provided to the decompression system as compressed data item 212. The system 200 receives a compressed data item 212, i.e., a compressed representation of the data item 202, e.g., a representation that includes a sequence of bits. The system 200 then generates a lossless reconstruction of the data item 202 of the first modality, that includes generating, using the language model neural network 210, a lossless reconstruction of a sequence of text tokens 206 from a vocabulary of text tokens for the language model neural network 210. The decompression system 200 can then map each text token in the sequence to at least a portion of the bits used to represent one of the numerical values in the data item 202. For example, the system 200 can decode, e.g., using entropy or arithmetic decoding, the sequence of tokens 206 in the lossless reconstruction using language model neural network 210, e.g., by, as part of the decoding, using the language model neural network 210 to generate a probability distribution for each of the text tokens conditioned on the already- decoded text tokens that precede the text token in the sequence of text tokens 206. Then, the system 200 can map the lossless reconstruction of the sequence of text tokens 206 to the plurality of numerical values included in the data item 202. Further details of generating a lossless reconstruction of the data item 202 of the first modality from the compressed representation of the data item 212, are described below with reference to FIG.4. FIG.3 is a flow diagram of an example process 300 for compressing a data item. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a compression system, e.g., the compression system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 300. The system receives a data item of first modality (step 302). As described earlier, generally the data item of a first modality is represented as a plurality of numerical values and the modality of the data item can include text, image, video, audio signal and so on. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application The system generates a sequence of text tokens from a vocabulary of text tokens for a language model neural network that represents the data item of the first modality (step 304). That is, the system encodes the data item as a sequence of text tokens. In some implementations, the plurality of numerical values included in the data item are represented in a first numerical format that represents each numerical value as a plurality of bits. The numerical values can then have a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens. For such implementations, the system generates a sequence of text tokens from the vocabulary of text tokens by mapping each numerical value to a corresponding text token in the vocabulary. As a specific example, if the data item is of a time series integer temperature senor modality, the plurality of numerical values included can be Celsius readings with a range of [0°C, 127°C] for a sequence of time readings (i.e. each numerical value for each time reading is represented as a 7 bit representation with 2^7 or 128 representable numerical values). If the total number of text tokens in the vocabulary of text tokens is 128, then the system can map each Celsius temperature reading number to a corresponding text token through a one-to-one mapping. In other implementations, the first number of representable numerical values is greater than a total number of text tokens in the vocabulary of text tokens. For such implementations, the system generates a sequence of text tokens from the vocabulary of text tokens by, for each numerical value, identifying a first subset of the plurality of bits representing the numerical value and a second subset of the plurality of bits representing the numerical value. Then mapping the second subset of the plurality of bits to a corresponding text token in the vocabulary. (The first subset of the plurality of bits can be included in, e.g. prepended or appended to, the compressed data representation, as described later). As a specific example, if the data item is of a time series integer temperature sensor modality in an extended Celsius reading range (i.e. the temperature readings range is [0°C, 255°C] and each numerical value is represented as a 8 bit representation with 2^8 or 256 representable numerical values) but the total number of text tokens in the vocabulary of text tokens is only 128, the system can identify the least significant 7 bit numbers of the 8 bit representation as the second subset of the plurality of bits. The system can then map each numerical value’s second subset to one of the 128 text tokens. As another example, if the data item is an image patch of size (32, 64) pixels in grayscale such that the intensity value for each pixel fall in the range of [0, 255], i.e., each DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application pixel intensity corresponds to an 8 bit representation with 2^8 or 256 representable numerical values, but the total number of text tokens in the vocabulary of text tokens is only 128, the system can identify the most significant 7 bit numbers of the 8 bit representation as the second subset of the plurality of bits and map each pixel’s numerical value’s second subset to one of the 128 text tokens. As another example, if the data item is sound data that includes samples of 2 byte size (i.e., each sample is encoded as a 16 bit representation with 2^16 or 65536 representable numerical values), but the total number of text tokens in the vocabulary of text tokens is only 128 the sampling rate of the sample can be halved or the size of a sample can be reduced from 2 bytes to 1 byte, then the system can identity the least significant 7 bit numbers of the 8 bit representation as the second subset of the plurality of bits and map each sound sample’s numerical value’s second subset to one of the 128 text tokens. The system generates a lossless compression of the data item of the first modality by processing an input that that includes the input sequence of text tokens using the language model neural network (step 306). For example, the system can process an input that includes the input sequence of text tokens using the language model neural network to generate a respective probability distribution for each of the text tokens in the input sequence. For example, for a sequence ^^ of text tokens of length ^^ from a vocabulary (or alphabet) of ^^ tokens each partial sequence ^^ழ^(where ^^ runs from 1 to ^^) can be processed by the language model neural network to obtain a probability distribution ^^^^^| ^^ழ^^for each token ^^ in the vocabulary (or alphabet). Where the sequence of text tokens is longer than a context input of the language model neural network the sequence may be divided into sequences that are equal to or less than the context length and processed one by one (or a sliding window can be used). A score distribution output from the language model neural network can be normalized to obtain a probability distribution. In some implementations there may only be access to the top ^^ token log probabilities, and then these top ^^ log probabilities can be re-normalized to sum to 1, before being processed as described below. Then, the system can encode the input sequence of text tokens to generate a lossless compression that includes a sequence of bits. Specifically, the system can use the respective probability distributions for the text tokens in the input sequence to generate the sequence of bits. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application As a more specific example, building on the previous example, the system can use entropy encoding or arithmetic encoding to encode the input sequence of text tokens using the respective probability distributions for the text tokens in the input sequence. In some cases, when the system identifies a first subset of the plurality of bits representing the numerical value and a second subset of the plurality of bits representing the numerical value, the compressed data item includes the first subset of bits for the plurality of numerical values. For example, with respect to the example of a data item that includes time series integer temperature senor modality in the extended Celsius reading range for which the system omitted the most significant bit of the 8 bit representation to map each numerical value to one of the 128 text tokens, the dropped most significant bit can be prepended or appended to the compressed data item representation. More specifically, if the compressed data item is a sequence of binary bits, the first subset of bits, i.e., the dropped most significant bit, can be prepended or appended to the compressed representation of the data item. As a particular illustration of the use of arithmetic coding, the arithmetic code of a sequence of text tokens ^^^:^can be the binary representation of a number ^^ ∈ ^0,1^. A value for ^^ can be identified by narrowing down an interval that encloses ^^ step by step (maintaining a growing prefix on the binary representation of ^^ throughout the process). Initially the interval is ^^^ൌ^0,1^. In step ^^ ^ 0 (i.e. encoding ^^^) the previous interval^^^ି^ ൌ ^ ^^^ି^, ^^^ି^^ is first partitioned into ^^ sub-intervals ^^^^^ ^^^^, ^^^^^ ^^ଶ^, …, one for eachtext token from the vocabulary ^^ ൌ ^ ^^^, ^^ଶ, … , ^^ே^ (i.e. where lower and upper bounds ^^^and ^^^ are defined iteratively). The size of the sub-interval ^^^^^^^^that represents token ^^ is^^^^ି^ െ ^^^ି^^∙ ^^^^^| ^^ழ^^and (assuming a defined order on the text tokens in ^^) ^^^^^^^^canbe defined as: and ^^^can be encoded using its corresponding interval, i.e. ^^^ൌ ^^^^^ ^^^^. Then ^^ can be chosen to represent ^^^(formally choosing ^^ ∈ ^^^as the shortest binary representation within the final interval ^^^), and the binary representation of ^^ is used to encode ^^^:^. Further details of an example of arithmetic encoding an input sequence of text tokens using the respective probability distributions for text tokens in the vocabulary are described below with reference to FIG.5. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application FIG.4 is a flow diagram of an example process 400 for decompressing a compressed data item. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a decompression system, e.g., the decompression system 200 of FIG.2, appropriately programmed in accordance with this specification, can perform the process 400. The system receives a compressed representation of a data item of a first modality (step 402). Generally, the compressed representation of a data item (also referred to as the compressed data item) can be received in any of a variety of ways, either from the same system or from a different system. For example, the system can receive the compressed data item over a communication network (e.g. the Internet). As described earlier, generally the modality of the compressed data item can include text, image, video, audio signal and so on. The system generates, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network (step 404). For example, the system can generate, using the language model neural network, probability distributions for entropy decoding the compressed representation. Then, using the probability distributions, the system can decode the compressed representation and generate a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network. As a more specific example, building on the previous example, the system can use entropy decoding or arithmetic decoding to decode the compressed data item. Continuing the previous particular illustration of the use of arithmetic coding, given ^^ and ^^, decoding the ^^th text token can proceed as follows: Starting with ^^^ൌ ^0,1^, find^^ such that ^^ ∈ ^^^^^^^^to decode ^^^ ൌ ^^, then set ^^^ ൌ ^^^^^^^^^and continue with the^ ^^ ^ 1^th letter. Further details of arithmetic decoding a compressed data item using the respective probability distributions for text tokens in the vocabulary are described below with reference to FIG.5. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application The system generates a lossless reconstruction of the data item of the first modality (step 406). That is, the system decodes the sequence of text tokens as a data item. In some implementations, the plurality of numerical values included in the data item are represented in a first numerical format that represents each numerical value as a plurality of bits. The numerical values can then have a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens. For such implementations, the system maps each text token in the lossless reconstruction of the sequence to a corresponding numerical value to generate a lossless reconstruction of the data item of the first modality. As a specific example, if the first number of representable numerical values is 128, e.g., the first numerical format that represents each numerical value is a 7 bit number, and the total number of text tokens in the vocabulary of text tokens is 128, the system can map each text token to a corresponding numerical value to generate a lossless reconstruction of the data item of the first modality. In other implementations, the first number of representable numerical values is greater than a total number of text tokens in the vocabulary of text tokens. For such implementations, the system maps each text token in the lossless reconstruction of the sequence to a respective second subset of the plurality of bits representing a corresponding one of the numerical values to generate a lossless reconstruction of the data item of the first modality. For example, consider a data item that is an image patch of size (32, 64) pixels in grayscale such that the intensity value for each pixel falls in the range of [0, 255], that the compression system identified the least significant 7 bit numbers of the 8 bit representation of each pixel as the second subset of the plurality of bits and the most significant bit as the first subset of the plurality of bits, that the compression system mapped each pixel’s numerical value’s second subset to one of 128 text tokens in the vocabulary during compression, and that the compression system prepended the least significant bit of the pixel intensity, i.e., the first subset of bits, to the compressed data item representation. The system can generate, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network using the compressed data item without the prepended bit as described above with respect to step 404. Then, the system can map each text token to a 7 bit number that represents the second subset of the plurality of bits that correspond to the least significant 7 bit numbers of the 8 bit representation of the original pixel intensity. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application Further in other implementations, generating a lossless reconstruction of the data item of the first modality includes, for each numerical value, combining the respective first and second subsets of the plurality of bits representing each numerical value. For example, building upon the previous example where the first subset of the plurality of bits is the most significant bit of a pixel intensity value prepended to the compressed data item, the system can prepend the first subset of the plurality of bits to the second subset of the plurality of bits to generate a lossless reconstruction of 8 bit number representation of a pixel intensity in the data item. As previously described, an example of arithmetic coding can iteratively partition the interval between 0 and 1 according to the probabilities assigned to the input sequence of text tokens by the language model neural network. FIG.5 is a visualization of an example 500 of arithmetic encoding a sequence of text tokens ‘AIXI’ and arithmetic decoding of compressed data item ‘0101010’B using a language model neural network. When referring to FIG.5, the language model neural network is denoted as function ^^; the data item represented as a sequence of text tokens is denoted as ‘AIXI’ for which the sequence of text tokens are ‘A’,’I’,’X’, and ’I’; the vocabulary of text tokens is an ordered set (e.g., ‘A’ ≺ ‘I’ ≺ ‘X’ or any other consistent, defined ordering of the text tokens); and the compressed data item is the sequence of bits ‘0101010’B. Before beginning the Arithmetic encoding process, the system represents the data item of a (non-text) first modality as a sequence of text tokens, e.g., the system performs step 304 with reference to FIG.3 as described above. Regarding example 500 of FIG.5, consider the text sequence ‘AIXI’ as the representation of a 4-pixel grayscale (256 levels of intensity) image for ease of illustration. That is, a one by four pixel greyscale image with intensities in decimal format [65, 73, 88, 73] (and in bit format [01000001, 01001001, 01011000, 01001001]) can be mapped to a 256 text token vocabulary, such as the extended ASCII vocabulary, to be represented as ‘AIXI’. Arithmetic encoding involves iteratively partitioning the interval I = [0,1) according to the language model neural network ^^ and the data item to be encoded ‘AIXI’. Arithmetic encoding of ‘AIXI’ begins by subdividing the interval I = [0,1) proportionally to, and in the order of, the probabilities of all text tokens in the vocabulary for the language model neural network. That is, the interval [0,1) is partitioned DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application proportionally to P(A), P(I), and P(X) in that order starting from zero, in this example^0,0.45^ൌ 0.45 for P(A),^0.45,0.75^ൌ 0.3 for P(I), and^0.75,1^ൌ 0.25 for P(X).Next, the current interval is narrowed according to the first text token in the sequence of text tokens. The first text token is ‘A’ therefore P(A) determines the narrowed current interval as [0, 0.45). Then, for each remaining text token in the sequence of text tokens, the current interval is subdivided proportionally to, and in the order of, posterior probabilities of all text tokens in the vocabulary conditioned on previously processed text tokens and narrowed according to the current text token of the sequence. For example, for the second text token ‘I’, the current interval is subdivided according to P(A|A), P(I|A), and P(X|A) and because the second text token is ‘I’, P(I|A)narrows the current interval to be [0.09, 0.36). That is, ^0.2 ൈ 0,0.2 ൈ 0.45^ ൌ ^0,0.09^ ൌ0.2 for P(A|A); ^0.09, ^0.2 ^ 0.6^ ൈ 0.45^ ൌ ^0.09,0.36^ ൌ 0.3 for P(I|A); and^0.36, ^^0.2^ 0.6^ 0.2^ ൈ 0.45^ ൌ ^0.36,0.45^ ൌ 0.2 for P(X|A). As another example, for the final text token ‘I’, the current interval is subdivided according to P(A|AIX), P(I|AIX), and P(X|AIX) and because the final text token is ‘I’, P(I|AIX) narrows the current interval to be [0.322, 0.341). Once the final current interval is determined, i.e., [0.322,0.341), the compressed data item is determined by iteratively splitting the interval [0,1) in half and assigning a binary code to each sub-interval until the system discovers a sub interval fully contained in the interval [0.322,0.341). The binary code that corresponds to the subinterval that is fully contained then becomes the final compressed data item, and for the interval [0.322,0.341) is ‘0101010’B → [0.328125, 0.3359375). In this example: ^^0 →^0,0.5^^^01 →^0.25,0.5^^^010 → ^0.25,0.375^ ^^0101 → ^0.3125,0.375^ ^^01010 → ^0.3125,0.34375^ ^^010101 → ^0.328125,0.34375^ ^^0101010 → ^0.328125,0.3359375^ And since ^0.328125,0.3359375^ is fully contained in ^^ ൌ ^0.322,0.341^ the no further bits are needed and the compressed output is 0101010, representing the text sequence as 7 bits rather than as 4 bytes of, e.g., ASCII coding. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application Arithmetic decoding involves iteratively partitioning the interval I = [0,1) according to the language model neural network ^^ and the compressed data item to be decoded ‘0101010’B. Arithmetic decoding of ‘0101010’B begins by subdividing the interval I = [0,1) proportionally to, and in the order of, the probabilities of all text tokens in the vocabulary for the language model neural network probabilities P(A), P(I), and P(X). Next, the current interval is narrowed according to which text token probability interval contains the sub-interval [0.328125, 0.3359375) that corresponds to the compressed data item ‘0101010’B. For ‘0101010’B the sub-division corresponding to P(A) contains the sub-interval [0.328125, 0.3359375). Therefore, the first text token in the lossless reconstruction of a sequence of text tokens is ‘A’. Then, the current interval is iteratively subdivided proportionally to, and in the order of, posterior probabilities of all text tokens in the vocabulary conditioned on previously determined text tokens for the lossless reconstruction of a sequence of text tokens until a current interval is found to fully contain the sub-interval corresponding to the compressed data item, i.e., ‘0101010’B → [0.328125, 0.3359375) →’AIXI’. Each text token determined for each iteration corresponds to the lossless reconstruction of a sequence of text tokens. FIG.6 shows an example 600 of the performance of the described techniques. More specifically, FIG.6 shows an example 600 of the performance of the described techniques with respect to raw compression rates (compressed memory size / raw memory size) on different datasets (lower is better) relative to conventional techniques. The column labeled “Compressor” indicates if the compression system corresponds to the described techniques (e.g. Chinchilla 70B, i.e., the Chinchilla language model neural network with 70 billion parameters used with arithmetic coding to obtain lossless compression) or conventional techniques (e.g., gzip, LZMA2, PNG, or FLAC). The column labeled “Chunk” refers to the size of the data item being compressed where 2048 refers to the number of numerical values included in the data item, i.e., the number of text tokens in the sequence of text tokens representing the data item. The columns labeled ‘enwiki9’, ‘ImageNet’, and ’LibriSpeech’ refer to the datasets containing data items of text, image, and audio modality respectively. The Chinchilla language neural network is pretrained on a mix of internet text data (Wikipedia, websites, github) and books. DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application Example 600 illustrates that while conventional general-purpose compression systems, such as gzip, perform well on a wide range of data item modalities, the compression system using the described techniques (i.e., Chinchilla 70B) with a language model neural network trained only on text is a better general-purpose compression system and outperforms all other compression systems, even on image and audio data. More specifically, example 600 shows that the compression system using the described techniques (i.e., Chinchilla 70B) only requires 9% to 17% the memory that the conventional compression systems for the enwik9 dataset (text modality dataset) require. Similarly, for ImageNet (image modality dataset) and LibriSpeech (audio modality dataset), Chinchilla 70B only requires, respectively, 70% to 79% and 55% to 69% of the memory that the conventional compression systems require. One reason the compression system using the describe techniques (i.e., Chinchilla 70B) achieves its impressive compression performance is that its language model neural network can take advantage of in-context learning when producing probabilities for entropy encoding. That is, the language model neural network can identify complex patterns within the data item to produce probability distributions for the text tokens in the input sequence that result in concise compressed representations of data items. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application CLAIMS 1. A method performed by one or more computers, the method comprising: receiving a data item of a first modality, the data item of the first modality comprising a plurality of numerical values; generating a sequence of text tokens from a vocabulary of text tokens for a language model neural network that represents the data item of the first modality, wherein the first modality is not text; and generating a lossless compression of the data item of the first modality by processing an input that comprises the input sequence of text tokens using the language model neural network.

2. The method of claim 1, wherein the language model neural network has been pre- trained on text data.

3. The method of claim 2, wherein, after the pre-training, the language model neural network has not been fine-tuned on any data of the first modality.

4. The method of any preceding claim, wherein the language model neural network has not been trained on data of the first modality.

5. The method of any preceding claim, wherein data of the first modality makes up less than five percent of a total amount of training data on which the language model neural network has been trained.

6. The method of claim 5, wherein data of the first modality makes up less than one percent of a total amount of training data on which the language model neural network has been trained.DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application 7. The method of any preceding claim, wherein generating a lossless compression of the data item of the first modality by processing an input that comprises the input sequence of text tokens using the language model neural network comprises: processing the input that comprises the input sequence of text tokens using the language model neural network to generate a respective probability distribution for each of the text tokens in the input sequence; and encoding the input sequence of text tokens using the respective probability distributions for the text tokens in the input sequence to generate a sequence of bits, wherein the lossless compression comprises the sequence of bits.

8. The method of claim 7, wherein the encoding is entropy encoding or arithmetic encoding.

9. The method of any preceding claim, wherein: the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and that has a first number of representable numerical values that is greater than a total number of text tokens in the vocabulary of text tokens; and generating a sequence of text tokens from the vocabulary of text tokens comprises, for each numerical value: identifying a first subset of the plurality of bits representing the numerical value and a second subset of the plurality of bits representing the numerical value; and mapping the second subset of the plurality of bits to a corresponding text token in the vocabulary.

10. The method of any one of claims 1-8, wherein: the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and that has a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens; and generating a sequence of text tokens from the vocabulary of text tokens comprises mapping each numerical value to a corresponding text token in the vocabulary.DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application 11. The method of claim 9 when dependent on claim 7, wherein the lossless compression comprises the sequence of bits and the first subset of bits for the plurality of numerical values.

12. A method performed by one or more computers, the method comprising: receiving a data item of a first modality, the data item of the first modality comprising a plurality of numerical values; and losslessly compressing the data item of the first modality using a language model neural network, wherein the language model neural network has been pre-trained on text and wherein the first modality is not text.

13. The method of any preceding claim, wherein the data item comprises an image.

14. The method of any preceding claim, wherein the data item comprises a video.

15. The method of any preceding claim, wherein the data item comprises an audio signal.

16. The method of any preceding claim, wherein the language model neural network is configured to process an input sequence to generate as output a probability distribution over a vocabulary of text tokens, wherein the probability assigned to each text token by the probability distribution represents a likelihood that the text token follows a last input in the input sequence.

17. A method performed by one or more computers, the method comprising: receiving a compressed representation of a data item of a first modality, the data item of the first modality comprising a plurality of numerical values; and generating a lossless reconstruction of the data item of the first modality, comprising generating, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network.

18. The method of claim 17, wherein generating, using a language model neural network, a lossless reconstruction of a sequence of text tokens from a vocabulary of text tokens for the language model neural network comprises: generating, using the language model neural network, probability distributions for entropy decoding the compressed representation.DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application 19. The method of claim 17 or 18, wherein: the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and that has a first number of representable numerical values that is greater than a total number of text tokens in the vocabulary of text tokens; and generating a lossless reconstruction of the data item of the first modality comprises mapping each text token in the sequence to a respective second subset of the plurality of bits representing a corresponding one of the numerical values.

20. The method of any one of claims 17 or 18, wherein: the plurality of numerical values are represented in a first numerical format that represents each numerical value as a plurality of bits and that has a first number of representable numerical values that is fewer than or equal to a total number of text tokens in the vocabulary of text tokens; and generating a lossless reconstruction of the data item of the first modality comprises mapping each text token in the lossless reconstruction of the sequence to a corresponding numerical value.

21. The method of claim 19, wherein the compressed representation comprises a respective first subset of the plurality of bits representing each of the numerical values, and wherein generating a lossless reconstruction of the data item of the first modality comprises, for each numerical value, combining the respective first and second subsets of the plurality of bits representing each numerical value.

22. The method of any one of claims 17-21, wherein the data item comprises an image.

23. The method of any one of claims 17-22, wherein the data item comprises a video.

24. The method of any one of claims 17-23, wherein the data item comprises an audio signal.DeepMind Technologies Limited F&R Ref.: 45288-0369WO1 PCT Application 25. The method of any one of claims 17-24, wherein the language model neural network is configured to process an input sequence to generate as output a probability distribution over a vocabulary of text tokens, wherein the probability assigned to each text token by the probability distribution represents a likelihood that the text token follows a last input in the input sequence.

26. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1-25.

27. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-25.

Images