Compressing data exhibiting mixed compressibility

Abstract

Systems and methods are provided for compressing data that exhibits mixed compressibility, such as floating point data. As one example, aspects of the present disclosure can be used to compress floating point data representing parameter values of a machine learning model. Thus, aspects of the present disclosure can be used to compress a machine learning model (e.g., to reduce storage requirements associated with the model, to reduce bandwidth consumed to transmit the model, etc.).

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for compressing data exhibiting mixed compressibility. Specifically, this disclosure relates to compressing floating-point data, such as, for example, floating-point data encoding parameter values ​​of machine learning models. Background Technology

[0002] Information stored in a computing system can be stored as data in memory. For example, information can be stored as "bits" of data, where each bit is equal to 1 or 0.

[0003] It may be necessary to reduce the amount of data and / or memory used to store information (e.g., reduce the number of bits required to store data). Techniques used to reduce the memory used to store information are often referred to as compression techniques.

[0004] Lossless compression refers to compression techniques that allow for the accurate reproduction of the information obtained after compression. Lossy compression refers to compression techniques that only allow for estimations of the information to be recreated after compression.

[0005] In some cases, data can exhibit mixed compressibility. For example, a first part of the data may be highly suitable for one or more compression techniques (i.e., it may have high "compressibility"), while a second part may be unsuitable for compression techniques (i.e., it may have low "compressibility").

[0006] For example, if the application of compression technology results in a significant reduction in storage usage and / or only zero or a small amount of information loss, then the data can have high compressibility. However, if the application of compression technology results in only a small reduction in storage usage and / or a large amount of information loss, then the data can have low compressibility. Summary of the Invention

[0007] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0008] One example aspect of this disclosure relates to a computing system configured to compress data. The computing system includes one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to operate. The operation includes: obtaining subject data; performing a first compression technique to compress a first portion of the data and obtaining first compressed data, wherein a second portion of the data remains uncompressed after performing the first compression technique; transforming the second portion of the data to obtain transformed data; quantizing the transformed data to obtain quantized data; and storing the first compressed data and the quantized data.

[0009] Another example aspect of this disclosure relates to a computer-implemented method for transmitting compressed data. The method includes: receiving subject data at a first computing device. The method includes: determining a first portion and a second portion of the subject data by the first computing device. The method includes: transforming the second portion by the first computing device to obtain transformed data. The method includes: quantizing the transformed data by the first computing device to obtain quantized data. The method includes: transmitting the quantized data from the first computing device to a second computing device.

[0010] Another example aspect of this disclosure relates to a non-transitory computer-readable medium containing one or more storage instructions that, when executed by one or more computing devices, cause the computing devices to compress a machine learning model by performing operations. The operations include: obtaining model parameter data, which accordingly includes multiple floating-point numbers for multiple parameters of the machine learning model, wherein each of the multiple floating-point numbers includes one or more sign bits, multiple exponent bits, and multiple mantissa bits. The operations include, for each of the multiple floating-point numbers: generating first compressed data from one or more most significant mantissa bits of the one or more sign bits, multiple exponent bits, and multiple mantissa bits to obtain first compressed data; performing Kashin decomposition on one or more remaining least significant mantissa bits to obtain transformed data, the one or more remaining least significant mantissa bits including multiple mantissa bits not included in the one or more most significant mantissa bits from which the first compressed data was generated; performing uniform quantization on the transformed data to obtain quantized data; and storing the first compressed data and the quantized data.

[0011] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0013] This specification provides a detailed discussion of embodiments that are suitable for those skilled in the art, and the specification refers to the accompanying drawings, in which:

[0014] Figure 1A A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0015] Figure 1B A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0016] Figure 1CA block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0017] Figure 2 A block diagram depicts an example machine learning model according to an example embodiment of the present disclosure.

[0018] Figure 3 An example block diagram depicts example data according to an example embodiment of the present disclosure.

[0019] Figure 4 A flowchart depicts an example method for transforming data according to an example embodiment of the present disclosure.

[0020] Figure 5 A flowchart depicts an example method for storing compressed data according to an example embodiment of the present disclosure.

[0021] Figure 6 A flowchart depicts an example method for transmitting compressed data according to an example embodiment of the present disclosure.

[0022] The repeated reference numerals on multiple figures are intended to identify the same features in various embodiments. Detailed Implementation

[0023] Overview

[0024] Generally, this disclosure relates to systems and methods for compressing data exhibiting mixed compressibility, such as floating-point data. As an example, aspects of this disclosure can be used to compress floating-point data representing parameter values ​​of a machine learning model. Therefore, aspects of this disclosure can be used to compress machine learning models (e.g., to reduce storage requirements associated with the model, reduce bandwidth consumed by the model during transmission, etc.). Specifically, one aspect of this disclosure relates to a compression technique in which a first portion of highly compressible floating-point data is compressed using predictive techniques, while a second, less compressible portion of the floating-point data is compressed by combining decomposition (e.g., applying Kashin decomposition) with quantization (e.g., uniform quantization, bit truncation, etc.).

[0025] More specifically, information can be stored as data in a computing system. Data can represent words, numbers, or other suitable types of information in a format that can be processed and / or otherwise interpreted by the computing system. For example, data can be represented as a set of numbers (e.g., bits) in a storage device. Example storage devices include, but are not limited to, non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and combinations thereof.

[0026] It may be necessary to reduce the amount of memory used to store information. For example, it may be necessary to reduce the size of the data used to store the information. Techniques used to reduce the size of data are called data compression techniques. Lossless compression refers to compression techniques that allow data to be compressed without loss of information and then decompressed. Lossy compression refers to compression techniques that introduce irreversible errors into the data after compression and / or decompression, resulting in information loss. For example, lossy compression techniques can slightly increase and / or decrease the numerical values ​​represented by data. As another example, lossy compression techniques can cause a loss of precision associated with the data. As yet another example, lossy compression techniques can introduce blur or other distortions into an image.

[0027] Although it may be necessary to reduce or eliminate information loss, lossy compression allows for a significant reduction in data size compared to lossless compression, making it an option for some applications. For example, in many cases, the errors introduced by lossy compression do not significantly affect the functionality of the data.

[0028] Compression techniques may be needed to store and / or transmit large datasets, such as datasets with a large number of data items and / or a large number of numbers per data item. As an example, a machine learning model can be represented using a tensor with a large number of model parameter values ​​(e.g., the corresponding values ​​of millions of model parameters). In other words, the parameters of a machine learning model can be represented as a tensor with a large number (millions) of component values.

[0029] As another example, measurement data (such as time-series measurement data) can have a large sample size. Time-series measurement data can include sensor data, such as environmental sensor data (e.g., seismic data, temperature data, etc.), biosensor data, and / or any other data collected over time. Additionally, in many cases, model parameters and / or measurement data can be represented as floating-point data, which may require storing relatively large amounts of data. For example, in some cases, each item in the floating-point data set (e.g., each floating-point number) can be 32, 64, or 128 bits, which can significantly contribute to the size of the data.

[0030] Additionally, in some cases, a significant portion of the data can exhibit low compressibility. For example, low-compressibility data may exhibit a lack of statistical redundancy in the dataset, meaning that lossless compression techniques will not significantly reduce the data size. Throughout this disclosure, the term "incompressible" is generally used to refer to data that is low in compressibility. This term does not necessarily mean that the data cannot be compressed at all, but rather refers to data that is unsuitable for conventional compression techniques and / or data for which conventional compression techniques do not provide a satisfactory trade-off between computational overhead, information loss, and compression gain. As an example, data may be incompressible if it experiences a size reduction of less than a threshold amount (e.g., 10%) when compressed according to conventional compression methods.

[0031] As an example, floating-point data can be defined by a sign component, an exponent component, and a mantissa component (also known as a significantand component). In some cases, a small fraction of the sign, exponent, and / or mantissa components (e.g., approximately one to approximately three bits or digits) can have some statistical redundancy across the dataset, allowing them to be efficiently compressed using lossless compression techniques. However, in some cases, a large portion of the mantissa component can be incompressible. This can be particularly limiting of the effectiveness of lossless compression techniques when the mantissa component is the largest component in the floating-point data. For example, in some cases, the mantissa component can have approximately 75% to approximately 80% of the total size of the floating-point data. Similarly, in some cases, the size of the incompressible portion of the data can be approximately 75% to approximately 80% of the total size of the data.

[0032] One approach to this problem is to quantize at least a portion of the data. For example, quantization techniques might map the set of values ​​to an "index" during a compression phase and then map the index to a representative value for the entire set during a decompression phase. By performing this process, precision is removed because the entire set of values ​​is mapped to a single element (e.g., via the index). Therefore, quantization techniques can reduce the size of data by decreasing its precision (e.g., by removing less significant numbers from the data and truncating or rounding to the nearest lower precision value), at the cost of losing any information at the original precision. This can introduce error into the data, which can adversely affect the functionality of systems utilizing the data. The problem can be exacerbated by the fact that errors introduced by quantization can affect every data item in the data unevenly. For example, if the data includes parameters that define a machine learning model, some parameters may be more susceptible to error than others, negatively impacting the training and / or prediction tasks of the machine learning model.

[0033] The systems and methods based on the exemplary aspects of this disclosure can provide technical solutions to these and other problems. According to the exemplary aspects of this disclosure, one or more computing devices can receive subject data, such as datasets. A dataset may include multiple data items. For example, each of the data items may have a defined value, such as lexical and / or numerical. Data can be stored in any suitable format. For example, the data may be stored as one or more tensors. For example, each component of one or more tensors may correspond to a data item among multiple data items.

[0034] Subject data may be stored in any suitable manner, based on exemplary aspects of this disclosure. For example, data may be stored in any suitable memory device, such as one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and combinations thereof.

[0035] In some embodiments, data may include one or more numbers. For example, each of one or more data items may include one or more numbers. For example, one or more numbers may include one or more bits. In some embodiments, one or more bits may represent computer-readable information. For example, bits may encode information (e.g., numeric information, such as base-10 numeric information, lexical information, or any other suitable information) into a binary data string (e.g., base-2 numeric data) such that the information can be processed by a computing system or otherwise interpreted.

[0036] In some embodiments, the subject data may be floating-point data. For example, the data may represent floating-point numbers. For example, the data may be defined and / or interpreted according to a floating-point standard (such as, but not limited to, IEEE 754). For example, a floating-point standard may define a real value according to the following formula:

[0037]

[0038] Where s is the value of the sign component, b is the base value, e is the value of the exponent component, o is the offset value, p is the precision value, and m i It is the value of the i-th digit of the mantissa component.

[0039] In some embodiments, a floating-point standard may define a base or radix. The base may represent the number of values ​​that can be represented by a single digit. For example, for binary representation, the base may be two, or for decimal representation, the base may be ten. Additionally and / or alternatively, a floating-point standard may define precision. For example, precision may define the maximum number of digits in the mantissa component. Additionally and / or alternatively, a floating-point standard may define an exponent range. An exponent range may define a set of allowed values ​​for the exponent component of floating-point data. Additionally and / or alternatively, an exponent range may define the deviation or offset of the exponent component and / or the number of digits. Additionally and / or alternatively, a floating-point standard may define one or more reserved values. For example, reserved values ​​may represent special cases or other anomalies of the format, such as, but not limited to, infinity or values ​​that are infinite in representation (e.g., infinity, negative infinity), NaN (Not a Number), or any other suitable reserved value.

[0040] In some embodiments, floating-point data may include a sign component. For example, the sign component may define whether the number represented by the floating-point data is positive or negative. For example, the sign component may include a bit capable of having one of two values. The first of the two values ​​may indicate that the number is positive, while the second of the two values ​​may indicate that the number is negative. For example, logic low (e.g., zero) may indicate that the number is positive, while logic high (e.g., one) may indicate that the number is negative. For example, the sign component may be based on (-1). s Define the sign of a number, where s is the sign component.

[0041] In some embodiments, floating-point data may include an exponent component. The exponent component may represent the base raised to a power of 1. For example, the exponent component may define the order of magnitude of the number being represented. As an example, the exponent component may define the order of magnitude as b. (e-o) , where b is the base value, e is the value of the exponential component, and o is the offset value. For example, the offset value can be chosen such that the smallest possible exponent that can be represented by the standard is the minimum value (e.g., zero or one).

[0042] In some embodiments, floating-point data may include a mantissa component (or a significant digit component). The mantissa component defines the significant digits of the floating-point number. The value of the mantissa component gives the value of the floating-point number when multiplied by the value of the sign component and / or raised to the base value of the exponent component. In some embodiments, the mantissa component may be stored such that the first digit of the floating-point number has a known value. For example, in binary representation, the mantissa component may be defined as the value following the decimal point. In some cases, the value of the digit immediately preceding the decimal point is defined by a standard. For example, in binary representation, the value of the digit immediately preceding the decimal point may always be zero or always be one.

[0043] According to exemplary aspects of this disclosure, one or more computing devices can obtain a first portion and / or a second portion of subject data. For example, the first portion and / or the second portion may be a subset of the data. For example, the first portion and / or the second portion may include one or more numbers. In some embodiments, the first portion and / or the second portion is defined for each of a plurality of data items in the subject data. For example, each of the plurality of data items may be divided into a first portion and / or a second portion. In some embodiments, each of the plurality of data items may be split at the same point. For example, in one exemplary embodiment, the first portion is a compressible portion, while the second portion is an incompressible portion.

[0044] In some embodiments, the first portion may be a compressible portion of the subject data. The first (e.g., compressible) portion can be effectively compressed using lossless compression techniques. For example, the first (e.g., compressible) portion can be effectively compressed without information loss. For example, the size of the first (e.g., compressible) portion can be significantly reduced without information loss.

[0045] The first (e.g., compressible) portion can exhibit statistical redundancy, allowing at least a portion of the compressible portion to be predicted or otherwise recreated based on this redundancy. An example of statistical redundancy includes multiple data items across a topic having consecutively recurring values ​​of a portion of the data item (e.g., one or more numbers). For example, portions of the multiple data items may remain unchanged or otherwise change infrequently among the multiple data items. The compressible portion can exhibit statistical redundancy that allows it to be compressed using any suitable compression technique, such as, but not limited to, the Lempel-Zip-Markov chain algorithm (LZMA), the Burrows-Wheeler algorithm (e.g., bzip2), the DEFLATE algorithm (e.g., gzip), machine learning compression techniques, or any other suitable compression technique. For example, consecutively recurring values ​​can be represented as a value and the number of times that value is consecutively recurred. This representation may require less memory for storage and / or transmission compared to uniquely representing the value at each recurrence.

[0046] In some embodiments, the second portion may be an incompressible portion of the data. That is, the second (e.g., incompressible) portion may be a portion of the data that may not be effectively compressed according to lossless compression techniques. For example, the second (e.g., incompressible) portion may exhibit limitations due to the absence of statistical redundancy, such that the size of the second (e.g., incompressible) portion can be significantly reduced without information loss. For example, the second (e.g., incompressible) portion may include portions of multiple data items such that the values ​​of these portions exhibit very little or no repetition across the multiple data items, especially consecutive repetition. For example, the second (e.g., incompressible) portion may experience a size reduction of less than about 10% when compressed according to conventional compression methods. However, even when non-quantized values ​​are difficult to compress, indexes can generally benefit from LZMA / bzip or similar compression methods.

[0047] In some embodiments, the subject data may include multiple data blocks. The data blocks may be sequential. The data blocks may have similar or identical formats. In some embodiments where the subject data includes multiple data blocks, each data block may be compressed in general as described herein. For example, the corresponding data included in each data block may include a first (e.g., compressible) portion and a second (e.g., incompressible) portion. As described herein, first and second techniques may be used to compress the first and second portions of each corresponding data block.

[0048] In some embodiments, the first portion and / or the second portion may be portions of floating-point data. For example, the first portion may be a compressible portion of the floating-point data, while the second portion may be an incompressible portion of the floating-point data. For example, the first portion may include at least a portion of the sign component and / or the exponent component and / or the mantissa component. For example, at least a portion of the mantissa component may include approximately one to approximately three digits of the mantissa component. In some embodiments, approximately one to approximately three digits may be approximately one to approximately three of the most significant digits of the mantissa component. If a portion of the mantissa component is present in the first portion, then the second portion may include the mantissa component or the remainder of the mantissa component. The first portion and / or the second portion may include multiple data items. In other embodiments, different components of the data or their sub-components may be used to other allocations of two or more portions (e.g., in addition to the example allocations described above).

[0049] As an example application, in some cases, floating-point data defining machine learning model parameters can be stored as tensors. That is, subject data can be tensors, such as tensors storing machine learning model parameters. Each component in a tensor can correspond to a floating-point number that defines a unique machine learning parameter. Each component in a tensor can therefore have an associated sign component, exponent component, and / or mantissa component. In some cases, a portion of a tensor component can be compressible. For example, a portion of a tensor component (e.g., the sign and exponent components) can exhibit statistical redundancy across some aspects of the tensor (e.g., across one or more dimensions of the tensor). In some cases, a portion of a tensor component can be incompressible. For example, a portion of a tensor component (e.g., the mantissa component or a portion thereof) can be uniquely defined at each tensor component.

[0050] According to an example aspect of this disclosure, one or more computing devices can transform a portion of subject data to obtain transformed data. For example, one or more computing devices can transform an incompressible portion of the subject data. For example, one or more computing devices can transform a portion of the subject data such that the norm of that portion of the subject data is minimized.

[0051] For example, in some embodiments, transforming a portion of the subject data may include linearizing the subject data and / or linearizing a portion of the subject data. For example, linearizing the subject data and / or a portion of the subject data may include representing the subject data and / or a portion of the subject data as vectors. For example, in some cases, the subject data may be represented as a multidimensional matrix or tensor. In this case, a portion of the subject data and / or the subject data (e.g., an incompressible portion) may be linearized into a vector representation. Any suitable linearization method may be employed according to this disclosure.

[0052] One example transformation according to an exemplary aspect of this disclosure is the Kashin decomposition. The Kashin decomposition transforms a frame representation of vectors in a complex frame with n dimensions into a frame extension with N dimensions, N > n, where the coefficients have the minimum possible dynamic range, referred to herein as the Kashin representation. For example, the dynamic range of the coefficients could be approximately... The information contained in the vector can therefore be distributed approximately uniformly and / or evenly among the coefficients. For example, Kashin decomposition can divide the vector x and the frame... Transformed into a form with coefficient a i The frame representation makes For example, the coefficient can be defined as a i = <x,u i Specifically, the coefficients can be delimited by the level K. For example, the coefficients can be delimited by... Let ||x||2 be the boundary, where ||x||2 is the L2 norm of x. (See reference) Figure 4 An example method for transforming a vector x according to Kashin decomposition is discussed.

[0053] Transforming a portion of the topic data according to Kashin decomposition offers several advantages. For example, transforming the topic data according to Kashin decomposition can distribute the information contained in that portion of the topic data roughly uniformly and / or evenly among multiple coefficients. For instance, each iterative step of truncating the frame representation and determining the frame representation of the residuals can conceptually "shave" the information in x at larger coefficients and "redistribute" that information at smaller coefficients. This can be repeated until the error is distributed uniformly in the desired manner. Therefore, if an error is introduced into multiple coefficients (e.g., from the error of quantizing multiple coefficients), then the error is associated with multiple coefficients, rather than with the portion of the topic data. Thus, if a portion of the topic data is recreated from the transformed data with error, then the error is distributed roughly uniformly and / or evenly among the recreated data, which better preserves the portion of the topic data in the recreated data.

[0054] According to an example aspect of this disclosure, one or more computing devices can quantize transformed data to obtain quantized data. Quantization can reduce the precision associated with the transformed data. For example, the quantized data can have fewer numbers than the transformed data. By reducing the number of numbers, the quantized data can have a smaller size in terms of memory than the transformed data. According to an example aspect of this disclosure, any suitable quantization method can be employed. The quantization method may include compression techniques and / or the quantized data can be further compressed after quantization.

[0055] In some embodiments, the transformed data can be quantized using uniform quantization. For example, the transformed data can be rounded to the nearest value with a lower precision (e.g., a smaller number of digits) to obtain quantized data. For example, the quantized data can define multiple quantization levels, where the transformed data has values ​​that are generally between two quantization levels. In one embodiment, for example, if a transformed data item has a value that is between two quantization levels and less than the midpoint defined between the two quantization levels, then the value of the corresponding quantized data item is selected as the lower of the two quantization levels. Otherwise, the value of the corresponding quantized data item is selected as the higher of the two quantization levels.

[0056] In some embodiments, transformed data can be quantized by truncating at least a portion of the transformed data. For example, less significant bits (e.g., bits with relatively low associated magnitude values) can be truncated from the transformed data. In other words, the value of the transformed data can be rounded down to the nearest of a plurality of quantization levels, resulting in the rounded data having one or more trailing zeros. The one or more trailing zeros can be omitted from storage and / or recovered at a later point in computation to reduce the memory requirements for storing the quantized data compared to the transformed data.

[0057] In some embodiments, the transformed data can be quantized by mapping the transformed data to one of a plurality of index values. For example, in one example embodiment, the transformed data is rounded down to the nearest value associated with one of the plurality of index values, and the index value associated with the nearest value is stored, for example, as an alternative to storing the transformed data.

[0058] Quantizing transformed data to obtain quantized data can introduce quantization error into the quantized data. However, transforming the original data before quantization (e.g., through Kashin decomposition) allows the quantization error to be spread or "smeared" across the original data. In other words, the quantization error is not concentrated on a specific term in the original data, but rather spread across these terms. Therefore, while error is still introduced into the quantized data, it can be better handled in the application of the data. For example, machine learning models with data-defined parameters can be less affected by errors in the data compared to conventional compression.

[0059] According to an example aspect of this disclosure, one or more computing devices may store quantized data. For example, one or more computing devices may store quantized data in one or more memory devices. Example memory devices include, but are not limited to, non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and combinations thereof.

[0060] Additionally and / or alternatively, one or more computing devices may store a compressed portion of the subject data. For example, the compressed portion may be a compressible portion of the subject data compressed using any suitable compression technique, particularly lossless compression. For instance, the compressed portion may be compressed using any suitable compression technique, such as, but not limited to, the Lempel-Zip-Markov chain algorithm (LZMA), the Burrows-Wheeler algorithm (e.g., bzip2), the DEFLATE algorithm (e.g., gzip), machine learning compression techniques, or any other suitable compression technique.

[0061] Additionally and / or alternatively, one or more computing devices may transmit quantized data. For example, quantized data may be transmitted from a first computing device to a second computing device. As another example, quantized data may be transmitted from a first location (e.g., a first memory address) to a second location (e.g., a second memory address). Quantized data may be transmitted using any suitable transmission method. For example, it may be transmitted via wired transmission (e.g., Ethernet) and / or via wireless transmission (e.g., Bluetooth, ZigBee, WLAN, IEEE 802.11) or any other suitable transmission method or technology.

[0062] Additionally and / or alternatively, one or more computing devices may transmit a compressed portion of the subject data. For example, the compressed portion may be a compressible portion of the subject data compressed using any suitable compression technique, particularly lossless compression. For instance, the compressed portion may be compressed using any suitable compression technique, such as, but not limited to, the Lempel-Zip-Markov chain algorithm (LZMA), the Burrows-Wheeler algorithm (e.g., bzip2), the DEFLATE algorithm (e.g., gzip), machine learning compression techniques, or any other suitable compression technique.

[0063] In some embodiments, after storing and / or transmitting quantized data, one or more computing devices can reconstruct subject data based on compressed data and / or quantized data. For example, one or more computing devices can decompress compressed data to obtain a compressible portion from the compressed data. For example, a compressible portion can be obtained from compressed data without information loss.

[0064] Additionally and / or alternatively, one or more computing devices may reconstruct the incompressible portion from the quantized data. For example, one or more computing devices may reconstruct the transform data from the quantized data. In some embodiments, the transform data cannot be accurately reconstructed from the quantized data due to quantization errors introduced during quantization. In these cases, reconstructing the transform data may include estimating the transform data based on the quantized data. As another example, the quantized data may include an index mapped to a corresponding value in the transform data.

[0065] For example, quantized data can be mapped to corresponding values ​​in transformed data with less precision than the corresponding values. Therefore, reconstructing transformed data may require increasing the precision of the quantized data. For instance, precision can be improved by appending numbers to the quantized data. As an example, if the transformed data is 32 bits and the quantized data is 24 bits, then the precision of the quantized data is 8 bits less than that of the transformed data. In some embodiments, the most significant 24 bits of the transformed data are the same as the 24 bits of the quantized data. Therefore, to estimate the transformed data from the quantized data, 8 trailing (i.e., least significant) bits can be appended to the 24 bits of the quantized data. In some embodiments, the trailing bits can be zero. In some embodiments, the quantized data can be jittered so that the trailing bits can have random values. By introducing controlled noise into the reconstructed data, jitter allows the reconstructed transformed data to be offset from the quantized values.

[0066] Additionally and / or alternatively, the incompressible portion can be reconstructed from the transformed data (such as reconstructed transformed data). For example, an inverse transform can be applied to the transformed data to undo the transform and restore the original data. For example, in the case where the incompressible portion is transformed by Kashin decomposition, the incompressible portion can be reconstructed from the frame extension of the incompressible portion. For example, if the incompressible portion is represented as having frames sum coefficient a i If the frame representation of vector x is given, then the original data can be calculated as follows: Additionally and / or alternatively, if the incompressible portion is transformed by linearization, then the incompressible portion can be reconstructed by restoring the dimensions to the transformed data.

[0067] Various advantages can be achieved by storing and / or transmitting the subject data as both compressed and transformed / quantized data components. For example, the compressed data component can be transmitted without information loss. Furthermore, compressed data can be transmitted efficiently due to the ability to achieve a high level of compression. Additionally, although errors are introduced into the transformed / quantized data, these errors have a smaller impact on the application of the transformed / quantized data compared to quantized data alone.

[0068] For example, one technical effect of the exemplary aspects of this disclosure is to provide a system and method for lossy data compression of low-compressibility data, such that the compression error inevitably resulting from the compression of low-compressibility data is distributed across the low-compressibility data. For example, the compression error can be distributed across multiple data items in the low-compressibility data. By distributing the compression error, certain characteristics of the dataset can be better preserved after compression and decompression compared to conventional methods. For example, if the low-compressibility data defines the parameters of a machine learning model, then the tuning characteristics of the parameters can be better preserved. As another example, if the low-compressibility data defines measurements, then the measurements and / or the relationships between the values ​​of the measurements can be better preserved. As yet another example, errors can be more easily compensated for by having a more uniform effect across the low-compressibility data.

[0069] Another technical effect of aspects of this disclosure is that it provides techniques that allow for higher levels of data compression than conventional methods without adversely affecting the system using the data. For example, in cases where the subject data defines the parameters of a machine learning model, the parameters can be compressed to a higher level without adversely affecting the tuning of the machine learning model. For example, systems and methods according to exemplary aspects of this disclosure can be particularly advantageous in embodiments where the level of error and / or value compression is more sensitive to the speed of compression and / or decompression. Furthermore, some lossy compression can even increase the robustness of the model.

[0070] For example, the systems and methods according to this disclosure can be applied to federated learning techniques for machine learning models to reduce the amount of data transferred and / or improve the accuracy of the machine learning models. In federated learning techniques, a machine learning model can be trained at one or more computing devices using data found at one or more computing devices. The machine learning model can then be transferred to a second computing device (e.g., a device seeking to utilize the machine learning model). The benefits of federated learning include, for example, the ability to train a model using training data and to use the model elsewhere without necessarily needing to transfer the training data. This is particularly useful when the training data is private or, or where it is not desired, to transfer it.

[0071] More specifically, in some example applications, a server can send a machine learning model (e.g., partially trained, possibly "blank") to several devices. The model can be compressed as described in this article. This version of the model can be referred to as... Each device can perform tasks as described in this article. Decompression is then performed. Each device can then be trained based on local (e.g., private) data. To refine the copy of the model. Assuming this is for the k-th device, the result is... The local update of the refinement model can be performed as a form of compression, as described in this paper. Or as an update difference compressed in the same way as described in this article Send it back to the server. The server can then respond to each local update from each device k (e.g., ...). or Decompression can be performed and the refined model can be aggregated. The process can continue iteratively. For example, it can be sent again to the group of devices (e.g., compressed). For use and / or further training. In some implementations, the group of devices at t+1 may differ from the group of devices in the previous step t and / or the local data used for updates may differ. Therefore, training can continue with the above steps, while the devices can use the model (e.g., assuming the quality is sufficient) for local inference. It is also possible to use the updated aggregated model on the server side. In some implementations, for privacy interests, the server does not materialize (e.g., store on disk or other supporting devices) the model updates received from the devices, but instead keeps all model updates in volatile memory until aggregation.

[0072] As another example of a technologically beneficial application, the compression / decompression scheme described herein can be used in a model-sharing framework, where a compressed model is sent from a first device (e.g., a server) to a second device (e.g., a client, such as a user's device). As another example, the first and second devices can be peer devices, rather than a server / client. The first device can also provide a codec (also referred to as a compressor / decompressor) to the second device, which the second device can use to decompress and / or recompress or update the model. The first device can provide new or updated codecs to the second device over time.

[0073] As another example of technical effect and benefit, reduced transmission bandwidth is possible. However, specifically, transmitting machine learning models may require significant network bandwidth because machine learning models can include large amounts of data. Furthermore, lossless compression techniques can have limited effectiveness due to the uniqueness of model parameters. Additionally, conventional lossy compression techniques can introduce errors into model parameters, particularly unevenly, which can affect the tuning of the machine learning model. However, by employing the system and method according to this disclosure, the model can be transmitted using less data than with lossless compression, while having fewer negative impacts on the tuning of the machine learning model.

[0074] Example devices and systems

[0075] Figure 1AA block diagram of an example computing system 100 according to an exemplary embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0076] User computing device 102 can be any type of computing device, such as (for example), personal computing device (e.g., laptop computer or desktop computer), mobile computing device (e.g., smartphone or tablet), game console or controller, wearable computing device, embedded computing device, or any other type of computing device.

[0077] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 that, when executed by processor 112, cause user computing device 102 to operate.

[0078] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Reference Figure 2 Example machine learning model 120 is discussed.

[0079] In some implementations, one or more machine learning models 120 may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. For example, model 120 may be based on... Figure 4 The method for storing compressed data illustrated herein is stored in the user computing device memory 114. As another example, model 120 can be based on... Figure 5The method illustrated for transmitting compressed data is used between user computing device 102 and / or server computing system 130. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., performing parallel prediction tasks across multiple instances of relevant and / or irrelevant input features).

[0080] Alternatively or additionally, one or more machine learning models 140 may be included in, stored in, or otherwise implemented by, server computing system 130, which communicates with user computing device 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 140 as part of a web service (e.g., a machine learning service). Thus, one or more models 120 may be stored and implemented at user computing device 102 and / or one or more models 140 may be stored and implemented at server computing system 130.

[0081] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other components that a user can use to provide user input.

[0082] User computing device 102 may include one or more compressors / decompressors 124. The compressor / decompressor 124 may compress and / or decompress data according to any of the compression techniques described herein. The compressor / decompressor 124 may be included in a particular application or may be implemented by the operating system of user computing device 102. The compressor / decompressor 124 includes computer logic for providing desired functionality. The compressor / decompressor 124 may be implemented using hardware, firmware, and / or software that controls a general-purpose processor.

[0083] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 that, when executed by processor 132, cause server computing system 130 to operate.

[0084] In some embodiments, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, these server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0085] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be, or can otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Reference Figure 2 Example model 140 was discussed.

[0086] Server computing device 130 may include one or more compressors / decompressors 142. The compressor / decompressor 142 may compress and / or decompress data according to any of the compression techniques described herein. The compressor / decompressor 142 may be included in a particular application or may be implemented by the operating system of server computing system 130. The compressor / decompressor 142 includes computer logic for providing desired functionality. The compressor / decompressor 142 may be implemented using hardware, firmware, and / or software that controls a general-purpose processor.

[0087] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled through network 180. Training computing system 150 may be separate from or part of server computing system 130. For example, models 120 and / or 140 may be trained at training computing system 150 and transferred to user computing device 102 and / or server computing system 130. For example, it can be based on... Figure 5The method for transmitting compressed data illustrated in the figure is used to transmit models 120 and / or 140.

[0088] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that, when executed by the processor 152, cause the training computing system 150 to operate. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0089] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques, such as (e.g.) backpropagation of error, to train machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., gradients based on the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update parameters across multiple training iterations.

[0090] In some implementations, backpropagation of the error may include performing truncated backpropagation across time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropouts, etc.) to improve the generalization ability of the trained model.

[0091] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, pairs of input features and associated category labels. For example, input features can include features for natural language processing, such as raw or processed language information. As another example, input features can include image features, such as raw or processed images. As another example, input features can include features for content recommendation services, such as web usage or other suitable information. Input features can have associated labels. For example, labels can indicate the desired aspect of training, such as the true category associated with some or all of the input features.

[0092] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Therefore, in this implementation, the model 120 provided to the user computing device 102 can be trained by training the computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as a personalized model.

[0093] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more computer-executable instruction sets stored in a tangible computer-readable storage medium (such as RAM, hard disk, optical media, or magnetic media). In some embodiments, model trainer 160 can perform any of the compression / decompression techniques described herein.

[0094] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication via network 180 can be conducted using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0095] Figure 1A An example computing system that can be used to implement this disclosure is illustrated. Other computing systems may also be used. For example, in some embodiments, user computing device 102 may include a model trainer 160 and a training dataset 162. In this embodiment, model 120 can be trained and used locally at user computing device 102. In some such embodiments, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data. As another example, in some embodiments, training computing system 150 may include a compressor and / or decompressor (e.g., as described at 124 and 142). By having a compressor / decompressor at training computing system 150, training computing system 150 can participate in the example federated learning applications described above. For example, all communication across network 180 may occur in compressed form.

[0096] Figure 1BA block diagram of an example computing device 100 according to an exemplary embodiment of the present disclosure is depicted. The computing device 100 may be a user computing device or a server computing device.

[0097] Computing device 10 includes several applications (e.g., applications 1 to N). Example applications contain their own machine learning libraries and machine learning models. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0098] like Figure 1B As illustrated, each application can communicate with several other components of the computing device, such as (for example), one or more sensors, a context manager, a device status component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0099] Figure 1C A block diagram of an example computing device 50 according to an exemplary embodiment of the present disclosure is depicted. The computing device 50 may be a user computing device or a server computing device.

[0100] Computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0101] The central intelligence layer includes several machine learning models. For example, such as... Figure 1C As illustrated, a corresponding machine learning model (e.g., a model) may be provided for each application and managed by a central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of computing device 50.

[0102] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized storage repository for data from computing device 50. For example... Figure 1CAs illustrated, the central device data layer can communicate with several other components of the computing device, such as (e.g., one or more sensors, a field manager, a device status component, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a proprietary API).

[0103] Figure 2 A block diagram of an example machine learning model 200 according to an exemplary embodiment of the present disclosure is depicted. In some embodiments, the machine learning model 200 is trained to receive a set of input data 204 describing input features, and as a result of receiving the input data 204, output data 206 indicating one or more outputs (e.g., predictions) is provided. Those skilled in the art will recognize that the systems and methods according to exemplary aspects of the present disclosure can be extended to any suitable machine learning task.

[0104] Figure 3 A block diagram depicts example data 300 according to an exemplary embodiment of the present disclosure. Data 300 may be floating-point data. For example, data 300 includes a sign component 302, an exponent component 304, and a mantissa component 306. For illustrative purposes, data 300 is a 16-bit floating-point number having one digit in the sign component 302, six digits in the exponent component 304, and nine digits in the mantissa component 306. Those skilled in the art will understand that data having any suitable number of digits in each of the sign component 302, exponent component 304, and mantissa component 306 can be used according to the present disclosure. For example, 32-bit floating-point numbers having one, eight, and twenty-three digits respectively in the sign, exponent, and mantissa components can be used according to the present disclosure. As another example, 64-bit "double-precision" floating-point numbers having one, eleven, and fifty-two digits respectively in the sign, exponent, and mantissa components can be used according to the present disclosure.

[0105] Data 300 may include a first part 310 and a second part 312. For example... Figure 3 As illustrated, portions 310 and 312 are divided to the left or right of line 311. However, those skilled in the art will understand that any suitable number of any of the sign component 302, exponent component 304, and / or mantissa component 306 can be included in the first portion 310 and / or the second portion 312. For example, line 311 can be located between any two numbers in data 300. As another example, the numbers in data 300 can be divided separately, such as without considering their position within data 300.

[0106] In some embodiments, the first portion 310 may include numbers that can be compressed across a data set (e.g., a dataset including data 300). For example, such as Figure 3 As illustrated, the entire sign component 302 and exponent component 304, along with the two digits of the mantissa component 306, are included in the first part 310. In some cases, these digits may exhibit statistical redundancy across multiple datasets (e.g., a dataset including data 300), thus allowing for efficient compression using conventional compression techniques.

[0107] In some embodiments, the second portion 312 may include numbers that cannot be compressed across data sets (e.g., a dataset including data 300). For example, as Figure 3 As illustrated, a portion of the mantissa component 306 is included in the second part 312. In some cases, these numbers may exhibit a lack of statistical redundancy across multiple datasets (e.g., a dataset including data 300), thus preventing them from being effectively compressed using conventional compression techniques.

[0108] Example Method

[0109] Figure 4 A flowchart depicts an example method for transforming data according to an exemplary embodiment of this disclosure. Although for illustrative and discussion purposes, Figure 4 The steps are described in a specific order, but the method disclosed herein is not limited to the specific order or arrangement described. Various steps of method 400 may be omitted, rearranged, combined, and / or modified without departing from the scope of this disclosure.

[0110] In section 402, the computing system can receive vector data. For example, the vector data may include data representing vectors. As another example, the vector data may be represented in a vector format. For example, in some embodiments, the vector data may represent a linearized multidimensional matrix having a single dimension. For example, the vector data may be a vector x having n dimensions.

[0111] In some embodiments, an initial Kashin frame representation and / or an initial truncation level may be initialized. For example, the initial Kashin frame representation may be initialized with one or more coefficients a. i , where i = 1, ..., N. In some embodiments, each of the coefficients can be initialized to zero. As another example, the initial cutoff level can be initialized to

[0112] In a 404 error, the computational system can determine a vector frame representation of the vector data. For example, it can be done in a frame with N dimensions. The vector x is transformed into a frame representation, where N > n. The frame representation can include one or more coefficients, such as N coefficients. For example, the frame representation can include coefficient b. i , making For example, the coefficient can be defined as b i = <x,u i In some embodiments, the frame may satisfy the uncertainty principle. For example, if ||∑ i∈Ω b i u i ||≤η(Σ i∈Ω |b i | 2 ) 1 / 2 Then the uncertainty principle can be satisfied, where Ω is every subset of {1,2,…,N}, and |Ω|≤δN.

[0113] In 406, the computing system can truncate the vector frame representation to obtain a truncated frame representation. For example, the vector frame representation can be at levels The vector is truncated, where j represents the iteration number (e.g., j = 0, ..., r). For example, a truncated frame representation may include one or more coefficients b of the truncated vector frame representation. i To obtain one or more cutoff coefficients For example, a truncated frame representation could be a Kashin representation of a vector x', where the level

[0114] In 408, the calculation system can adjust one or more coefficients of the Kashin frame representation based on the truncated frame representation. For example, in some embodiments, one or more truncated coefficients can be added to the Kashin frame coefficients. In some embodiments, one or more truncated coefficients can be factored (e.g., by...). ) Scaling and adding to the Kashin frame coefficients. The truncation level can be adjusted, additionally and / or alternatively. For example, the truncation level can be increased by a factor of η.

[0115] In 410, the computational system can determine the residual. For example, the residual could be the difference between the vectors represented by the frame before and after truncation (e.g., x - x').

[0116] In 412, the computational system can determine the residual frame representation of the residual. For example, it can be based on frames that satisfy the uncertainty principle. To calculate the frame representation of the residual.

[0117] Steps 406 to 412 can be repeated iteratively. For example, steps 406 to 412 can be repeated iteratively, where the residual frame representation is truncated from the second iteration onwards, instead of the vector frame representation, until the coefficients of the Kashin frame representation satisfy... Until then. For example, in some implementations, the result of the last iteration is a Kashin frame representation that satisfies... In some implementations, the residual frame representation is not truncated before adjusting the Kashin frame representation. This allows for accurate Kashin frame representations.

[0118] Figure 5 A flowchart is depicted illustrating an example method for storing compressed data according to an exemplary embodiment of the present disclosure. Although for purposes of illustration and discussion... Figure 5 The steps are described in a specific order, but the method disclosed herein is not limited to the specific order or arrangement described. Various steps of method 500 may be omitted, rearranged, combined, and / or modified without departing from the scope of this disclosure.

[0119] In step 502, the computing system can receive topic data. For example, topic data may include a dataset. Topic data and / or a dataset may include multiple data items. For example, each of the data items may have a defined value, such as a lexical and / or numerical value. Topic data can be stored in any suitable format. For example, in some embodiments, topic data may be floating-point data. In some embodiments, floating-point data may include a sign component, an exponent component, and / or a mantissa component.

[0120] In 504, the computational system can obtain an incompressible portion of the subject data. For example, an incompressible portion could be a part of the subject data that might not have been effectively compressed using lossless compression techniques. For example, an incompressible portion could exhibit limitations due to the absence of statistical redundancy, allowing its size to be significantly reduced without information loss. For example, an incompressible portion could include portions of multiple data items, such that the values ​​of these portions exhibit very little or no repetition across multiple data items, especially consecutive repetition.

[0121] In 506, the computational system can transform the incompressible portion to obtain transformed data. For example, the computational system can transform the incompressible portion to minimize its norm. For example, the computational system can transform the incompressible portion based on Kashin decomposition.

[0122] In the 508, the computing system can quantize transformed data to obtain quantized data. For example, transformed data can be quantized to reduce the precision associated with it. For example, quantized data can have fewer digits than transformed data. Any suitable quantization method can be used to quantize transformed data. For example, transformed data can be quantized by uniform quantization. As another example, transformed data can be quantized by truncating at least a portion of it. For example, less significant bits can be truncated from the transformed data. As another example, transformed data can be quantized by mapping it to one of a plurality of index values.

[0123] In 510, the computing system can store quantized data. For example, one or more computing devices can store quantized data in one or more memory devices. Example memory devices include, but are not limited to, non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and combinations thereof.

[0124] Additionally and / or alternatively, one or more computing devices may store a compressed portion of the subject data. For example, the compressed portion may be a compressible part of the subject data that can be compressed using any suitable compression technique, especially lossless compression. For example, the compressible portion of the compressed data may be a part of the compressed data that is separate from the incompressible portion. For example, the compressible portion may be data that is not in the incompressible portion but is in the compressible portion. For example, the compressed portion may be compressed using any suitable compression technique, such as, but not limited to, the Lempel-Zip-Markov chain algorithm (LZMA), the Burrows-Wheeler algorithm (e.g., bzip2), the DEFLATE algorithm (e.g., gzip), machine learning compression techniques, or any other suitable compression technique.

[0125] In some implementations, quantized data may be stored in association with a compressed portion of the subject data. For example, storing two forms of data in an associative manner may include storing both forms of data in the same data file. As another example, storing two forms of data in an associative manner may include storing the two forms of data such that they are logically related and / or reference each other or their storage locations. As yet another example, storing two forms of data in an associative manner may include storing the two forms of data within a data structure, wherein the data structure contains logical connections between the two forms of data (e.g., pointers to another pointer or lookup table within a database).

[0126] Figure 6 A flowchart depicts an example method for transmitting compressed data according to an example embodiment of the present disclosure. Although for illustrative and discussion purposes, Figure 6 The steps are described in a specific order, but the method disclosed herein is not limited to the specific order or arrangement described. Various steps of method 600 may be omitted, rearranged, combined, and / or adjusted without departing from the scope of this disclosure.

[0127] In step 602, the computing system can receive topic data. For example, the topic data may include a dataset. The topic data and / or dataset may include multiple data items. For example, each of the data items may have a defined value, such as a lexical term and / or a numerical value. The topic data can be stored in any suitable format. For example, in some embodiments, the topic data may be floating-point data. In some embodiments, floating-point data may include a sign component, an exponent component, and / or a mantissa component.

[0128] In 604, the computing device can obtain a first portion and a second portion of the subject data. For example, in some embodiments, the second portion may be an incompressible portion of the subject data. For example, the second portion may be a portion of the subject data that may not be effectively compressed according to lossless compression techniques. For example, the second portion may exhibit limitations due to the absence of statistical redundancy, such that the size of the second portion can be significantly reduced without information loss. For example, the second portion may include portions of multiple data items, such that the values ​​of these portions exhibit very little or no repetition among the multiple data items, especially continuous repetition.

[0129] Additionally and / or alternatively, in some embodiments, the first portion may be a compressible portion of the subject data. The first portion can be compressed efficiently using lossless compression techniques. For example, the first portion can be compressed efficiently without information loss. For example, the size of the first portion can be significantly reduced without information loss. For example, the first portion may exhibit statistical redundancy, such that at least a portion of the first portion can be predicted or otherwise recreated based on the statistical redundancy.

[0130] In 606, the computational system can transform the second part to obtain transformed data. For example, the computational system can transform the second part to minimize its norm. For example, the computational system can transform the second part according to the Kashin decomposition.

[0131] In 608, the computing system can quantize transformed data to obtain quantized data. For example, transformed data can be quantized to reduce the precision associated with it. For example, quantized data can have fewer digits than transformed data. Any suitable quantization method can be used to quantize transformed data. For example, transformed data can be quantized by uniform quantization. As another example, transformed data can be quantized by truncating at least a portion of the transformed data. For example, less significant bits can be truncated from the transformed data. As another example, transformed data can be quantized by mapping it to one of a plurality of index values.

[0132] In 610, the computing system can transmit quantized data. For example, the computing system can transmit quantized data to a second computing system. The quantized data can be transmitted by any suitable transmission method. For example, it can be transmitted via wired transmission (e.g., Ethernet) and / or via wireless transmission (e.g., Bluetooth, ZigBee, WLAN, IEEE 802.11) or via any other suitable transmission method or technology.

[0133] Additionally and / or alternatively, one or more computing devices may transmit a compressed portion of the subject data. For example, the compressed portion may be a first part of the subject data compressed using any suitable compression technique, particularly lossless compression. For example, the compressed portion may be compressed using any suitable compression technique, such as, but not limited to, the Lempel-Zip-Markov chain algorithm (LZMA), the Burrows-Wheeler algorithm (e.g., bzip2), the DEFLATE algorithm (e.g., gzip), machine learning compression techniques, or any other suitable compression technique. For example, the computing device may compress the first part to obtain compressed data. Additionally and / or alternatively, the computing device may transmit the compressed data (e.g., to a second computing device). Additionally and / or alternatively, the compressed data may be decompressed (e.g., via the second computing device) to obtain the first part.

[0134] Additional Publication

[0135] This document discusses technical reference servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functionalities between and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can run sequentially or in parallel.

[0136] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of illustration and not limitation. Modifications, variations, and equivalents to such embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not preclude such modifications, variations, and / or additions to the subject matter as will be readily apparent to those skilled in the art. For example, features illustrated or described as part of an embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for compressing data, the computer-implemented method comprising: Obtain floating-point data, the floating-point data including multiple floating-point numbers, each of the multiple floating-point numbers including one or more sign bits, multiple exponent bits and multiple mantissa bits; wherein, the first part of each floating-point number includes one or more most significant mantissa bits among the one or more sign bits, the multiple exponent bits and the multiple mantissa bits; and the second part of each floating-point number includes one or more least significant mantissa bits among the multiple mantissa bits; Perform a first lossless compression technique to compress the first portion of each floating-point number of the floating-point data and obtain first lossless compressed data; The second part of each floating-point number of the floating-point data is transformed to obtain transformed data; The transformed data is quantized to obtain quantized data; and Store the first lossless compressed data and the quantized data; and / or The first lossless compressed data and the quantized data are transmitted to another computing device, wherein the other computing device is configured to decompress the compressed data and dequantize the quantized data to obtain a reconstruction of the first portion and the second portion.

2. The computer-implemented method according to claim 1, wherein, The floating-point data encodes the parameter values ​​of the machine learning model.

3. The computer-implemented method according to claim 1, wherein, Transforming the second part of the data includes performing Kashin decomposition on the second part of the data.

4. The computer-implemented method according to claim 1, wherein, Quantizing the transformed data includes applying uniform quantization to the transformed data.

5. The computer-implemented method according to claim 1, wherein, Quantizing the transformed data includes truncating at least a portion of the transformed data.

6. The computer-implemented method according to claim 1, further comprising: The transformation data is reconstructed based on the quantized data; as well as The second part is reconstructed based on the transformed data.

7. The computer-implemented method according to claim 6, wherein, Reconstructing the transformed data includes jittering the quantized data.

8. The computer-implemented method according to claim 6, further comprising: The transformed data is reconstructed by the other computing device based on the quantized data; as well as The second part is reconstructed by the other computing device based on the transformed data.

9. The computer-implemented method according to claim 1, further comprising: The compressed data is decompressed by the other computing device to obtain the first portion.

10. A computing system configured to compress data, the computing system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform the operation of a computer-implemented method according to any one of claims 1 to 9.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations of a computer-implemented method according to any one of claims 1 to 9.

12. A computer-implemented method for reconstructing data, the computer-implemented method comprising: Obtain lossless compressed data, wherein the lossless compressed data includes a first part of floating-point data; Obtain quantized data, the quantized data including a second portion of the floating-point data, wherein the second portion of the floating-point data remains uncompressed; The first portion of the floating-point data is reconstructed from the lossless compressed data to reconstruct one or more sign bits, multiple exponent bits, and multiple mantissa bits of the floating-point data, including one or more most significant mantissa bits. as well as The second portion of the floating-point data is reconstructed from the quantized data to reconstruct one or more least significant mantissa bits among the plurality of mantissa bits.

13. The computer-implemented method according to claim 12, wherein, The floating-point data encodes the parameter values ​​of the machine learning model.

14. The computer-implemented method of claim 12, wherein reconstructing the first portion of the floating-point data from the lossless compressed data comprises decompressing the lossless compressed data to obtain the first portion of the floating-point data.

15. The computer-implemented method of claim 12, wherein the second portion of reconstructing the floating-point data from the quantized data comprises reconstructing transformed data from the quantized data.

16. The computer-implemented method of claim 15, further comprising reconstructing the second portion of the floating-point data from the transformed data.

17. The computer-implemented method according to claim 16, wherein, The second part of reconstructing the floating-point data from the transformed data includes applying an inverse transform to the transformed data.

18. The computer-implemented method according to claim 15, wherein, Reconstructing the transformed data from the quantized data includes estimating the transformed data based on the quantized data.

19. The computer-implemented method according to claim 15, wherein, Reconstructing the transformed data from the quantized data includes increasing the precision of the quantized data, wherein the quantized data includes one or more indices that map to corresponding values ​​in the transformed data.

20. The computer-implemented method of claim 19, wherein increasing the precision of the quantized data comprises appending one or more numbers to the quantized data.

21. A computing system configured to reconstruct data, the computing system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform the operations of a computer-implemented method according to any one of claims 12 to 20.

22. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations of the computer-implemented method according to any one of claims 12 to 20.

Images