A data compression method, a data decompression method, and related equipment.

By updating the parent model during compression and combining optimization algorithms and early stopping mechanisms, the problem of low compression ratio in lossless compression of deep generative models is solved, achieving efficient data compression and decompression, and is suitable for lossless compression of data such as images, videos, and text.

CN115409150BActive Publication Date: 2026-11-10HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110584986.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-27
Publication Date
2026-11-10
Estimated Expiration
2041-05-27

AI Technical Summary

Technical Problem

How to use deep generative models in practical applications to improve data compression ratio, especially to improve compression efficiency and storage space utilization in lossless compression.

Method used

By acquiring the parent model and updating and compressing data batches simultaneously during the compression process, the universality and adaptability of the parent model are utilized to avoid saving the new model after each update. Combined with optimization algorithms and early stopping mechanisms, the compression ratio is improved and storage space is saved.

Benefits of technology

It achieves improved compression ratio and decompression time efficiency during lossless compression, is suitable for large-scale datasets, and saves storage space and computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115409150B_ABST
    Figure CN115409150B_ABST
Patent Text Reader

Abstract

This application discloses a data compression method, a data decompression method, and related equipment, which can be applied to lossless compression of data such as images, videos, and text. The method can be executed by a compression device or by a component of the compression device (e.g., a processor, chip, or chip system). The method includes: acquiring a first data batch and a second data batch; during the compression of the first data batch, updating the parent model based on the first data batch to obtain a first model, and using the first model to participate in the compression of the second data batch. On the one hand, during the compression process, the parent model can be updated while compressing the data batch to be compressed, without needing to save the updated model, thus making it suitable for compressing large-scale datasets. On the other hand, during the parent model update process, the parent model will increasingly conform to the distribution of the data batch to be compressed, therefore, the compression ratio will become increasingly higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a data compression method, a data decompression method, and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Currently, lossless compression is an important technical direction in modern AI. Lossless compression can be mainly achieved by deep generative models. Deep generative models can model the distribution of data very accurately, thus theoretically achieving a higher compression rate than traditional algorithms.

[0004] However, how to use deep generative models to improve data compression rates in practical applications is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a data compression method, a data decompression method, and related equipment. These methods can improve the compression ratio of batches of data to be compressed.

[0006] The first aspect of this application provides a data compression method applicable to lossless compression of data such as images, videos, and text. This method can be executed by a compression device or by a component of the compression device (e.g., a processor, chip, or chip system). The method includes: acquiring a first data batch and a second data batch, for example, acquiring a first data batch and a second data batch from a dataset to be compressed; acquiring a parent model; compressing the first data batch based on the first data batch, the parent model, and a first encoder to obtain a first sequence, wherein the first encoder corresponds to the parent model; updating the parent model based on the first data batch to obtain a first model; and compressing the second data batch based on the second data batch, the first model, the first sequence, and the first encoder to obtain a second sequence. The parent model is used to calculate first probability distribution information, which represents the probability distribution of the values ​​of each variable in the first data batch. The variable value can be understood as the smallest compression unit (or the numerical value corresponding to the smallest compression unit) in the data batch. For image compression, the variable value can be one pixel value or multiple pixel values.

[0007] In this embodiment, on the one hand, during the compression process, the parent model can be updated while compressing the batch of data to be compressed, without needing to save the updated model, thus making it suitable for compressing large-scale datasets. On the other hand, during the parent model update process, the parent model will increasingly conform to the distribution of the batch of data to be compressed, therefore, the compression ratio will become higher and higher. Furthermore, compared to existing technologies that require training a new model specifically for the dataset to be compressed, the parent model in this embodiment has strong versatility and does not require specific training for the dataset to be compressed; the model is continuously updated during the compression process. Moreover, this update process can be reproduced during decompression by updating the parent model; therefore, by saving the parent model, the updated model can be obtained, without needing to store every model generated during the compression process, saving storage space.

[0008] Optionally, in one possible implementation of the first aspect, the above steps further include: obtaining a third data batch; compressing the second data batch based on the second data batch, the first model, the first sequence, and the first encoder to obtain a second sequence, including: compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; updating the first model based on the third data batch to obtain a second model; and compressing the second data batch based on the second data batch, the second model, the third sequence, and the first encoder to obtain a second sequence.

[0009] In this possible implementation, the parent model can be updated multiple times, and the updated model can be used to compress subsequent batches of data, thereby improving the compression ratio.

[0010] Optionally, in one possible implementation of the first aspect, the above steps further include: obtaining a third data batch; compressing the second data batch based on the second data batch, the first model, the first sequence, and the first encoder to obtain a second sequence, including: compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; and compressing the second data batch based on the second data batch, the first model, the third sequence, and the first encoder to obtain a second sequence. Additionally, if the model is no longer being updated, i.e., early stopping, a prompt message can be saved. This prompt message is used to indicate early stopping information and avoid using an incorrect model during decompression.

[0011] In this possible implementation, early stopping can be used to terminate model updates prematurely, and the model obtained at the time of early stopping can be used to compress subsequent data batches. Since early stopping halts further model updates, compression after early stopping only requires forward propagation of the neural network, thus saving the time required for backward propagation and the space for model updates (saving space to allow for larger data batches). In other words, this is an improvement scheme that sacrifices compression ratio to enhance compression and decompression time efficiency compared to continuously updating the model.

[0012] Optionally, in one possible implementation of the first aspect, the above steps, which update the parent model based on the first batch of data to obtain the first model, include: using an optimization algorithm and the first batch of data to update the parent model to obtain the first model. The optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning. Optionally, to achieve the reproducibility of the parent model update during decompression, a random seed that appears during the parent model update process can be saved and stored together with the parent model.

[0013] In this possible implementation, the parent model can be updated based on the optimization algorithm, which is easy to reproduce during the decompression process. Since it does not require special training on the dataset to be compressed as in existing technologies, the compression time is reduced compared to the time spent on special training on the dataset to be compressed.

[0014] Optionally, in one possible implementation of the first aspect, the parent model in the above steps is obtained by training the neural network model with the training data batch in the training dataset as input and the loss function value being less than a first threshold. The data type of the training dataset is the same as the data type of the dataset to be compressed. The dataset to be compressed includes a first data batch and a second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

[0015] In this possible implementation, a loss function is used to train the neural network model to obtain a trained master model, and the data type of the training dataset is the same as the data type of the dataset to be compressed, thereby ensuring that the trained master model conforms to the distribution of the dataset to be compressed and improving the compression ratio.

[0016] Optionally, in one possible implementation of the first aspect, the above steps, which compress the first data batch to obtain a first sequence based on the first data batch, the parent model, and the first encoder, include: inputting the first data batch into the parent model to obtain first probability distribution information; and compressing the first data batch to obtain a first sequence based on the first probability distribution information and the first encoder.

[0017] In this possible implementation, the first probability distribution information of the first data batch is obtained through the master model, and this probability distribution information is used to compress the first data batch with the first encoder to obtain the first sequence. During the compression of the first data batch, it is not necessary to pre-train the first data batch specifically; therefore, the compression time is reduced compared to existing compression times by the time spent specifically training the first data batch.

[0018] Optionally, in one possible implementation of the first aspect, the above steps, which compress the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder, include: inputting the second data batch into the first model to obtain second probability distribution information, wherein the second probability distribution information is used to represent the probability distribution of the values ​​of each variable in the second data batch; and compressing the second data batch to obtain the second sequence based on the second probability distribution information, the first sequence, and the first encoder.

[0019] In this possible implementation, subsequent data batches (i.e., the second data batch) use the first model obtained by updating the parent model. The first model is more consistent with the probability distribution of the second data batch, thereby improving the compression ratio.

[0020] Optionally, in one possible implementation of the first aspect, the above steps, which compress the second data batch to obtain the second sequence based on the second data batch, the second model, the third sequence, and the first encoder, include: inputting the second data batch into the second model to obtain second probability distribution information, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; and compressing the second data batch to obtain the second sequence based on the second probability distribution information, the third sequence, and the first encoder.

[0021] In this possible implementation, subsequent data batches (i.e., the second data batch) use the second model obtained by updating the first model. The second model is more consistent with the probability distribution of the second data batch, thereby improving the compression rate.

[0022] Optionally, in one possible implementation of the first aspect, the above steps, which compress the second data batch to obtain the second sequence based on the second data batch, the first model, the third sequence, and the first encoder, include: inputting the second data batch into the first model to obtain second probability distribution information, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; and compressing the second data batch to obtain the second sequence based on the second probability distribution information, the third sequence, and the first encoder.

[0023] In this possible implementation, early stopping can be used to terminate model updates prematurely, and the model obtained at the time of early stopping can be used to compress subsequent data batches. Since early stopping halts further model updates, compression after early stopping only requires forward propagation of the neural network, thus saving the time required for backward propagation and the space for model updates (saving space to allow for larger data batches). In other words, this is an improvement scheme that sacrifices compression ratio to enhance compression and decompression time efficiency compared to continuously updating the model.

[0024] Optionally, in one possible implementation of the first aspect, the above steps, based on the third data batch, the first model, the first sequence, and the first encoder, compress the third data batch to obtain the third sequence, including: inputting the third data batch into the first model to obtain third probability distribution information, the third probability distribution information being used to represent the probability distribution of the values ​​of each variable in the third data batch; and compressing the third data batch based on the third probability distribution information, the first sequence, and the first encoder to obtain the third sequence.

[0025] In this possible implementation, the subsequent data batch (i.e., the third data batch) uses the first model, which is more consistent with the probability distribution of the third data batch, thereby improving the compression ratio.

[0026] Optionally, in one possible implementation of the first aspect, the above steps further include: determining the first encoder corresponding to the type of the parent model based on the first association relationship, wherein the first association relationship is used to represent the association relationship between the type of the parent model and the first encoder, and the type includes a fully observed model and a latent variable model.

[0027] In this possible implementation, selecting a suitable encoder based on the type of the parent model can improve the subsequent compression rate.

[0028] Optionally, in one possible implementation of the first aspect, the above steps of obtaining the parent model include: obtaining the parent model based on the data type of the dataset to be compressed, where the data type includes image data type and sequence data type.

[0029] In this possible implementation, deep generative models perform differently for different data types. For example, streaming models or variational autoencoder models are more effective for image data. Choosing a suitable parent model based on the data type can help improve the compression ratio.

[0030] Optionally, in one possible implementation of the first aspect, the steps of obtaining the first data batch and the second data batch include: obtaining the dataset to be compressed; splitting the dataset to be compressed to obtain the first data batch and the second data batch.

[0031] This possible implementation is suitable for compressing large-scale datasets by splitting the dataset into batches.

[0032] A second aspect of this application provides a data compression method applicable to lossless compression of data such as images, videos, and text. This method can be executed by a compression device or by a component of the compression device (e.g., a processor, chip, or chip system). The method includes: acquiring a first data batch and a second data batch; acquiring a parent model, which is used to calculate first probability distribution information, representing the probability distribution of each variable's value in the first data batch; updating the parent model based on the first data batch to obtain a first model; compressing the second data batch based on the second data batch, the first model, and a first encoder to obtain a first sequence, wherein the first encoder corresponds to the parent model; and compressing the first data batch based on the first data batch, the parent model, the first sequence, and the first encoder to obtain the second sequence.

[0033] In this embodiment, a Last-In-First-Out (LIFO) encoder can be applied. On one hand, during compression, the parent model can be updated while compressing the batch of data to be compressed, without needing to save the updated model, thus making it suitable for compressing large-scale datasets. On the other hand, during the parent model update process, the parent model increasingly conforms to the distribution of the batch of data to be compressed, resulting in a higher compression ratio. Furthermore, by reversing the order of the LIFO encoder, the decompression process can obtain the data batches sequentially.

[0034] Optionally, in one possible implementation of the second aspect, the above steps further include: obtaining a third data batch; compressing the second data batch based on the second data batch, the first model, and the first encoder to obtain a first sequence, including: updating the first model based on the third data batch to obtain a second model; compressing the second data batch based on the second data batch, the second model, and the first encoder to obtain a first sequence; compressing the first data batch based on the first data batch, the parent model, the first sequence, and the first encoder to obtain a second sequence, including: compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; compressing the first data batch based on the first data batch, the parent model, the third sequence, and the first encoder to obtain a second sequence.

[0035] In this possible implementation, the parent model can be updated multiple times, and the updated model can be used to compress subsequent batches of data, thereby improving the compression ratio.

[0036] Optionally, in one possible implementation of the second aspect, the above steps further include: obtaining a third data batch; compressing the first data batch to obtain a second sequence based on the first data batch, the parent model, the first sequence, and the first encoder, including: compressing the third data batch to obtain a third sequence based on the third data batch, the first model, the first sequence, and the first encoder; and compressing the first data batch to obtain a second sequence based on the first data batch, the parent model, the third sequence, and the first encoder. Additionally, if the model is no longer being updated, i.e., early stopping, a prompt message can be saved. This prompt message is used to indicate early stopping and avoid using an incorrect model during decompression.

[0037] In this possible implementation, early stopping can be used to terminate model updates prematurely, and the model obtained at the time of early stopping can be used to compress subsequent data batches. Since early stopping halts further model updates, compression after early stopping only requires forward propagation of the neural network, thus saving the time required for backward propagation and the space for model updates (saving space to allow for larger data batches). In other words, this is an improvement scheme that sacrifices compression ratio to enhance compression and decompression time efficiency compared to continuously updating the model.

[0038] A third aspect of this application provides a data decompression method applicable to lossless compression of data such as images, videos, and text. This method can be executed by a decompression device or by a component of the decompression device (e.g., a processor, chip, or chip system). The method includes: obtaining a second sequence; obtaining a parent model; decompressing the second sequence based on the parent model and a first encoder to obtain a fourth sequence and a first data batch, wherein the first encoder corresponds to the parent model; updating the parent model based on the first data batch to obtain a first model; and decompressing the fourth sequence based on the first model and the first encoder to obtain a second data batch.

[0039] In this embodiment, on the one hand, during decompression, the parent model can be updated while the encoded sequence is being decompressed, without needing to save the updated model, thus making it suitable for compressing large-scale datasets. On the other hand, the parent model in this embodiment has strong versatility and does not require specific training on the dataset to be compressed; the model is continuously updated during the compression process. Furthermore, this update process can be reproduced during decompression by updating the parent model. Therefore, by saving the parent model, the updated model can be obtained, eliminating the need to store every model generated during compression, thus saving storage space.

[0040] Optionally, in one possible implementation of the third aspect, the above steps, based on the first model and the first encoder, decompress the fourth sequence to obtain the second data batch, include: decompressing the fourth sequence based on the first model and the first encoder to obtain the fifth sequence and the third data batch; updating the first model based on the third data batch to obtain the second model; and decompressing the fifth sequence based on the second model and the first encoder to obtain the second data batch.

[0041] In this possible implementation, the parent model can be updated multiple times, and the updated model is used to decompress the encoded sequence to obtain a data batch, thus achieving lossless compression.

[0042] Optionally, in one possible implementation of the third aspect, the above steps, based on the first model and the first encoder, decompress the fourth sequence to obtain the second data batch, including: decompressing the fourth sequence based on the first model and the first encoder to obtain the fifth sequence and the third data batch; and decompressing the fifth sequence based on the first model and the first encoder to obtain the second data batch. Optionally, prompting information can be obtained to indicate early stop information and avoid using an incorrect model during decompression.

[0043] Among these possible implementations, an improved approach that sacrifices compression ratio to enhance decompression time efficiency is an alternative to continuously updating the model.

[0044] Optionally, in one possible implementation of the third aspect, the above steps, which update the parent model based on the first batch of data to obtain the first model, include: using an optimization algorithm and the first batch of data to update the parent model to obtain the first model. The optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning. Optionally, to achieve the reproducibility of the parent model update during decompression, a random seed that appears during the parent model update process can be saved and stored together with the parent model.

[0045] In this possible implementation, the parent model can be updated based on an optimization algorithm, which is easy to reproduce during the decompression process.

[0046] Optionally, in one possible implementation of the third aspect, the parent model in the above steps is obtained by training the neural network model with the training data batch in the training dataset as input and the loss function value being less than a first threshold. The data type of the training dataset is the same as the data type of the dataset to be compressed. The dataset to be compressed includes a first data batch and a second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

[0047] In this possible implementation, a loss function is used to train the neural network model to obtain a trained master model, and the data type of the training dataset is the same as the data type of the dataset to be compressed, thereby ensuring that the trained master model conforms to the distribution of the dataset to be compressed and improving the compression ratio.

[0048] Optionally, in one possible implementation of the third aspect, the above steps, based on the parent model and the first encoder, decompress the second sequence to obtain the fourth sequence and the first data batch, including: decompressing the second sequence based on the second sequence, the parent model, and the first encoder to obtain the fourth sequence and the first data batch; and decompressing the fourth sequence based on the first model and the first encoder to obtain the second data batch, including: decompressing the fourth sequence based on the fourth sequence, the first model, and the first encoder to obtain the second data batch.

[0049] This possible implementation method can be applied to cases where the model contains conditional probabilities.

[0050] Optionally, in one possible implementation of the third aspect, the above steps, based on the second sequence, the parent model, and the first encoder, decompress the second sequence to obtain the fourth sequence and the first data batch, include: obtaining a first probability based on the parent model; decompressing the second sequence based on the first probability and the first encoder to obtain a portion of data; inputting the portion of data into the parent model to obtain a second probability; decompressing the second sequence based on the second probability and the first encoder to obtain another portion of data and the fourth sequence; and obtaining the first data batch based on the portion of data and the other portion of data.

[0051] In this possible implementation, a portion of the probability can be obtained from the model first, then the encoded sequence can be decompressed using that portion of the probability to obtain a portion of the data, and then the data can be input into the model to obtain another portion of the probability. This other portion of the probability can be used to decompress the encoded sequence to obtain another portion of the data, thus obtaining a complete batch of data.

[0052] Optionally, in one possible implementation of the third aspect, the above steps, based on the parent model and the first encoder, decompress the second sequence to obtain the fourth sequence and the first data batch, include: obtaining first probability distribution information based on the parent model, the first probability distribution information being used to represent the probability distribution of the values ​​of each variable in the first data batch; and decompressing the second sequence based on the first probability distribution information and the first encoder to obtain the fourth sequence and the first data batch.

[0053] In this possible implementation, the first probability distribution information can be directly obtained through the parent model, and the sequence can be decompressed using the first probability distribution information to obtain the first data batch.

[0054] Optionally, in one possible implementation of the third aspect, the above steps, based on the first model and the first encoder, decompress the fourth sequence to obtain the second data batch, including: obtaining second probability distribution information based on the first model, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; and decompressing the fourth sequence based on the second probability distribution information and the first encoder to obtain the second data batch.

[0055] In this possible implementation, the second probability distribution information can be directly obtained through the first model, and the sequence can be decompressed using the first probability distribution information to obtain the second data batch.

[0056] Optionally, in one possible implementation of the third aspect, the above steps further include: determining the first encoder corresponding to the type of the parent model based on the first association relationship, wherein the first association relationship is used to represent the association relationship between the type of the parent model and the first encoder, and the types include fully observed models and latent variable models.

[0057] In this possible implementation, selecting a suitable encoder based on the type of the parent model can improve the subsequent compression rate.

[0058] Optionally, in one possible implementation of the third aspect, the above steps further include: merging the first data batch and the second data batch to obtain an initial dataset.

[0059] This possible implementation is suitable for decompressing large-scale datasets and obtaining the dataset by merging multiple data batches.

[0060] A fourth aspect of this application provides a compression device that can be applied to lossless compression of data such as images, videos, and text. The compression device includes:

[0061] The acquisition unit is used to acquire the first data batch and the second data batch;

[0062] The acquisition unit is also used to acquire the parent model, which is used to calculate the first probability distribution information, which is used to represent the probability distribution of the values ​​of each variable in the first data batch.

[0063] A compression unit is used to compress the first data batch to obtain a first sequence based on the first data batch, the master model, and the first encoder, wherein the first encoder corresponds to the master model;

[0064] The update unit is used to update the parent model based on the first data batch to obtain the first model;

[0065] The compression unit is also used to compress the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder.

[0066] Optionally, in one possible implementation of the fourth aspect, the above-mentioned acquisition unit is further configured to acquire a third data batch; the compression unit is further configured to compress the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; the update unit is further configured to update the first model based on the third data batch to obtain a second model; and the compression unit is further configured to compress the second data batch based on the second data batch, the second model, the third sequence, and the first encoder to obtain a second sequence.

[0067] Optionally, in one possible implementation of the fourth aspect, the above-mentioned acquisition unit is further configured to acquire a third data batch; the compression unit is further configured to compress the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; the compression unit is further configured to compress the second data batch based on the second data batch, the first model, the third sequence, and the first encoder to obtain a second sequence.

[0068] Optionally, in one possible implementation of the fourth aspect, the aforementioned update unit is specifically used to update the parent model using an optimization algorithm and a first batch of data to obtain a first model. The optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning.

[0069] Optionally, in one possible implementation of the fourth aspect, the aforementioned parent model is obtained by training the neural network model with the training data batch in the training dataset as input and aiming to make the loss function value less than a first threshold. The data type of the training dataset is the same as the data type of the dataset to be compressed. The dataset to be compressed includes a first data batch and a second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

[0070] Optionally, in one possible implementation of the fourth aspect, the compression unit is specifically used to input the first data batch into the parent model to obtain the first probability distribution information; the compression unit is specifically used to compress the first data batch based on the first probability distribution information and the first encoder to obtain the first sequence.

[0071] Optionally, in one possible implementation of the fourth aspect, the compression unit is specifically used to input the second data batch into the first model to obtain the second probability distribution information, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; the compression unit is specifically used to compress the second data batch based on the second probability distribution information, the first sequence, and the first encoder to obtain the second sequence.

[0072] Optionally, in one possible implementation of the fourth aspect, the compression unit is specifically used to input the second data batch into the second model to obtain the second probability distribution information, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; the compression unit is specifically used to compress the second data batch based on the second probability distribution information, the third sequence, and the first encoder to obtain the second sequence.

[0073] Optionally, in one possible implementation of the fourth aspect, the compression unit is specifically used to input the second data batch into the first model to obtain the second probability distribution information, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; the compression unit is specifically used to compress the second data batch based on the second probability distribution information, the third sequence, and the first encoder to obtain the second sequence.

[0074] Optionally, in one possible implementation of the fourth aspect, the compression unit is specifically used to input the third data batch into the first model to obtain the third probability distribution information, the third probability distribution information being used to represent the probability distribution of the values ​​of each variable in the third data batch; the compression unit is specifically used to compress the third data batch based on the third probability distribution information, the first sequence, and the first encoder to obtain the third sequence.

[0075] Optionally, in one possible implementation of the fourth aspect, the compression device further includes: a determining unit, used to determine the first encoder corresponding to the type of the parent model based on a first association relationship, wherein the first association relationship is used to represent the association relationship between the type of the parent model and the first encoder, and the type includes a fully observed model and a latent variable model.

[0076] Optionally, in one possible implementation of the fourth aspect, the aforementioned acquisition unit is specifically used to acquire a parent model based on the data type of the dataset to be compressed, including image data types and sequence data types.

[0077] Optionally, in one possible implementation of the fourth aspect, the aforementioned acquisition unit is specifically used to acquire the dataset to be compressed; the acquisition unit is specifically used to split the dataset to be compressed to obtain a first data batch and a second data batch.

[0078] A fifth aspect of this application provides a compression device that can be applied to lossless compression of data such as images, videos, and text. The compression device includes:

[0079] The acquisition unit is used to acquire the first data batch and the second data batch;

[0080] The acquisition unit is also used to acquire the parent model, which is used to calculate the first probability distribution information, which is used to represent the probability distribution of the values ​​of each variable in the first data batch.

[0081] The update unit is used to update the parent model based on the first data batch to obtain the first model;

[0082] A compression unit is used to compress the second data batch based on the second data batch, the first model, and the first encoder to obtain a first sequence. The first encoder corresponds to the parent model.

[0083] The compression unit is also used to compress the first data batch to obtain the second sequence based on the first data batch, the master model, the first sequence, and the first encoder.

[0084] Optionally, in one possible implementation of the fifth aspect, the aforementioned acquisition unit is further configured to acquire a third data batch; the update unit is further configured to update the first model based on the third data batch to obtain a second model; the compression unit is specifically configured to compress the second data batch based on the second data batch, the second model, and the first encoder to obtain a first sequence; the compression unit is specifically configured to compress the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; and the compression unit is specifically configured to compress the first data batch based on the first data batch, the parent model, the third sequence, and the first encoder to obtain a second sequence.

[0085] Optionally, in one possible implementation of the fifth aspect, the above-mentioned acquisition unit is further used to acquire a third data batch; the compression unit is specifically used to compress the third data batch based on the third data batch, the first model, the first sequence, and the first encoder to obtain a third sequence; the compression unit is specifically used to compress the first data batch based on the first data batch, the parent model, the third sequence, and the first encoder to obtain a second sequence.

[0086] A sixth aspect of this application provides a decompression device that can be applied to lossless compression of data such as images, videos, and text. The decompression device includes:

[0087] The acquisition unit is used to acquire the second sequence;

[0088] The acquisition unit is used to acquire the parent model;

[0089] The decompression unit is used to decompress the second sequence based on the parent model and the first encoder to obtain the fourth sequence and the first data batch. The first encoder corresponds to the parent model.

[0090] The update unit is used to update the parent model based on the first data batch to obtain the first model;

[0091] The decompression unit is also used to decompress the fourth sequence based on the first model and the first encoder to obtain the second data batch.

[0092] Optionally, in one possible implementation of the sixth aspect, the decompression unit is specifically used to decompress the fourth sequence based on the first model and the first encoder to obtain the fifth sequence and the third data batch; the update unit is further used to update the first model based on the third data batch to obtain the second model; and the decompression unit is specifically used to decompress the fifth sequence based on the second model and the first encoder to obtain the second data batch.

[0093] Optionally, in one possible implementation of the sixth aspect, the decompression unit is specifically used to decompress the fourth sequence based on the first model and the first encoder to obtain the fifth sequence and the third data batch; the decompression unit is specifically used to decompress the fifth sequence based on the first model and the first encoder to obtain the second data batch.

[0094] Optionally, in one possible implementation of the sixth aspect, the aforementioned update unit is specifically used to optimize the parent model using an optimization algorithm and a first batch of data to obtain a first model, wherein the optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning.

[0095] Optionally, in one possible implementation of the sixth aspect, the aforementioned parent model is obtained by training the neural network model with the training data batch in the training dataset as input and aiming to make the loss function value less than a first threshold. The data type of the training dataset is the same as the data type of the dataset to be compressed. The dataset to be compressed includes a first data batch and a second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

[0096] Optionally, in one possible implementation of the sixth aspect, the decompression unit is specifically used to decompress the second sequence based on the second sequence, the master model, and the first encoder to obtain the fourth sequence and the first data batch; the decompression unit is specifically used to decompress the fourth sequence based on the fourth sequence, the first model, and the first encoder to obtain the second data batch.

[0097] Optionally, in one possible implementation of the sixth aspect, the aforementioned decompression unit is specifically used to obtain a first probability based on the parent model; the decompression unit is specifically used to decompress the second sequence based on the first probability and the first encoder to obtain a portion of data; the decompression unit is specifically used to input the portion of data into the parent model to obtain a second probability; the decompression unit is specifically used to decompress the second sequence based on the second probability and the first encoder to obtain another portion of data and a fourth sequence; and the decompression unit is specifically used to obtain a first data batch based on the portion of data and the other portion of data.

[0098] Optionally, in one possible implementation of the sixth aspect, the decompression unit is specifically used to obtain first probability distribution information based on the parent model, the first probability distribution information being used to represent the probability distribution of the values ​​of each variable in the first data batch; the decompression unit is specifically used to decompress the second sequence based on the first probability distribution information and the first encoder to obtain the fourth sequence and the first data batch.

[0099] Optionally, in one possible implementation of the sixth aspect, the decompression unit is specifically used to obtain second probability distribution information based on the first model, the second probability distribution information being used to represent the probability distribution of the values ​​of each variable in the second data batch; the decompression unit is specifically used to decompress the fourth sequence based on the second probability distribution information and the first encoder to obtain the second data batch.

[0100] Optionally, in one possible implementation of the sixth aspect, the decompression device further includes: a determining unit, used to determine the first encoder corresponding to the type of the parent model based on a first association relationship, wherein the first association relationship is used to represent the association relationship between the type of the parent model and the first encoder, and the types include fully observed models and latent variable models.

[0101] Optionally, in one possible implementation of the sixth aspect, the decompression device further includes: a merging unit for merging the first data batch and the second data batch to obtain an initial dataset.

[0102] A seventh aspect of this application provides a compression device, which or a component of the compression device (e.g., a processor, chip, or chip system) performs the method in the first aspect or any possible implementation thereof.

[0103] The eighth aspect of this application provides a compression device, which or a component of the compression device (e.g., a processor, chip, or chip system) performs the methods in the second aspect or any possible implementation thereof.

[0104] The ninth aspect of this application provides a decompression device, which or a component of the decompression device (e.g., a processor, chip, or chip system) performs the method in the third aspect or any possible implementation thereof.

[0105] A tenth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods of the first aspect or any possible implementation thereof, the second aspect or any possible implementation thereof, or the third aspect or any possible implementation thereof.

[0106] The eleventh aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the methods in the first aspect or any possible implementation of the first aspect, the second aspect or any possible implementation of the second aspect, or the third aspect or any possible implementation of the third aspect.

[0107] A twelfth aspect of this application provides a compression device, including: a processor coupled to a memory for storing programs or instructions, wherein when the programs or instructions are executed by the processor, the compression device implements the method of the first aspect or any possible implementation thereof.

[0108] The thirteenth aspect of this application provides a compression device, including: a processor coupled to a memory for storing programs or instructions, wherein when the programs or instructions are executed by the processor, the compression device implements the method in the second aspect or any possible implementation thereof.

[0109] The fourteenth aspect of this application provides a decompression device, including: a processor coupled to a memory for storing programs or instructions, wherein when the programs or instructions are executed by the processor, the decompression device implements the methods in the third aspect or any possible implementation thereof.

[0110] The technical effects of aspects four, seven, ten, eleven, and twelfth, or any of their possible implementations, can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.

[0111] The technical effects of aspects 5, 8, 10, 11, and 13, or any of their possible implementations, can be found in aspect 2 or the technical effects of different possible implementations of aspect 2, and will not be repeated here.

[0112] The technical effects of the sixth, ninth, tenth, eleventh, and fourteenth aspects, or any one of their possible implementations, can be found in the third aspect or the technical effects of different possible implementations of the third aspect, and will not be repeated here.

[0113] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Firstly, a first data batch and a second data batch are obtained. During the compression of the first data batch, the parent model can be updated based on the first data batch to obtain a first model, which is then used to participate in the compression of the second data batch. On the one hand, during the compression process, the parent model can be updated while compressing the data batch to be compressed, without needing to save the updated model, thus making it suitable for compressing large-scale datasets. On the other hand, during the parent model update process, the parent model will increasingly conform to the distribution of the data batch to be compressed, therefore, the compression ratio will become increasingly higher. Furthermore, compared to the prior art which requires training a new model specifically for the dataset to be compressed, the parent model in the embodiments of this application has strong versatility and does not require special training for the dataset to be compressed; the model is continuously updated during the compression of the dataset to be compressed. Moreover, this update process can be reproduced during decompression by updating the parent model; therefore, by saving the parent model, the updated model can be obtained, without needing to store every model generated during the compression process, saving storage space. Attached Figure Description

[0114] Figure 1 The system architecture diagram provided in this application is shown below.

[0115] Figure 2 A schematic diagram of a convolutional neural network structure is provided in this application;

[0116] Figure 3 A schematic diagram of another convolutional neural network structure provided in this application;

[0117] Figure 4 A schematic diagram of a chip hardware structure is provided in this application;

[0118] Figure 5 A flowchart illustrating a training method for a master model provided in this application;

[0119] Figure 6A The encoding process of the first-in-first-out encoder provided in this application;

[0120] Figure 6B Decoding process of the first-in-first-out encoder provided in this application;

[0121] Figure 7A The encoding process of the advanced rear-output encoder provided in this application;

[0122] Figure 7B Decoding process of the advanced rear-output encoder provided in this application;

[0123] Figure 7C This application provides a lossless compression framework;

[0124] Figure 8 A flowchart illustrating the data compression method provided in this application;

[0125] Figures 9-12 Several data compression process diagrams provided for this application;

[0126] Figure 13 Another flowchart illustrating the data compression method provided in this application;

[0127] Figures 14-17 Schematic diagrams of other data compression frameworks provided for this application;

[0128] Figure 18 A flowchart illustrating the data decompression method provided in this application;

[0129] Figures 19-22 A schematic diagram illustrating several data decompression frameworks provided for this application;

[0130] Figure 23 A schematic diagram of the compression device according to an embodiment of this application;

[0131] Figure 24 Another structural schematic diagram of the compression device according to an embodiment of this application;

[0132] Figure 25 A schematic diagram of the decompression device according to an embodiment of this application;

[0133] Figure 26 Another structural schematic diagram of the compression device according to an embodiment of this application;

[0134] Figure 27 Another structural schematic diagram of the decompression device according to an embodiment of this application. Detailed Implementation

[0135] This application provides a data compression method, a data decompression method, and related equipment. These methods can improve the compression ratio of batches of data to be compressed.

[0136] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of this application will be introduced below.

[0137] 1. Neural Networks

[0138] Neural networks can be composed of neural units, which can refer to units such as X. s The arithmetic unit takes an intercept of 1 as input, and its output can be:

[0139]

[0140] Where s = 1, 2, ..., n, n is a natural number greater than 1, W sFor X s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0141] 2. Deep Neural Networks

[0142] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three types based on their layer positions: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. However, deep neural networks may not necessarily include hidden layers; this is not a limitation here.

[0143] The function of each layer in a deep neural network can be expressed mathematically. To describe it: From a physical perspective, the work of each layer in a deep neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... Completed, operation 4 is performed by Complete, operation 5 is implemented by α(). The term "space" is used here because the object being classified is not a single thing, but a class of things; space refers to the set of all individuals of this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, as described above; that is, the weights W of each layer control how the space is transformed. The purpose of training a deep neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0144] 3. Convolutional Neural Networks

[0145] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between the same trainable filter and an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in an image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0146] Convolutional kernels can be initialized as matrices of random size. During the training of the convolutional neural network, the kernels can learn to acquire appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers of the convolutional neural network, while also reducing the risk of overfitting. The parent model in this embodiment can specifically be a CNN.

[0147] 4. Recurrent Neural Networks

[0148] In traditional neural network models, layers are fully connected, but nodes within each layer are unconnected. However, this type of ordinary neural network cannot solve many problems. For example, predicting the next word in a sentence, because words in a sentence are not independent; the preceding words are usually needed. Recurrent neural networks (RNNs) refer to a sequence where the current output is related to the previous outputs. Specifically, the network memorizes previous information, storing it in its internal state, and applies it to the calculation of the current output.

[0149] 5. Loss Function

[0150] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value, and this adjustment continues until the network can predict the target value accurately. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0151] 6. Cross-entropy

[0152] Cross-entropy is a commonly used concept in deep learning. It is generally used to calculate the difference between the target value and the predicted value, and is often used as the loss function of neural networks.

[0153] 7. Lossless compression

[0154] Lossless compression: A technique for compressing data so that the compressed data is shorter than the original. The compressed data, when decompressed, must be identical to the original. The core of lossless compression is finding statistical patterns within the data and representing the more probable variable values ​​in the file to be compressed with shorter codes. For example, the letter 'e' appears much more frequently than 'z' in English documents. If 'e' is represented with a shorter code, the document's code length will be shorter. Because this representation is reversible, the compressed file can be recovered from the code, thus achieving lossless compression.

[0155] 8. Compression ratio

[0156] Compression ratio is the ratio of the length of the original data to the length of the compressed data. Without compression, the value is 1. A higher value is better.

[0157] 9. Number of bits per dimension

[0158] Bits per dimension refers to the average bit length of each dimension (byte) of the compressed data. The formula is: 8 / compression ratio. If there is no compression, this value is 8. A smaller value is better.

[0159] 10. Deep Generative Models

[0160] Deep generative models (DGMs) are a probabilistic modeling technique based on deep neural networks. They can model the distribution of data very accurately, thus theoretically achieving higher compression rates than traditional algorithms. Deep generative models are mainly divided into two categories: explicit DGMs and implicit DGMs.

[0161] 11. Display the generative model

[0162] Compared to implicit generative models, explicit generative models can provide the probability distribution values ​​of the input samples. Since lossless compression encoding mechanisms require probability distribution information, the deep generative models currently applicable to lossless compression are explicit generative models. Explicit generative models can be divided into fully observed models and latent variable models based on their model structure.

[0163] 12. Full Observation Model

[0164] A fully observed model directly models the distribution p(x) of a data point x, providing its probability density value directly for each input data point x. In other words, a fully observed model directly models the target variable itself; a representative example is the autoregressive model.

[0165] 13. Latent variable model

[0166] Latent variable models introduce one (or more) latent variables z. For an input data point x, the density value of the data point is indirectly given by p(x) = ∫p(x|z)p(z)dz (taking one latent variable as an example). In other words, latent variable models model the target variable by introducing additional latent variables and modeling the relationship between the latent variables and the target variable. Representative examples include normalizing flows (Flows) and variational autoencoders (VAEs).

[0167] 14. Data Batch

[0168] A data batch can be understood as a large number of data sets, where the data sets belong to the same type (e.g., images, audio, or sequences), or where there is a logical relationship or association between the data sets.

[0169] 15. Probability Distribution Information

[0170] Probability distribution information refers to the probability distribution of a variable's values, or the probability distribution of a symbol. For image data batches, a batch contains multiple data points (or data points), each data point representing an image (including several sub-pixels), and each sub-pixel representing a random variable. For a single-channel 8-bit image, this variable can range from 0 to 255.

[0171] Since deep generative models default to modeling continuous random variables, while lossless compression operates on discrete random variables, current research focuses on adapting existing explicit deep generative models that model continuous variables to lossless compression encoders that operate on discrete variables. For example, for flow models, representative schemes include integer discrete flow (IDF), local bits-back (LBB), and invertible volumetric flow preserving (iVPF); for variational autoencoder models, representative schemes include bits-back asymmetric digital systems (BB-ANS), bit-swap algorithms, and HiLLoC (Hi-Low-Variable Locator Coding).

[0172] While current technologies have solved the compatibility issues between most deep generative models and lossless compression, they have overlooked the storage costs of deep generative models. In practical applications, training deep generative models on large-scale data requires enormous training time and computational resources. Furthermore, training and storing a model for each dataset incurs not only high time costs but also high model storage costs. Although data compression based on deep generative models saves coding space compared to traditional lossless compression algorithms, it becomes counterproductive if the cost of model storage outweighs the space saved through encoding.

[0173] To address the aforementioned issues, this application provides a data compression method, a data decompression method, and related equipment. During the compression process, data batches within the dataset to be compressed can be compressed simultaneously while the parent model is updated (or optimized). As the parent model is updated, it increasingly conforms to the distribution of the dataset to be compressed, thus achieving a higher compression ratio. Furthermore, the updated model does not need to be saved, making it suitable for compressing large-scale datasets.

[0174] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0175] First, the system architecture provided in the embodiments of this application will be introduced.

[0176] See appendix Figure 1This invention provides a system architecture 100. As shown in the system architecture 100, a data acquisition device 160 is used to acquire training data. In this embodiment, the training data includes a training dataset. Further, the training data may include training datasets of multiple data types. These data types may be image data or sequence data (e.g., audio, video, or binary data). The training data is stored in a database 130, and the training device 120 trains a target model / rule 101 based on the training data maintained in the database 130. The following describes in more detail how the training device 120 obtains the target model / rule 101 based on the training data. This target model / rule 101 can be used to implement the compression and / or decompression methods provided in this embodiment. That is, by inputting the data into the target model / rule 101 after relevant preprocessing, the probability distribution information of each variable value in the data can be obtained. Specifically, the target model / rule 101 in this embodiment can be a deep generative model. In the embodiment provided in this application, the deep generative model is a parent model, which is obtained by training the training dataset. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0177] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, AR / VR, vehicle terminal, etc., or it can be a server or cloud, etc. (See attached...) Figure 1 In this embodiment, the execution device 110 is equipped with an I / O interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include: a dataset to be compressed or a batch of data to be compressed (i.e., the first batch of data and the second batch of data). It may be input by the user, uploaded by the user through the shooting device, or it may come from the database. The specific details are not limited here.

[0178] If the input data is a dataset to be compressed, the preprocessing module 113 is used to preprocess the dataset to be compressed received by the I / O interface 112. In this embodiment, the preprocessing module 113 can be used to split the dataset to be compressed into a first data batch and a second data batch. If the input data is a data batch to be compressed, in other words, if the input data is a first data batch and a second data batch, the above-mentioned preprocessing may not be required.

[0179] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0180] Finally, the I / O interface 112 returns the processing results, such as the probability distribution information of the values ​​of each variable in the data batch obtained above, to the client device 140, thereby providing it to the user or input encoder.

[0181] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results or providing input for subsequent compression / decompression.

[0182] In the appendix Figure 1 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0183] It is worth noting that, attached Figure 1This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0184] like Figure 1 As shown, the target model / rule 101 is obtained by training the training device 120. In this embodiment of the application, the target model / rule 101 can be the parent model. Specifically, in the network provided in this embodiment of the application, for image processing, the parent model can be a convolutional neural network.

[0185] Since CNN is a very common type of neural network, the following will combine... Figure 2 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0186] like Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0187] Convolutional / pooling layers 120:

[0188] Convolutional layers:

[0189] like Figure 2 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0190] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0191] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0192] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0193] Pooling layer:

[0194] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 2Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0195] Neural network layer 130:

[0196] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 2 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0197] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0198] It should be noted that, as Figure 2The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 3 The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0199] The following describes a chip hardware structure provided by an embodiment of this application.

[0200] Figure 4 A chip hardware structure provided in this embodiment of the invention includes a neural network processor 40. This chip can be configured as follows: Figure 1 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be placed in, for example... Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 2 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 4 This is achieved in the chip shown.

[0201] The neural network processor 40 can be any processor suitable for large-scale XOR operations, such as a neural network processing unit (NPU), tensor processing unit (TPU), or graphics processing unit (GPU). Taking an NPU as an example: the neural network processor NPU40 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the main CPU. The core of the NPU is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to retrieve data from the memory (weight memory or input memory) and perform operations.

[0202] In some implementations, the arithmetic circuit 403 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 403 is a two-dimensional pulsating array. The arithmetic circuit 403 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.

[0203] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 402 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 401 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 408.

[0204] The vector computation unit 407 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 407 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0205] In some implementations, the vector computation unit 407 can store the processed output vector into a unified buffer 406. For example, the vector computation unit 407 can apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 407 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 403, for example, for use in subsequent layers of a neural network.

[0206] The unified memory 406 is used to store input data and output data.

[0207] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller 405 (DMAC), the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 506 is stored in the external memory.

[0208] The bus interface unit (BIU) 410 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 409 via a bus.

[0209] The instruction fetch buffer 409, which is connected to the controller 404, is used to store the instructions used by the controller 404.

[0210] The controller 404 is used to call the instructions cached in the instruction memory 409 to control the operation of the computing accelerator.

[0211] Generally, the unified memory 406, input memory 401, weight memory 402, and instruction fetch memory 409 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0212] in, Figure 2 or Figure 3 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 403 or the vector calculation unit 407.

[0213] The training method, data compression method, and data decompression method of the master model according to the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0214] First, combined Figure 5 The training method of the parent model in the embodiments of this application will be described in detail. Figure 5 The training method shown can be executed by a training device for the parent model. This training device can be a cloud service device or a terminal device, such as a computer, server, or other device with sufficient computing power to execute the training method for the parent model. It can also be a system composed of cloud service devices and terminal devices. For example, the training method can be executed by... Figure 1 Training equipment 120 Figure 4 The neural network processor 40 in the system executes the commands.

[0215] Optionally, the training method can be processed by the CPU, or by both the CPU and GPU, or it can use other processors suitable for neural network computing instead of a GPU. This application does not impose any restrictions.

[0216] Figure 5 The training method shown includes steps 501 and 502. Steps 501 and 502 are described in detail below.

[0217] Step 501: Obtain training data batch.

[0218] There are several ways to obtain training data batches in this application embodiment. It can be obtained by first acquiring the training dataset and then dividing it, or by directly acquiring the training data batch. The specific method is not limited here. The training dataset or training data batch can be acquired by a data acquisition device (such as a camera, terminal, etc.), or it can be directly selected from datasets such as CIFAR-10 or ImageNet as the training dataset. The specific method is not limited here.

[0219] Optionally, the training dataset can be obtained and split into multiple training data batches, such as a first training data batch, a second training data batch, and a third training data batch. Of course, the number of batches into which the training dataset can be split can be set according to actual needs, such as 3 training data batches or 10 training data batches, etc., and is not limited here.

[0220] Step 502: Using the training data batch as input to the neural network model, train the neural network model with the goal of the loss function value being less than the first threshold, and obtain the trained parent model.

[0221] Optionally, after obtaining the training data batch, a corresponding deep generative model can be selected as the neural network model based on the data type of the training data batch (or the data type of the training dataset). The data type can be image data type, text data type, audio data type, or binary data type, etc., and is not specifically limited here.

[0222] Optionally, if a training dataset is obtained in step 501, the parent model can be selected based on the data type of the training dataset (similar to the batch of training data), which is similar to the batch of training data described above, and will not be elaborated here.

[0223] Because deep generative models have good generalization ability to similar data, they are beneficial for subsequent compression or decompression of similar data. Therefore, this application only describes deep generative models as a parent model in its embodiments.

[0224] For example, the correspondence between the data types of the training dataset or training data batch and the parent model can be shown in Table 1.

[0225] Table 1

[0226]

[0227] The data types and parent models shown in Table 1 are just examples. In actual applications, there are other data types, parent models, and other ways of corresponding data types and parent models. These are not specified here.

[0228] In this scenario, the neural network model is trained with the goal of reducing the value of the loss function. This means continuously narrowing the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch (i.e., the loss function indicates the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch). This training process can be understood as a probability generation task. The loss function can be understood as the loss function corresponding to the probability generation task.

[0229] Optionally, the optimized parent model is obtained by using an optimization algorithm and batch optimization of the training data to obtain an optimized neural network model. This optimization algorithm can be a gradient-based optimization algorithm, a meta-learning algorithm, or a reinforcement learning algorithm, etc., and is not limited here. It can also be understood that the process of optimizing the neural network model is the process of optimizing the loss function in the neural network model, generally aiming to minimize the loss.

[0230] For example, continuing the above example, the training dataset is divided into a first training data batch, a second training data batch, and a third training data batch. The deep generative model is then optimized sequentially using the first training data batch, the second training data batch, and the third training data batch to obtain the parent model.

[0231] The following description uses only the first batch of training data and a gradient-based optimization algorithm as examples to illustrate the process of optimizing a neural network model using the first batch of training data. Assume that the first batch of training data obtained... Where x 1i Let be the i-th data point in the first training data batch, and n be the size of the first training data batch. From the neural network model... Obtain the probability distribution information of each variable value in x, and use the first training data batch B1 to calculate the gradient of the loss function L(θ0,B1). Then, the parameters of the neural network model are updated using a gradient-based optimization algorithm, resulting in the parameters of the new neural network model. That is, a new neural network model is obtained. Here, λ represents the learning rate, which can be set according to actual needs; no specific limit is imposed here. To ensure good performance of gradient descent, the learning rate should be set within an appropriate range. A learning rate that is too large leads to learning instability, while a rate that is too small results in extremely long training times. Adaptive learning rate achieves a reasonably high rate while ensuring stable training, thus reducing training time.

[0232] In this approach, the neural network model is optimized based on batches of training data in the training dataset (e.g., the first batch of training data), so that the trained parent model increasingly conforms to the distribution of the training dataset.

[0233] It should be noted that the training process may also use other training methods instead of the aforementioned methods, and no restrictions are imposed here.

[0234] In this application embodiment, due to the different technical characteristics of different encoders, encoders can be divided into two types: first-in-first-out (FIFO) and last-in-first-out (LIFO).

[0235] The following is combined Figure 6A , Figure 6B , Figure 7A , Figure 7B This section provides a brief overview of the encoding and decoding processes of First-In-First-Out (FIFO) and Last-In-First-Out (LIFO) encoders (probability distribution information is omitted).

[0236] Please see Figure 6A and Figure 6B , Figure 6A For the encoding (or compression) process of a first-in-first-out encoder, Figure 6B This describes the decoding (or decompression) process for a first-in-first-out (FIFO) encoder. For FIFO encoders (such as arithmetic encoders), during the decompression stage, different data points (or data batches) can only be decompressed from the encoded file (or encoded sequence, final code, etc.) in the order of compression.

[0237] Please see Figure 7A and Figure 7B , Figure 7A For the encoding (or compression) process of a first-in, last-out encoder, Figure 7B This describes the decoding (or decompression) process for a Last-In-First-Out (LIFO) encoder. For LIFO encoders (such as asymmetric digital systems), the decompression stage can only extract different data points (or data batches) from the encoded file (or encoded sequence, final code, etc.) in reverse order of compression. To achieve the data batch order of data batch 1-data batch 2-data batch 3-data batch 4 in the decompression process of 7B, the compression process of 7A can be reversed, i.e., starting from the last data batch (e.g.,... Figure 7A The data batch 4) in the middle is compressed forward so that all data batches can be decompressed from front to back during decompression.

[0238] Please see Figure 7C This application provides a lossless compression framework, which includes a compression process, an optimization process, and a decompression process. The branch on the left of this figure explains how p is used in step t during the compression process. t-1 Compress the t-th data batch (i.e., batch) t ), and use this data batch to generate a deep generative model from p t-1 Update to p t The branch on the right of this diagram explains how p is used in step t during the decompression process. t-1Decompress the t-th data batch and use this data batch to extract the deep generative model from p. t-1 Update to p t Here, p0 can be understood as the parent model (or initial model). This framework can compress the new dataset while simultaneously transferring (or optimizing) the parent model, without needing to save the transferred new model. During decompression, the framework can decompress the encoding while simultaneously transferring the parent model, thus recovering all the models needed for decompression and the corresponding original data. During the transfer process, the model will increasingly conform to the distribution of the target data, thus increasing the compression ratio. Furthermore, due to the good generalization ability of deep generative models to similar data, the compression ratio is also relatively high at the beginning of the transfer process.

[0239] The data compression method and data decompression method of this application embodiment will be described in detail below with reference to the above application scenarios and the accompanying drawings.

[0240] The compression process differs slightly between First-In-First-Out (FIFO) and Last-In-First-Out (LIFO) encoders, and is described below. For the compression process of FIFO encoders, please refer to [link to relevant documentation]. Figure 8 For the compression process of advanced-last-out encoders, please refer to [link / reference]. Figure 13 .

[0241] Please see Figure 8 This application provides an embodiment of a data compression method for a first-in-first-out encoder. The method can be executed by a compression device or by a component of the compression device (such as a processor, chip, or chip system). This embodiment includes steps 801 to 806.

[0242] In step 801, the first data batch and the second data batch are obtained.

[0243] In this embodiment, there are multiple ways to obtain the first data batch and the second data batch. These methods include directly obtaining the first and second data batches, or obtaining the dataset to be compressed and splitting it to obtain the first and second data batches. No specific method is limited here. The dataset to be compressed can have the same or similar data type as the training dataset. Of course, if the data type of the dataset to be compressed is the same as the data type of the training dataset, it is beneficial to improve the compression ratio of the dataset to be compressed.

[0244] In step 802, the parent model is obtained.

[0245] Optionally, the compression device can obtain the parent model based on the data type of the dataset to be compressed or the batch of data to be compressed (e.g., the first batch or the second batch). The specific process for determining the parent model based on data type can be found in the preceding text. Figure 5The correspondence between data types and the parent model described in step 502 of the illustrated embodiment can be found in Table 1 above, and will not be repeated here.

[0246] Optionally, the parent model can be adopted Figure 5 The training method shown is used to train the parent model.

[0247] In step 803, the first data batch is compressed based on the first data batch, the master model, and the first encoder to obtain the first sequence.

[0248] Because different deep generative models have different probabilistic graph structures, the encoders adapted to this model may differ. Optionally, after obtaining the parent model, a first encoder corresponding to the type of the parent model is determined based on a first association relationship, which represents the association between the type of the parent model and the first encoder. The type of the parent model may include the aforementioned fully observed model and latent variable model.

[0249] Optionally, the type of the parent model and the first association with the first encoder can be as shown in Table 2.

[0250] Table 2

[0251]

[0252] For the fully observed model, a wide range of encoders are suitable, including arithmetic coding, asymmetric digital systems, and Huffman coding. For the latent variable model, since it requires the use of inverse coding techniques, it is more suitable for encoders that combine inverse coding techniques with asymmetric digital systems; however, encoders that combine inverse coding techniques with arithmetic coding can also be used.

[0253] It is understandable that Table 2 above is just an example. In actual applications, there are other types, encoders, or first associations, which are not specified here.

[0254] After obtaining the first data batch and the parent model, inputting the first data batch into the parent model yields the first probability distribution information. This first probability distribution information represents the probability distribution of each variable's value in the first data batch. The first probability distribution information and the first data batch are then input into the first encoder for compression to obtain the first sequence. For example, the process for obtaining the first sequence can be referred to... Figure 9 .

[0255] In step 804, the parent model is updated based on the first data batch to obtain the first model.

[0256] In this embodiment of the application, there are many ways to update (or optimize) the parent model based on the first data batch to obtain the first model. Gradient-based optimization algorithms, meta-learning, reinforcement learning, etc. can be used, and no specific limitation is made here.

[0257] Optionally, the process of optimizing the parent model based on the first data batch to obtain the first model is the same as described above. Figure 5 The method of optimizing the parent model based on the first training data batch in step 502 of the illustrated embodiment is similar, and will not be described in detail here. For example, the process of optimizing the parent model can be referred to... Figure 9 .

[0258] Optionally, if the dataset to be compressed includes N data batches, then M data batches can be used to optimize the parent model separately, where N is a positive integer greater than or equal to 2, and M is a positive integer less than or equal to N. This can also be understood as the parent model being optimized once or N-1 times; the specific number of optimizations is not limited here.

[0259] Optionally, the first model can be obtained by optimizing the parent model using the first data batch multiple times. It is understood that the data batch in the data batch-based optimization model proposed in this application embodiment can be used once or multiple times; no specific limitation is made here. If the data batch optimization model is used multiple times, the number of times is saved, or it can be understood as saving the number of gradient update steps, thereby achieving reproducibility in the decompression process.

[0260] In step 805, the third data batch is obtained. This step is optional.

[0261] In this embodiment, obtaining the third data batch is similar to step 801 described above. It can be done by directly obtaining the third data batch or by obtaining the third data batch from the dataset to be compressed. The specific method is not limited here.

[0262] It is understood that this step can be a standalone step or part of step 801. If step 805 is part of step 801, that is, the dataset to be compressed includes a first data batch, a second data batch, and a third data batch, the compression device splits the dataset to be compressed to obtain the first data batch, the second data batch, and the third data batch.

[0263] Optionally, if the dataset to be compressed includes N data batches, the dataset to be compressed can be split into N data batches.

[0264] In step 806, the second data batch is compressed based on the second data batch, the first model, the first sequence, and the first encoder to obtain the second sequence.

[0265] Optionally, if the data batch to be compressed includes a first data batch and a second data batch, the second data batch can be input into the first model to obtain second probability distribution information. This second probability distribution information is used to represent the probability distribution of each variable value in the second data batch. The second data batch, the second probability distribution information, and the first sequence are then input into the first encoder for compression to obtain the second sequence. For example, the process for obtaining the second sequence can be referred to... Figure 9 .

[0266] Alternatively, if arithmetic coding is used, the first sequence and the second sequence may be the same.

[0267] The steps differ depending on the number of updates to the parent model in this embodiment. The following description uses a dataset to be compressed that includes a first batch of data, a third batch of data, and a second batch of data as an example.

[0268] The first scenario involves updating the parent model a number that differs from the number of data batches by one.

[0269] If the data batch to be compressed includes a first data batch, a second data batch, and a third data batch, then the third data batch can be compressed based on the third data batch, the first model, the first sequence, and the first encoder to obtain the third sequence. The first model is then optimized using the third data batch to obtain the second model. Finally, the second data batch is compressed based on the second data batch, the second model, the third sequence, and the first encoder to obtain the second sequence. Essentially, the dataset to be compressed includes three data batches, and the parent model is optimized twice.

[0270] Optionally, after obtaining the third data batch, the third data batch is input into the first model to obtain third probability distribution information. This third probability distribution information is used to identify the probability distribution of each variable value in the third data batch. The third probability distribution information and the first sequence are then input into the first encoder to compress the third data batch to obtain the third sequence. The third data batch is then used to optimize the first model to obtain the second model. The second data batch is input into the second model to obtain second probability distribution information, and the third sequence and the second probability distribution information are then input into the first encoder to compress the second data batch to obtain the second sequence. For example, the process for obtaining the second sequence can be referred to... Figure 10 .

[0271] The second scenario is that the number of times the parent model is updated differs from the number of data batches by more than two.

[0272] In this case, it can also be understood as an early cessation of model updates. That is, the first model was not optimized using the third batch of data. The difference between this case and the first case mentioned above is that the first model was not being optimized further.

[0273] If the data batch to be compressed includes a first data batch, a second data batch, and a third data batch, then the third data batch can be compressed based on the third data batch, the first model, the first sequence, and the first encoder to obtain the third sequence. Then, the second data batch can be compressed based on the second data batch, the first model, the third sequence, and the first encoder to obtain the second sequence.

[0274] Optionally, after obtaining the third data batch, the third data batch is input into the first model to obtain the third probability distribution information. The third probability distribution information and the first sequence are then input into the first encoder to compress the third data batch to obtain the third sequence. The second data batch is input into the first model to obtain the second probability distribution information, and the third sequence and the second probability distribution information are then input into the first encoder to compress the second data batch to obtain the second sequence. For example, the process for obtaining the second sequence can be referred to... Figure 11 This is equivalent to the dataset to be compressed consisting of three batches of data, with the parent model optimized once.

[0275] In this embodiment of the application, if the dataset to be compressed includes N data batches, the compression process for the first-in-first-out encoder can be referred to Figure 12 The specific process is similar to the steps in the embodiment shown in Figure 8, and will not be repeated here. The first data batch can be considered the first compressed data batch, the second data batch the last compressed data batch, and the third data batch the batch between the first and second data batches. Furthermore, there can be more data batches between the first and second data batches besides the third data batch; the specific number of data batches is not limited here. N is a positive integer greater than 3. The Nth sequence is equivalent to the second sequence, the first data batch is equivalent to the first data batch, and the Nth data batch is equivalent to the second data batch. It can be understood that the (N-1)th model and the (N-2)th model can be the same, or they can be obtained by optimizing the (N-2)th model and the (N-1)th data batch; the specifics are not limited here. This is equivalent to the parent model being optimized N-1 times, N-2 times, N-3 times, etc.

[0276] Optionally, if the parent model is not optimized using N-1 data batches, a prompt message can be generated and stored along with the parent model. This prompt message indicates how many times the parent model should be updated, or which datasets in the dataset to be compressed were not used to update the model. This allows the decompression device to determine whether to update the model during the decompression process, or to what extent (e.g., how many times to update, and which data batches to use to update the model).

[0277] Alternatively, it can be pre-configured so that the decompression device can determine whether to update the model, or to what extent.

[0278] In addition, if there is randomness in the optimization of the parent model (or compression process), the corresponding random seed is saved to facilitate lossless compression.

[0279] In one possible implementation, this embodiment includes steps 801 to 806; in another possible implementation, this embodiment includes steps 801, 802, 803, 804, and 806.

[0280] In this embodiment, the compression process and the optimization process of the parent model can be performed in parallel or sequentially. For example, step 804 of optimizing the parent model can be performed before step 803 of the compression process, or step 804 can be performed after step 803, or steps 803 and 804 can be performed simultaneously. No specific limitation is made here.

[0281] In this embodiment, on the one hand, during the compression process, the parent model can be optimized while compressing batches of data in the dataset to be compressed, without needing to save the optimized new model, thus making it suitable for compressing large-scale datasets. On the other hand, during the optimization process of the parent model, the parent model will increasingly conform to the distribution of the dataset to be compressed, thus the compression ratio will become higher and higher. In addition, compared with the prior art that requires training a new model specifically for the dataset to be compressed, the parent model in this embodiment has strong versatility and does not need to be trained specifically for the dataset to be compressed; the model is continuously optimized during the compression process. Moreover, this optimization process can be reproduced by optimizing the parent model during decompression, so the optimized model can be obtained by saving the parent model, without needing to store every model generated during the compression process, saving storage space.

[0282] Please see Figure 13 This application provides an embodiment of a data compression method for a first-in, last-out encoder. The method can be executed by a compression device or by a component of the compression device (such as a processor, chip, or chip system). This embodiment includes steps 1301 to 1306.

[0283] In step 1301, the first data batch and the second data batch are obtained.

[0284] In step 1302, the parent model is obtained.

[0285] Steps 1301 and 1302 in this embodiment are the same as those described above. Figure 8 Steps 801 and 802 in the illustrated embodiment are similar and will not be described in detail here.

[0286] Optionally, when splitting the dataset to be compressed, the multiple data batches obtained from the split can be sequentially marked, so that the compression device can identify which data batch is the last data batch during the subsequent compression process.

[0287] Optionally, after obtaining the parent model, probability distribution information can be obtained from the parent model and saved. In other words, since this embodiment is a last-in-first-out encoder (or a last-in-first-out encoder), it needs to process from the last data batch to the first data batch. Therefore, the first data batch obtained at the beginning and the first probability distribution information obtained from the parent model are first put into the cache.

[0288] In step 1303, the parent model is updated based on the first data batch to obtain the first model.

[0289] Step 1303 in this embodiment is the same as described above. Figure 8 Step 804 in the illustrated embodiment is similar and will not be described in detail here.

[0290] In step 1304, the second data batch is compressed based on the second data batch, the first model, and the first encoder to obtain the first sequence.

[0291] In this embodiment, the method for determining the first encoder is the same as described above. Figure 8 Step 803 in the illustrated embodiment is similar and will not be described in detail here.

[0292] Optionally, if the dataset to be compressed includes a first data batch and a second data batch, the second data batch is input into the first model to obtain second probability distribution information. This second probability distribution information is used to represent the probability distribution of each variable value in the second data batch. The second probability distribution information and the second data batch are then input into the first encoding and compressed to obtain the first sequence. For example, the process of obtaining the first sequence can be referred to... Figure 14 .

[0293] The steps differ depending on the number of updates to the parent model in this embodiment. The following description uses a dataset to be compressed that includes a first batch of data, a third batch of data, and a second batch of data as an example.

[0294] The first scenario involves updating the parent model a number that differs from the number of data batches by one.

[0295] Optionally, if the dataset to be compressed includes a first data batch, a second data batch, and a third data batch, the first data batch can be used to optimize the parent model to obtain a first model, and then the second data batch can be used to optimize the first model to obtain a second model. The second data batch is then input into the second model to obtain second probability distribution information, which represents the probability distribution of each variable value in the second data batch. The second probability distribution information and the second data batch are then input into the first encoding for compression to obtain a first sequence. For example, the process for obtaining the first sequence can be referred to... Figure 15 This is equivalent to the dataset to be compressed consisting of three batches of data, with the parent model optimized twice.

[0296] The second scenario is that the number of times the parent model is updated differs from the number of data batches by more than two.

[0297] In this case, it can also be understood as an early cessation of model updates. That is, the first model was not optimized using the third batch of data. The difference between this case and the first case mentioned above is that the first model was not being optimized further.

[0298] Optionally, if the dataset to be compressed includes a first data batch, a second data batch, and a third data batch, the first data batch can be used to optimize the parent model to obtain a first model. The second data batch is then input into the first model to obtain second probability distribution information, which represents the probability distribution of each variable's value in the second data batch. The second probability distribution information and the second data batch are then input into the first encoding for compression to obtain a first sequence. For example, the process for obtaining the first sequence can be referred to... Figure 16 This is equivalent to the dataset to be compressed consisting of three batches of data, with the parent model optimized once.

[0299] In this embodiment of the application, if the dataset to be compressed includes N data batches, the compression process for the Last-In-First-Out (LIFO) encoder can be referred to Figure 17 The specific process is similar to the steps in the embodiment shown in Figure 13, and will not be repeated here. The first data batch can be considered the first compressed data batch, the second data batch the last compressed data batch, and the third data batch the batch between the first and second data batches. Furthermore, there can be more data batches between the first and second data batches besides the third data batch; the specific number of data batches is not limited here. N is a positive integer greater than 3. The Nth sequence is equivalent to the second sequence, the first data batch is equivalent to the first data batch, and the Nth data batch is equivalent to the second data batch. It can be understood that the (N-1)th model and the (N-2)th model can be the same, or they can be obtained by optimizing the (N-2)th model and the (N-1)th data batch; the specifics are not limited here. This is equivalent to the parent model being optimized N-1 times, N-2 times, N-3 times, etc.

[0300] In step 1305, the third data batch is obtained. This step is optional.

[0301] Step 1305 in this embodiment is the same as described above. Figure 8 Step 805 in the illustrated embodiment is similar and will not be described in detail here.

[0302] In step 1306, the first data batch is compressed based on the first data batch, the master model, the first sequence, and the first encoder to obtain the second sequence.

[0303] After obtaining the first data batch and the parent model, inputting the first data batch into the parent model yields the first probability distribution information, which represents the probability distribution of each variable's value in the first data batch. The first probability distribution information, the first sequence, and the first data batch are then input into the first encoder for compression to obtain the second sequence. For example, the process for obtaining the second sequence can be referred to... Figure 13 , Figure 14 , Figure 15 , Figure 16 or Figure 17 .

[0304] Optionally, if the dataset to be compressed includes a first data batch and a second data batch, the first data batch is input into the parent model to obtain first probability distribution information, which represents the probability distribution of each variable value in the first data batch. The first probability distribution information, the first sequence, and the first data batch are then input into the first encoder to obtain the second sequence. For example, the process for obtaining the second sequence can be referred to... Figure 14 .

[0305] The number of updates to the parent model differs in this embodiment, but the steps are similar. The following description uses a dataset to be compressed, including a first batch of data, a third batch of data, and a second batch of data, as an example.

[0306] Optionally, after obtaining the first sequence, a third data batch can be input into the first model to obtain third probability distribution information, which represents the probability distribution of each variable value in the third data batch. The third probability distribution information, the first sequence, and the third data batch are then input into the first encoder for compression to obtain the third sequence. The first data batch is input into the parent model to obtain first probability distribution information, which represents the probability distribution of each variable value in the first data batch. The first probability distribution information, the third sequence, and the first data batch are then input into the first encoder for compression to obtain the second sequence.

[0307] For example, if the number of times the parent model is updated differs from the number of data batches by one, the process for obtaining the second sequence can be found in [reference needed]. Figure 15 This is equivalent to the dataset to be compressed consisting of three batches of data, with the parent model optimized twice.

[0308] For example, if the number of times the parent model is updated differs from the number of data batches by more than two, the process for obtaining the second sequence can be found in [reference needed]. Figure 16 This is equivalent to the dataset to be compressed consisting of three batches of data, with the parent model optimized once.

[0309] In this embodiment of the application, if the dataset to be compressed includes N data batches, the compression process for the Last-In-First-Out (LIFO) encoder can be referred to Figure 17 . Figure 17 Compression process and Figure 15 or Figure 16 The compression process shown is similar and will not be repeated here.

[0310] Optionally, if the parent model is not optimized using N-1 data batches, a prompt message can be generated and stored along with the parent model. This prompt message indicates how many times the parent model should be updated, or which datasets in the dataset to be compressed were not used to update the model. This allows the decompression device to determine whether to update the model, and to what extent, during the decompression process.

[0311] In addition, if there is randomness in the optimization of the parent model (or compression process), the corresponding random seed is saved to facilitate lossless compression.

[0312] In one possible implementation, this embodiment includes steps 1301 to 1306; in another possible implementation, this embodiment includes steps 1301, 1302, 1303, 1304, and 1306.

[0313] The timing relationship of the steps in the embodiments of this application is not fixed. For example, step 1303 of optimizing the parent model can be performed before step 1304 of the compression process, or step 1303 can be performed after step 1304, or steps 1303 and 1304 can be performed simultaneously. No specific limitation is made here.

[0314] In this embodiment, on the one hand, due to the characteristics of this type of encoder (i.e., Last-In-First-Out), the preceding data batches need to be compressed in the following order; that is, the order in which the data batches are compressed needs to be the reverse of the order of the transfer model data batches. All compressed data batch information is placed in a cache and compressed in reverse order after model optimization is completed. During this process, the optimized new model does not need to be saved, thus making it suitable for compressing large-scale datasets. On the other hand, during the optimization of the parent model, the parent model will increasingly conform to the distribution of the dataset to be compressed; therefore, the compression ratio will become increasingly higher. Furthermore, through the reverse adjustment of the Last-In-First-Out encoder, the decompression process can obtain data batches sequentially.

[0315] Please see Figure 18 This application provides an embodiment of a data decompression method, which can be executed by a decompression device or by a component of the decompression device (such as a processor, chip, or chip system). This embodiment includes steps 1801 to 1805.

[0316] In step 1801, the second sequence is obtained.

[0317] The second sequence in this embodiment can be the final encoded sequence (i.e., the second sequence) obtained by the compression method described above, or it can be the second sequence sent by the receiving compression device, or it can be obtained by other means, such as the second sequence forwarded by other devices. The specific details are not limited here.

[0318] In step 1802, the parent model is obtained.

[0319] In this embodiment, the method for obtaining the parent model can be as described above. Figure 5 The training method shown can be used to train a master model, or it can be a master model sent by a compression device, etc. The specific method is not limited here.

[0320] The parent model in this embodiment is the same as the one described above. Figure 13 The parent model in the illustrated embodiments is the same, thus enabling lossless compression.

[0321] In step 1803, the fourth sequence and the first data batch are obtained by decompressing the second sequence based on the parent model and the first encoder.

[0322] The method for determining the first encoder in this embodiment is the same as described above. Figure 8 The steps described in step 803 of the illustrated embodiment are similar and will not be repeated here.

[0323] After obtaining the parent model, the probability distribution carried by the parent model can be obtained. Depending on whether the parent model includes conditional probability, this step can be divided into two cases, which are described below.

[0324] The first type involves a parent model that does not include conditional probabilities, allowing direct access to probability distribution information.

[0325] In other words, the first probability distribution information (i.e., the distribution p(x) mentioned in the previous explanation) can be directly obtained from the parent model. This first probability distribution information is used to represent the probability distribution of the values ​​of each variable in the first data batch. The first probability distribution information and the first encoder are then used to decompress the second sequence to obtain the fourth sequence and the first data batch.

[0326] Alternatively, if arithmetic encoding is used, the fourth sequence may be the same as the second sequence.

[0327] The second type involves a parent model that includes conditional probabilities, which can indirectly provide information about probability distributions.

[0328] In other words, incomplete probability information (i.e., the first probability) can be obtained from the parent model, and the first probability and the first encoder can be used to decompress the second sequence to obtain a portion of data. This portion of data is then input into the parent model to obtain the second probability. Based on the second probability and the first encoder, the second sequence is decompressed to obtain another portion of data and the fourth sequence. The first data batch can be obtained from this portion of data and the other portion of data.

[0329] For example, a portion of the probability p is obtained through the parent model. x2 , using p x2 Decompress the second sequence to obtain A2, input A2 into the parent model to obtain p(x1|A2), decompress the second sequence using p(x1|A2) to obtain A1, and finally obtain the first data batch (A1,A2).

[0330] In step 1804, the parent model is updated based on the first batch of data to obtain the first model.

[0331] The method used in this step to obtain the first model based on updating (or optimizing) the parent model using the first batch of data is the same as described above. Figure 8 The steps described in step 804 of the illustrated embodiment are similar and will not be repeated here.

[0332] Optionally, if a random seed is present, optimization is performed based on the random seed, thereby ensuring that the optimized model during decompression is consistent with or within a certain error range as the optimized model during compression, thus achieving lossless compression.

[0333] Optionally, during or before optimizing the parent model using the first data batch, the number of times the data batch was used can be obtained, as described above. Figure 8 The gradient is updated a certain number of times in the illustrated embodiment, and the parent model is updated based on this number of times, thereby achieving reproducibility in the decompression process.

[0334] In step 1805, the fourth sequence is decompressed based on the first model and the first encoder to obtain the second data batch.

[0335] After optimizing the parent model to obtain the first model, the second probability distribution information can be obtained from the first model (the specific method of obtaining the probability distribution information can be either direct or indirect, as described above). This second probability distribution information is used to represent the probability distribution of each variable value in the second data batch. The second probability distribution information and the first encoder are then used to decompress the fourth sequence to obtain the second data batch. Thus, the first and second data batches are decompressed. For example, as shown... Figure 19 The decompression process is shown below. Optionally, the initial dataset, i.e., the dataset to be compressed before compression, is recovered based on the first data batch, the second data batch, and the decompression order. This achieves lossless compression.

[0336] Optionally, the second probability distribution information and the first encoder are used to decompress the fourth sequence to obtain the fifth sequence and the third data batch. The third data batch is then used to optimize the first model to obtain the second model. The second probability distribution information is obtained through the second model, and the second probability distribution information and the first encoder are used to decompress the fifth sequence to obtain the second data batch. This decompresses the first data batch, the second data batch, and the third data batch. For example, as shown... Figure 20 The decompression process is shown below.

[0337] Optionally, the second probability distribution information and the first encoder are used to decompress the fourth sequence to obtain the fifth sequence and the third data batch. If the first model carries a prompt message or receives a prompt message from the compression device, and this prompt message indicates that the first model was not optimized into the second model during compression (or can be understood as indicating early stop information during compression; the decompression device can determine the number of times the parent model was optimized based on the early stop information, and thus determine the corresponding probability distribution of the model), then the decompression device can obtain the second probability distribution information through the first model, and use the second probability distribution information and the first encoder to decompress the fifth sequence to obtain the second data batch. This decompresses the first data batch, the second data batch, and the third data batch. For example, as shown... Figure 21 The decompression process is shown below.

[0338] Optionally, the initial dataset, i.e., the dataset to be compressed before compression, is recovered based on the first data batch, the second data batch, the third data batch, and the decompression order. This achieves lossless compression.

[0339] In this embodiment of the application, if the dataset to be compressed includes N data batches, the decompression process can be referred to Figure 22 The specific process is similar to the steps in the embodiment shown in Figure 18, and will not be repeated here. The first data batch can be considered the first compressed data batch, the second data batch the last compressed data batch, and the third data batch the batch between the first and second data batches. Furthermore, there can be more data batches between the first and second data batches besides the third data batch; the specific number of data batches is not limited here. N is a positive integer greater than 3. The Nth sequence is equivalent to the second sequence, the first data batch is equivalent to the first data batch, and the Nth data batch is equivalent to the second data batch. It can be understood that the (N-1)th model and the (N-2)th model can be the same, or they can be obtained by optimizing the (N-2)th model and the (N-1)th data batch; the specifics are not limited here. This is equivalent to the parent model being optimized N-1 times, N-2 times, N-3 times, etc.

[0340] Optionally, the initial dataset, i.e., the dataset to be compressed before compression, is recovered based on the first data batch, the (N-1)th data batch, the Nth data batch, and the decompression order. This achieves lossless compression.

[0341] In this embodiment, on the one hand, during decompression, the encoded sequence can be decompressed while the parent model is optimized, and the optimized new model does not need to be saved, thus making it suitable for decompressing large-scale datasets. On the other hand, based on prompts or preset configuration information, the optimized model used in the compression process can be obtained, thereby determining the probability distribution, obtaining the initial dataset, and achieving lossless compression. Furthermore, since it does not require dedicated training on the dataset to be compressed as in existing technologies, the compression time is reduced compared to the time spent on dedicated training of the dataset to be compressed. In addition, compared to existing technologies that require dedicated training of new models on the dataset to be compressed, the parent model in this embodiment is more versatile and does not need to be trained on the dataset to be compressed. The model is continuously optimized during the compression process. Moreover, this optimization process can be reproduced by optimizing the parent model during decompression. Therefore, by saving the parent model, the optimized model can be obtained, and it is not necessary to store every model generated during the compression process, saving storage space.

[0342] In this embodiment of the application, if the dataset to be compressed is too large, it can be divided into multiple data clusters, and then the compression or decompression method described above can be used to process each data cluster separately. Thus, for Figures 13 to 17 In compression methods, the required cache space can be reduced. This refers to the storage space needed to store data batches and / or the probability distribution information corresponding to those data batches.

[0343] Optionally, each data cluster generates an independent code, and the order of the different independent codes must be the same as the order of the clusters. For example, six data batches are divided into two data clusters. Since the data compression direction is reversed within each data cluster, a larger cache space is required than that of a FIFO encoder. Before the online migration of the model for a data batch in the same data cluster is completed, the cache space will store the data batch and the probability quality distribution required for compression in sequence; when the model optimization of the data batch in that data cluster is completed, the encoder will use the information in the cache to compress and encode the data batch in reverse order. Due to the caching mechanism, in the online migration of a LIFO encoder, the encoder's compression process lags behind the online migration process of the model and can achieve a certain degree of parallelism. In the decompression stage, the encoded files are read in the order of the clusters, and the data batch of the corresponding cluster is decompressed from each encoded file. Each step of online compression and decompression first uses the current model to decompress the data batch of the current step, and then uses the data batch decompressed in the current step to migrate the model. Since model optimization cannot precede the decompression of the required data batch, this stage does not require additional caching and cannot achieve decoupling and high parallelism between the migration process (model optimization process) of the compression stage and the encoder operation process (data decompression process).

[0344] In this embodiment of the application, if the encoder is an encoder that combines reverse encoding technology with an asymmetric digital system, the input during the compression process and the output during the decompression process will also have initial bits.

[0345] In this embodiment, if the encoder is an arithmetic encoder, the intermediate information transmitted between each step in the compression process is a range, which can be understood as multiple encoded sequences. During decompression, if the decompression process is not yet complete, the same encoded sequence is transmitted between each step, as described above. Figure 21 The second sequence is the same as the fourth sequence.

[0346] To clearly demonstrate the beneficial effects of the data compression and decompression methods (hereinafter referred to as one-shot online adaptation, OSOA) in the embodiments of this application, a comparison with existing compression methods is provided below.

[0347] Two existing algorithms that use deep generative models for lossless compression were selected: the flow model from IDF++ and the inverse autoregressive flow variational autoencoder (IAF VAE) from HiLLoC. For both models, parent models were obtained on the pre-training dataset CIFAR-10. Gradient-based optimization was used. Since both models are latent variable models, an encoder combining inverse encoding techniques and an asymmetric digital system was employed for both compression and decompression, along with the aforementioned... Figures 13 to 17 The compression process is shown.

[0348] For example, online migration compression needs to be implemented on images of size N×C×H×W, where N is the batch size, C is the number of channels in the image (3 for color images and 1 for monochrome images), and H and W are the number of pixels in the height and width dimensions of the image, respectively. The dataset used for compression consists of 131,072 images randomly selected from YFCC100M, cropped into 3*256*256 images to form dataset SET256. Then, downsampling techniques are used to obtain three datasets from SET256: 3*128*128 (SET128), 3*64*64 (SET64), and 3*32*32 (SET32).

[0349] For the above IDF++, N is selected as 3, 3, 12, 48. For the above HiLLoC, N is selected as 4, 16, 64, 256. For IDF++, each SET256 image is cropped into four images of size 3*128*128.

[0350] The effect of the parent model in this application embodiment on the compression of the dataset to be compressed by OSOA compared with other methods can be seen in Table 3, with the unit being bits per dimension.

[0351] Table 3

[0352]

[0353]

[0354] Among them, HiLLoC OSOA and IDF++OSOA are compression schemes using the compression method provided in the embodiments of this application, while HiLLoC pre-train and IDF++pre-train are models specifically trained for datasets in the prior art. It can be seen that the number of bits per dimension using the OSOA scheme is lower than that of the corresponding existing schemes. For example, when the dataset to be compressed is SET256, IDF++OSOA reduces the number of bits per dimension by 0.836 (i.e., 3.170-2.334) compared to IDF++pre-train. The lower the number of bits per dimension, the higher the compression ratio; therefore, it can be said that IDF++OSOA improves the number of bits per dimension by 0.836 compared to IDF++pre-train.

[0355] The data compression and decompression methods in the embodiments of this application have been described above. The compression and decompression devices in the embodiments of this application are described below. Please refer to [link to relevant documentation]. Figure 23 One embodiment of the compression device in this application includes:

[0356] Acquisition unit 2301 is used to acquire the first data batch and the second data batch;

[0357] The acquisition unit 2301 is also used to acquire a parent model, the parent model being used to calculate first probability distribution information, the first probability distribution information being used to represent the probability distribution of each variable value in the first data batch;

[0358] Compression unit 2302 is used to compress the first data batch to obtain a first sequence based on the first data batch, the parent model and the first encoder, wherein the first encoder corresponds to the parent model;

[0359] Update unit 2303 is used to optimize the parent model based on the first data batch to obtain the first model;

[0360] Compression unit 2302 is also used to compress the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence and the first encoder.

[0361] Optionally, the compression device also includes:

[0362] The determining unit 2304 is used to determine the first encoder corresponding to the type of the parent model based on the first association relationship. The first association relationship is used to represent the association relationship between the type of the parent model and the first encoder. The type includes a fully observed model and a latent variable model.

[0363] In this embodiment, the operations performed by each unit in the compression device are the same as described above. Figures 8 to 12 The embodiments shown are similar and will not be repeated here.

[0364] In this embodiment, on the one hand, during the compression process, the compression unit 2302 compresses the batch of data to be compressed, and the update unit 2303 optimizes the parent model. Furthermore, the optimized new model does not need to be saved, making it suitable for compressing large-scale datasets. On the other hand, as the update unit 2303 optimizes the parent model, the parent model increasingly conforms to the distribution of the batch of data to be compressed, thus increasing the compression ratio. Additionally, compared to existing technologies that require training new models specifically for the dataset to be compressed, the parent model in this embodiment is more versatile and does not require dedicated training for the dataset. During the compression process of the compression unit 2302, the update unit 2303 continuously optimizes the model. Moreover, this optimization process can be reproduced during decompression by optimizing the parent model. Therefore, by saving the parent model, the optimized model can be obtained, eliminating the need to store every model generated during the compression process and saving storage space.

[0365] Please see Figure 24 Another embodiment of the compression device in this application includes:

[0366] Acquisition unit 2401 is used to acquire the first data batch and the second data batch;

[0367] The acquisition unit 2401 is also used to acquire a parent model, the parent model being used to calculate first probability distribution information, the first probability distribution information being used to represent the probability distribution of each variable value in the first data batch;

[0368] Update unit 2402 is used to optimize the parent model based on the first data batch to obtain the first model;

[0369] Compression unit 2403 is used to compress the second data batch to obtain a first sequence based on the second data batch, the first model and the first encoder, wherein the first encoder corresponds to the parent model;

[0370] Compression unit 2403 is also used to compress the first data batch to obtain a second sequence based on the first data batch, the master model, the first sequence and the first encoder.

[0371] In this embodiment, the operations performed by each unit in the compression device are the same as described above. Figures 13 to 17 The embodiments shown are similar and will not be repeated here.

[0372] In this embodiment, it can be applied to a Last-In-First-Out (LIFO) encoder. On one hand, during the compression process, the compression unit 2403 can compress the batch of data to be compressed, and the update unit 2402 optimizes the parent model without saving the optimized new model, thus making it suitable for compressing large-scale datasets. On the other hand, during the process of the update unit 2402 optimizing the parent model, the parent model will increasingly conform to the distribution of the batch of data to be compressed, thus the compression ratio will become higher and higher. Furthermore, through the reverse adjustment of the LIFO encoder, the decompression process can obtain the data batches in sequence.

[0373] Please see Figure 25 One embodiment of the decompression device in this application includes:

[0374] Acquisition unit 2501 is used to acquire the second sequence;

[0375] Unit 2501 is also used to acquire the parent model;

[0376] Decompression unit 2502 is used to decompress the second sequence based on the parent model and the first encoder to obtain a fourth sequence and a first data batch, wherein the first encoder corresponds to the parent model;

[0377] Update unit 2503 is used to optimize the parent model based on the first data batch to obtain the first model;

[0378] The decompression unit 2502 is also used to decompress the fourth sequence based on the first model and the first encoder to obtain a second data batch.

[0379] Optionally, the decompression device may further include a determining unit 2504 and / or a merging unit 2505.

[0380] The determining unit 2504 is used to determine the first encoder corresponding to the type of the parent model based on the first association relationship. The first association relationship is used to represent the association relationship between the type of the parent model and the first encoder. The type includes a fully observed model and a latent variable model.

[0381] Merging unit 2505 is used to merge the first data batch and the second data batch to obtain an initial dataset.

[0382] In this embodiment, the operations performed by each unit in the decompression device are the same as described above. Figures 18 to 22 The embodiments shown are similar and will not be repeated here.

[0383] In this embodiment, on the one hand, during the decompression process, the decompression unit 2502 decompresses the encoded sequence, and the update unit 2503 optimizes the parent model, without needing to save the optimized new model, thus making it suitable for compressing large-scale datasets. On the other hand, the parent model in this embodiment has strong versatility and does not require special training on the dataset to be compressed. Furthermore, the optimization process can be reproduced during decompression by optimizing the parent model through the update unit 2503. Therefore, by saving the parent model, the optimized model can be obtained, without needing to store every model generated during the compression process, saving storage space.

[0384] See Figure 26 This application provides a schematic diagram of another compression device. The compression device may include a processor 2601, a memory 2602, and a communication interface 2603. The processor 2601, memory 2602, and communication interface 2603 are interconnected via lines. The memory 2602 stores program instructions and data.

[0385] The aforementioned are stored in memory 2602 Figures 8 to 12 or Figures 13 to 17 The corresponding implementation shown includes the program instructions and data for the steps executed by the compression device.

[0386] Optionally, the memory 2602 stores the data batch to be compressed (e.g., a first data batch, a second data batch, or a third data batch) and the encoded sequence (e.g., a first sequence, a second sequence, etc.).

[0387] Processor 2601, for executing the aforementioned Figures 8 to 12 or Figures 13 to 17 The steps performed by the compression device are shown in any of the embodiments illustrated.

[0388] Optionally, the first encoder may be located in or outside the processor 2601; no specific limitation is made here.

[0389] Communication interface 2603 can be used to receive and send data, and to perform the aforementioned tasks. Figures 8 to 12 or Figures 13 to 17 The steps related to acquiring, sending, and receiving in any of the embodiments shown.

[0390] In one implementation, the compression device may include, relative to Figure 26 More or fewer components are merely illustrative in this application and are not intended to limit the scope of the application.

[0391] See Figure 27 This application provides a schematic diagram of another decompression device. The decompression device may include a processor 2701, a memory 2702, and a communication interface 2703. The processor 2701, memory 2702, and communication interface 2703 are interconnected via lines. The memory 2702 stores program instructions and data.

[0392] The aforementioned are stored in memory 2702 Figures 18 to 22 In the corresponding implementation, the program instructions and data corresponding to the steps executed by the decompression device.

[0393] Optionally, the memory 2702 stores the decompressed data batches (e.g., a first data batch, a second data batch, or a third data batch) and the encoded sequences (e.g., a first sequence, a second sequence, etc.).

[0394] Processor 2701, for performing the aforementioned Figures 18 to 22 The steps performed by the decompression device are shown in any of the embodiments illustrated.

[0395] Optionally, the first encoder may be located in or outside the processor 2701; no specific limitation is made here.

[0396] Communication interface 2703 can be used to receive and send data, and to perform the aforementioned tasks. Figures 18 to 22 The steps related to acquiring, sending, and receiving in any of the embodiments shown.

[0397] In one implementation, the decompression device may include, relative to... Figure 27 More or fewer components are merely illustrative in this application and are not intended to limit the scope of the application.

[0398] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0399] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0400] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented wholly or partially through software, hardware, firmware, or any combination thereof.

[0401] When the integrated unit is implemented using software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0402] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

Claims

1. A data compression method, characterized in that, Applied to a compression device, the method includes: Obtain a first data batch and a second data batch. The data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. A master model is obtained and stored in the memory of the compression device. The master model is used to calculate first probability distribution information, which represents the probability distribution of the values ​​of each variable in the first data batch. Based on the first data batch, the parent model, and the first encoder, the first data batch is compressed to obtain a first sequence, wherein the first encoder corresponds to the parent model; The first model is obtained by updating the parent model based on the first batch of data. The second data batch is compressed based on the second data batch, the first model, the first sequence, and the first encoder to obtain the second sequence; In this case, after obtaining the second sequence, the first model is not saved.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the third batch of data; The step of compressing the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder includes: The third sequence is obtained by compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder. The second model is obtained by updating the first model based on the third batch of data. The second sequence is obtained by compressing the second data batch based on the second data batch, the second model, the third sequence, and the first encoder.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the third batch of data; The step of compressing the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder includes: Based on the third data batch, the first model, the first sequence, and the first encoder, the third data batch is compressed to obtain the third sequence; The second sequence is obtained by compressing the second data batch based on the second data batch, the first model, the third sequence, and the first encoder.

4. The method according to any one of claims 1 to 3, characterized in that, The step of updating the parent model based on the first data batch to obtain the first model includes: The first model is obtained by updating the parent model using an optimization algorithm and the first batch of data, wherein the optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning.

5. The method according to any one of claims 1 to 4, characterized in that, The parent model is obtained by training the neural network model with a batch of training data from the training dataset as input and aiming to make the loss function value less than a first threshold. The data type of the training dataset is the same as that of the dataset to be compressed. The dataset to be compressed includes the first data batch and the second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

6. The method according to any one of claims 1 to 5, characterized in that, The step of compressing the first data batch based on the first data batch, the parent model, and the first encoder to obtain the first sequence includes: The first batch of data is input into the parent model to obtain the first probability distribution information; The first sequence is obtained by compressing the first data batch based on the first probability distribution information and the first encoder.

7. The method according to any one of claims 1 to 6, characterized in that, The step of compressing the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder includes: The second data batch is input into the first model to obtain the second probability distribution information, which is used to represent the probability distribution of each variable value in the second data batch. The second sequence is obtained by compressing the second data batch based on the second probability distribution information, the first sequence, and the first encoder.

8. The method according to claim 2, characterized in that, The step of compressing the second data batch to obtain the second sequence based on the second data batch, the second model, the third sequence, and the first encoder includes: The second data batch is input into the second model to obtain the second probability distribution information, which is used to represent the probability distribution of each variable value in the second data batch. The second sequence is obtained by compressing the second data batch based on the second probability distribution information, the third sequence, and the first encoder.

9. The method according to claim 3, characterized in that, The step of compressing the second data batch to obtain the second sequence based on the second data batch, the first model, the third sequence, and the first encoder includes: The second data batch is input into the first model to obtain the second probability distribution information, which is used to represent the probability distribution of each variable value in the second data batch. The second sequence is obtained by compressing the second data batch based on the second probability distribution information, the third sequence, and the first encoder.

10. The method according to claim 8 or 9, characterized in that, The process of compressing the third data batch to obtain the third sequence based on the third data batch, the first model, the first sequence, and the first encoder includes: The third data batch is input into the first model to obtain the third probability distribution information, which is used to represent the probability distribution of each variable value in the third data batch. The third sequence is obtained by compressing the third data batch based on the third probability distribution information, the first sequence, and the first encoder.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The first encoder corresponding to the type of the parent model is determined based on the first association relationship. The first association relationship is used to represent the association relationship between the type of the parent model and the first encoder. The type includes a fully observed model and a latent variable model.

12. The method according to any one of claims 1 to 10, characterized in that, The acquisition of the parent model includes: The parent model is obtained based on the data type of the dataset to be compressed, including image data type and sequence data type.

13. The method according to any one of claims 1 to 12, characterized in that, The acquisition of the first data batch and the second data batch includes: Obtain the dataset to be compressed; The dataset to be compressed is split into the first data batch and the second data batch.

14. A data compression method, characterized in that, Applied to a compression device, the method includes: Obtain a first data batch and a second data batch. The data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. A master model is obtained and stored in the memory of the compression device. The master model is used to calculate first probability distribution information, which represents the probability distribution of the values ​​of each variable in the first data batch. The first model is obtained by updating the parent model based on the first batch of data. A first sequence is obtained by compressing the second data batch, the first model, and the first encoder, wherein the first encoder corresponds to the parent model; wherein, after obtaining the first sequence, the first model is not saved; The second sequence is obtained by compressing the first data batch, the parent model, the first sequence, and the first encoder.

15. The method according to claim 14, characterized in that, The description also includes: Obtain the third batch of data; The step of compressing the second data batch to obtain the first sequence based on the second data batch, the first model, and the first encoder includes: The second model is obtained by updating the first model based on the third batch of data. Based on the second data batch, the second model, and the first encoder, the second data batch is compressed to obtain the first sequence; The step of compressing the first data batch to obtain the second sequence based on the first data batch, the parent model, the first sequence, and the first encoder includes: The third sequence is obtained by compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder. The second sequence is obtained by compressing the first data batch, the parent model, the third sequence, and the first encoder based on the first data batch.

16. The method according to claim 14, characterized in that, The description also includes: Obtain the third batch of data; The step of compressing the first data batch to obtain the second sequence based on the first data batch, the parent model, the first sequence, and the first encoder includes: The third sequence is obtained by compressing the third data batch based on the third data batch, the first model, the first sequence, and the first encoder. The second sequence is obtained by compressing the first data batch, the parent model, the third sequence, and the first encoder based on the first data batch.

17. A data decompression method, characterized in that, Applied to decompression equipment, the method includes: Obtain the second sequence; Obtain the master model, which is stored in the memory of the decompression device; Based on the parent model and the first encoder, the second sequence is decompressed to obtain the fourth sequence and the first data batch. The first encoder corresponds to the parent model. The data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The first model is obtained by updating the parent model based on the first batch of data. Based on the first model and the first encoder, the fourth sequence is decompressed to obtain a second data batch. The data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. Specifically, after obtaining the second batch of data, the first model is not saved.

18. The method according to claim 17, characterized in that, The process of decompressing the fourth sequence based on the first model and the first encoder to obtain the second data batch includes: Based on the first model and the first encoder, the fourth sequence is decompressed to obtain the fifth sequence and the third data batch; The second model is obtained by updating the first model based on the third batch of data. The second data batch is obtained by decompressing the fifth sequence based on the second model and the first encoder.

19. The method according to claim 17, characterized in that, The process of decompressing the fourth sequence based on the first model and the first encoder to obtain the second data batch includes: Based on the first model and the first encoder, the fourth sequence is decompressed to obtain the fifth sequence and the third data batch; The second data batch is obtained by decompressing the fifth sequence based on the first model and the first encoder.

20. The method according to any one of claims 17 to 19, characterized in that, The step of updating the parent model based on the first data batch to obtain the first model includes: The first model is obtained by updating the parent model using an optimization algorithm and the first batch of data, wherein the optimization algorithm includes gradient-based optimization algorithms, meta-learning, or reinforcement learning.

21. The method according to any one of claims 17 to 20, characterized in that, The parent model is obtained by training the neural network model with a batch of training data from the training dataset as input and aiming to make the loss function value less than a first threshold. The data type of the training dataset is the same as that of the dataset to be compressed. The dataset to be compressed includes the first data batch and the second data batch. The loss function is used to indicate the difference between the probability distribution information output by the neural network model and the actual probability distribution information of the variable values ​​in the training data batch.

22. The method according to any one of claims 17 to 21, characterized in that, The process of decompressing the second sequence based on the parent model and the first encoder to obtain the fourth sequence and the first data batch includes: Based on the second sequence, the parent model, and the first encoder, the second sequence is decompressed to obtain the fourth sequence and the first data batch; The process of decompressing the fourth sequence based on the first model and the first encoder to obtain the second data batch includes: The second data batch is obtained by decompressing the fourth sequence based on the fourth sequence, the first model, and the first encoder.

23. The method according to claim 22, characterized in that, The process of decompressing the second sequence based on the second sequence, the parent model, and the first encoder to obtain the fourth sequence and the first data batch includes: The first probability is obtained based on the parent model; Based on the first probability and the first encoder decompressing the second sequence, a portion of data is obtained; The second probability is obtained by inputting the aforementioned portion of the data into the parent model. Based on the second probability and the first encoder decompressing the second sequence to obtain another part of the data and the fourth sequence; The first data batch is obtained based on the aforementioned portion of data and the aforementioned portion of data.

24. The method according to any one of claims 17 to 21, characterized in that, The process of decompressing the second sequence based on the parent model and the first encoder to obtain the fourth sequence and the first data batch includes: Based on the parent model, a first probability distribution information is obtained, which is used to represent the probability distribution of each variable value in the first data batch. Based on the first probability distribution information and the first encoder decompressing the second sequence, the fourth sequence and the first data batch are obtained.

25. The method according to any one of claims 17 to 24, characterized in that, The process of decompressing the fourth sequence based on the first model and the first encoder to obtain the second data batch includes: Based on the first model, a second probability distribution information is obtained, which is used to represent the probability distribution of each variable value in the second data batch. The second data batch is obtained based on the second probability distribution information and the first encoder decompressing the fourth sequence.

26. The method according to any one of claims 17 to 25, characterized in that, The method further includes: The first encoder corresponding to the type of the parent model is determined based on the first association relationship. The first association relationship is used to represent the association relationship between the type of the parent model and the first encoder. The type includes a fully observed model and a latent variable model.

27. The method according to any one of claims 17 to 26, characterized in that, The method further includes: The first data batch and the second data batch are merged to obtain the initial dataset.

28. A compression device, characterized in that, The compression device includes: The acquisition unit is used to acquire a first data batch and a second data batch. The data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The acquisition unit is further configured to acquire a master model, which is stored in the memory of the compression device. The master model is used to calculate first probability distribution information, which represents the probability distribution of each variable value in the first data batch. A compression unit is used to compress the first data batch based on the first data batch, the parent model, and the first encoder to obtain a first sequence, wherein the first encoder corresponds to the parent model; An update unit is used to update the parent model based on the first data batch to obtain a first model; The compression unit is further configured to compress the second data batch to obtain the second sequence based on the second data batch, the first model, the first sequence, and the first encoder; In this case, after obtaining the second sequence, the first model is not saved.

29. A compression device, characterized in that, The compression device includes: The acquisition unit is used to acquire a first data batch and a second data batch. The data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. The acquisition unit is further configured to acquire a master model, which is stored in the memory of the compression device. The master model is used to calculate first probability distribution information, which represents the probability distribution of each variable value in the first data batch. An update unit is used to update the parent model based on the first data batch to obtain a first model; A compression unit is configured to compress the second data batch based on the second data batch, the first model, and the first encoder to obtain a first sequence, wherein the first encoder corresponds to the parent model; and wherein, after obtaining the first sequence, the first model is not saved. The compression unit is further configured to compress the first data batch to obtain a second sequence based on the first data batch, the parent model, the first sequence, and the first encoder.

30. A decompression device, characterized in that, The decompression device includes: The acquisition unit is used to acquire the second sequence; The acquisition unit is used to acquire the master model, which is stored in the memory of the decompression device; The decompression unit is used to decompress the second sequence based on the parent model and the first encoder to obtain a fourth sequence and a first data batch. The first encoder corresponds to the parent model, and the data type of the first data batch includes at least one of the following: image data type, video data type, audio data type, and text data type. An update unit is used to update the parent model based on the first data batch to obtain a first model; The decompression unit is further configured to decompress the fourth sequence based on the first model and the first encoder to obtain a second data batch, wherein the data type of the second data batch includes at least one of the following: image data type, video data type, audio data type, and text data type; Specifically, after obtaining the second batch of data, the first model is not saved.

31. A compression device, characterized in that, The method includes a processor coupled to a memory for storing computer programs or instructions, the processor for executing the computer programs or instructions in the memory such that the method of any one of claims 1 to 13 is performed, or the method of any one of claims 14 to 16 is performed.

32. A decompression device, characterized in that, The method includes a processor coupled to a memory for storing computer programs or instructions, the processor for executing the computer programs or instructions in the memory such that the method of any one of claims 17 to 27 is performed.

33. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 13, or cause the computer to perform the method as described in any one of claims 14 to 16, or cause the computer to perform the method as described in any one of claims 17 to 27.

34. A computer program product, characterized in that, When the computer program product is executed on a computer, it causes the computer to perform the method as described in any one of claims 1 to 13, or causes the computer to perform the method as described in any one of claims 14 to 16, or causes the computer to perform the method as described in any one of claims 17 to 27.