Data compression / decompression system, data compression / decompression method, and data compression / decompression program

Abstract

It achieves both high compression ratio and fast processing speed. [Solution] The parallel processing device of the data compression / decompression system divides the original data into multiple data units in predetermined units and stores the multiple data units in the memory. The parallel processing device then generates probability distributions for the multiple data units in parallel using an entropy estimater based on a neural network that predicts the probability distributions of the multiple data units, and generates a cumulative distribution function for the probability distributions corresponding to the multiple data units in parallel. The parallel processing device then performs entropy coding processing in parallel on the multiple data units based on the probability distributions and cumulative distribution functions to generate multiple coded data units, and creates compressed data from the multiple coded data units.

Description

Technical Field

[0001] The present invention generally relates to a data compression and decompression system, for example, compression and decompression processing.

Background Art

[0002] As a storage system for data compression and decompression, for example, there is a technique disclosed in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to improve productivity in the industry, expectations for IoT (Internet of Things) are increasing, and IoT data (for example, data acquired from various devices) continues to increase. For example, the multi-dimensional data (for example, images, videos, etc.) generated by IoT devices is enormous, and it is considered that it is required to transfer and store the enormous multi-dimensional data with high efficiency, and data compression technology is one of the approaches towards solving this.

[0005] In data compression technology, by using an advanced model such as a neural network, it is expected to achieve a high compression ratio, but there is a concern that the processing speed of compression or decompression will decrease. The coexistence of a high compression ratio and a high processing speed is an issue.

[0006] The above problems can also occur in fields other than IoT.

Means for Solving the Problems

[0007] In parallel processing devices, data compression and decompression are performed using advanced models such as neural networks. Data is divided into predetermined units before encoding, and the encoding process is performed in parallel. [Effects of the Invention]

[0008] It can achieve both high compression ratio and high processing speed. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example configuration of a data compression / decompression system according to Embodiment 1. [Figure 2] A diagram showing an example configuration of the compressor and extender according to Embodiment 1. [Figure 3] A diagram showing an example of the data structure of compressed data compressed by a compressor. [Figure 4] A diagram to explain FM tiling. [Figure 5] A flowchart showing an example of compression processing according to Embodiment 1. [Figure 6] A flowchart showing a detailed example of quantization processing (during inference). [Figure 7] A flowchart showing an example of quantization processing (during training). [Figure 8] A flowchart illustrating the details of the entropy coding process. [Figure 9] A diagram illustrating an example of entropy estimator processing. [Figure 10] A diagram showing an example algorithm for the CDF creation process. [Figure 11] A diagram showing an example algorithm for entropy encoder step processing. [Figure 12] A diagram illustrating an example of the process for creating compressed data. [Figure 13] A flowchart showing an example of the extension process according to Embodiment 1. [Figure 14] A flowchart illustrating a detailed example of the entropy decoding process. [Figure 15]Figure showing an example algorithm of entropy decoder step processing. [Figure 16] Figure showing a configuration example of a part of an entropy estimator and related compressor and expander according to Embodiment 2. [Figure 17] Figure showing a configuration example of a data compression and expansion system according to Embodiment 3. [Figure 18] Figure showing an example of compression processing according to Embodiment 3. [Figure 19] Figure for explaining an example of processing of an entropy estimator according to Embodiment 3.

Mode for Carrying Out the Invention

[0010] In the following description, the "interface device" may be one or more communication interface devices. The one or more communication interface devices may be one or more of the same type of communication interface devices (for example, one or more NICs (Network Interface Cards)). Or they may be two or more different types of communication interface devices (for example, NIC and HBA (Host Bus Adapter)).

[0011] Also, in the following description, the "memory" is one or more memory devices, and typically may be a main memory device. At least one of the memory devices in the memory may be a volatile memory device or a non-volatile memory device.

[0012] Also, in the following description, the "persistent storage device" is one or more persistent storage devices. The persistent storage device is typically a non-volatile storage device (for example, an auxiliary storage device), and specifically, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0013] Also, in the following description, the "storage device" may be a physical storage device such as a persistent storage device or a logical storage device associated with a physical storage device.

[0014] Furthermore, in the following explanation, "processor" refers to one or more processor devices. At least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit). At least one processor device may be single-core or multi-core. At least one processor device may be a processor core.

[0015] Furthermore, in the following explanation, "parallel processing device" is an example of a device capable of parallel processing. A parallel processing device may be implemented by a CPU having multiple physical or virtual cores, but typically it may be a hardware circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit).

[0016] Furthermore, the term "xxx table" is sometimes used below to describe information from which an output is obtained for a given input. This information can be data with any structure, or a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information." Also, in the following explanation, the structure of each table is just an example; one table may be divided into two or more tables, or all or part of two or more tables may be combined into one table.

[0017] Furthermore, the functions of the components of the compressor or expander (specifically, for example, encoder filters, FM (Feature Map) tylers, FM untylers, entropy encoders, entropy decoders, quantizers, and entropy estimators) will be described below. In this embodiment, these functions are implemented by a parallel processing device, but some functions may be implemented by a computer program executed by a processor. The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0018] Furthermore, in the following explanation, a "dataset" is a single logical block of electronic data as viewed from the perspective of a program such as an application program, and may be, for example, a record, a file, a key-value pair, or a tuple.

[0019] Furthermore, in the following explanation, when describing similar elements without distinction, the common part of the reference code may be used, and when describing similar elements without distinction, the reference code may be used. For example, when computers are not distinguished, they may be referred to as "computer 101," and when computer 101 is distinguished, they may be referred to as "computer 101C," "computer 101D," etc. Also, a system containing one or more computers 101 may be called a "computer system." A "computer" may be a physical computer, such as an on-premise type. Alternatively, a "computer" may be a cloud-based computer (for example, a device implemented on a system with multiple types of computing resources (e.g., processors and memory)). Alternatively, a "computer" may be a mobile device (for example, a smartphone or a laptop personal computer).

[0020] Several embodiments of the present invention will be described below. In any of the following embodiments, a lossy compression configuration may be adopted, or a lossless compression configuration may be adopted, for example, by not using filters or quantizers, or by using lossless filters.

[0021] The following embodiments show the computer of the present invention applied to a storage device, but are not limited to this; it can also be applied to devices that transmit and receive data, such as communication devices, without a persistent memory device.

[0022] [Embodiment 1] Figure 1 shows an example of the configuration of the data compression / decompression system S according to Embodiment 1.

[0023] The data compression and decompression system S comprises computers 101C and 101D. Computers 101C and 101D communicate via a network 150 (e.g., the Internet, a WAN (Wide Area Network), or a LAN (Local Area Network)). Computer 101 is, for example, a storage device. Computer 101C has a compressor 70, and computer 101D has a decompressor 71. A single computer 101 may have both the compressor 70 and the decompressor 71, and data compression and decompression may be completed within a single computer 101.

[0024] The hardware configurations of computers 101C and 101D may be the same or different. In this embodiment, for the sake of simplicity, the hardware configurations of computers 101C and 101D are the same. The hardware configuration of computer 101 will be explained using computer 101C as an example.

[0025] Computer 101C comprises interface devices 5C1 and 5C2 (an example of interface devices), persistent storage device 54C, memory 52C, processor 53C, parallel processing device 61C, and an internal bus 10C connecting them. Interface devices 5C1 and 5C2, persistent storage device 54C, memory 52C, and parallel processing device 61C are communicated to processor 53C, for example, via the internal bus 10C.

[0026] Interface device 5C1 is connected to network 150. Interface device 5C2 is connected to input device 40. Input device 40 inputs data to computer 101C. Input device 40 may be a sensor device (e.g., an optical camera or gravity sensor), a portable storage medium, or another computer.

[0027] The parallel processing device 61C receives the data to be compressed from the input device 40 via the interface device 5C2, either via the processor 53C or not. The parallel processing device 61C has memory 63C and multiple cores 62C. The parallel processing device 61C is implemented as a compressor 70. The compressor 70 (parallel processing device 61C) compresses the data to be compressed in parallel. This allows the data to be compressed at high speed. The compressed data is output from the parallel processing device 61C and stored in the persistent storage device 54C either via the processor 53C or not.

[0028] Memory 52C stores computer programs executed by processor 53C, as well as data input and output by processor 53C.

[0029] The processor 53C executes at least a portion of the processing performed by the computer 101C by reading and executing a program from the memory 52C. For example, the processor 53C reads compressed data (compressed data) stored in the persistent storage device 54C from the persistent storage device 54C and transmits the compressed data to the computer 101D via the interface device 5C1.

[0030] The differences between computer 101D and computer 101C are as follows: Computer 101D has a user terminal 80 (e.g., a personal computer or smartphone) connected to the interface device 5D2. In computer 101D, compressed data is received via the interface device 5D1, stored in the persistent storage device 54D and / or memory 52D (or without being stored in the persistent storage device 54D), and then input to the parallel processing device 61D. The parallel processing device 61D is implemented as a decompressor 71. The decompressor 71 (parallel processing device 61D) decompresses the compressed data in parallel. This allows for high-speed decompression of the compressed data. The decompressed data is transmitted to the user terminal 80 via the interface device 5D2.

[0031] The type of data to be compressed does not matter. In this embodiment, the data to be compressed is image data representing an image (still image). The image data is an example of N-dimensional (N is an integer greater than or equal to 1) tensor data. The image data to be compressed is input to the compressor 70. The image data input to the compressor 70 is sometimes called the "original image data". The compressor 70 irreversibly compresses the original image data, and the decompressor 71 decompresses the irreversibly compressed image data.

[0032] For example, if computers 101C and 101D are communication devices, the persistent storage devices 54 and 54D can be omitted. In this case, the original image data input to computer 101C via input device 40 is compressed by compressor 70 and then transmitted to computer 101D. Computer 101D decompresses the compressed data received from computer 101C using decompressor 71 and then outputs it to user terminal 80.

[0033] Figure 2 shows an example of the configuration of the compressor 70 and extender 71 according to Embodiment 1.

[0034] The compressor 70 comprises a first encoder filter 201, an FM (Feature Map) leveler 202, a second encoder filter 203, a quantizer 204, an entropy encoder 205, and an entropy estimator 210. These components are implemented on a parallel processing device 61C. The entropy estimator 210 can construct the same model and output the same predicted probability for the same symbol by providing the same distribution type, mean, and scale parameters, even if different entropy estimators are used.

[0035] The extender 71 comprises a first decoder filter 211, an FM anti-tyler 212, a second decoder filter 213, an entropy decoder 215, and an entropy estimator 210. These components are implemented on a parallel processing device 61D.

[0036] In the parallel processing device 61, the data stored in memory 63 is divided, and tensor operations are performed by multiple cores 62. This speeds up the compression (and decompression) process.

[0037] In this embodiment, each of the first encoder filter 201, the second encoder filter 203, the entropy encoder 205, the entropy estimator 210, the first decoder filter 211, the second decoder filter 213, and the entropy decoder 215 is configured to include a CNN (Convolutional Neural Network). Specifically, for example, it is configured from a multi-layer network including a Residual Block. For example, the first encoder filter 201 and the second encoder filter 203 may be networks that shrink in the vertical and horizontal directions, while the first decoder filter 211 and the second decoder filter 213 may be networks that expand in the vertical and horizontal directions.

[0038] The first encoder filter 201, the second encoder filter 203, the entropy encoder 205, the entropy estimator 210, the first decoder filter 211, the second decoder filter 213, and the entropy decoder 215. At least one of the entropy models may be a machine learning model other than a simple CNN (typically a deep learning-based model), such as a reversible generative model like GMM (Gaussian Mixture Models), HMM (Hidden Markov Model), SCFG (Stochastic Context-Free Grammar), GAN (Generative Adversarial Nets), VAE (Variational Auto Encoder), Flow, or genetic programming. Model compression such as Mimic Model may also be applied to reduce the information content of the model. Furthermore, commonly used compression processes such as DCT (Discrete Cosine Transform) and Wavelet Transform may be used. In addition, prediction processes such as Optical Flow may be used.

[0039] Figure 3 shows an example of the data structure of compressed data 300 compressed by the compressor 70. Each value shown in Figure 3 is an example.

[0040] In this embodiment, the uncompressed image data input to the compressor 70, the decompressed image data output from the decompressor 71, the first encoder filter 201, the second encoder filter 203, the entropy estimator 210, the first decoder filter 211, and the second decoder filter 213 are generalized as four-dimensional tensor data. Each dimension of the tensor data consists of a batch, number of channels, height (vertical width), and width (horizontal width).

[0041] "Original image height H" refers to the height (vertical width) of the original image. "Original image width W" refers to the width (horizontal width) of the original image. "FM batch size B" refers to the number of parallel quantized feature maps (FMs) output by the second encoder filter 203 and subsequently processed by the quantizer 204, and is a natural number. B=1 means no parallel processing. "FM channel size C3" refers to the number of channels in the quantized feature map. Here, for the uncompressed image data input to the compressor 70 and the decompressed image data output from the decompressor 71, for example, in the case of an RGB image, the batch size is 1, the number of channels is 3 (denoted as C1), the height is H, and the width is W.

[0042] The compressor 70 encodes the original image data and generates compressed data 300 in which configuration information such as "height of original image (H)", "width of original image (W)", "number of FM batches (B)", "number of FM channels (C3)", "number of vertical divisions (Dh)", "number of horizontal divisions (Dw)", "division height (Th)", and "division width (Tw)" are recorded in association with "map information", "encoded data", and "ancillary information of the original image", as shown in Figure 3.

[0043] Of these, "original image height (H)", "original image width (W)", and "original image supplementary information" are information inherited from the original image data. In addition, "FM batch size (B)", "FM channel size (C3)", "number of vertical divisions (Dh)", "number of horizontal divisions (Dw)", "division height (Th)", and "division width (Tw)" are information added by the FM typer 202.

[0044] "Map information" exists in quantities equal to the number of "encoded data" streams (R = B × C), and is information (an N-dimensional vector) representing the position and data length of each "encoded data" stream within the compressed data. "Encoded data" is binary data of length M, obtained by combining the unused portions of each of the R entropy-encoded data streams.

[0045] According to this embodiment, the compressor 70's filter is divided into a first encoder filter 201 (pre-encoder filter) and a second encoder filter 203 (post-encoder filter), with the FM leveler 202 located between the first encoder filter 201 and the second encoder filter 203.

[0046] Referring to Figure 4, we will now describe the FM tiling performed by the FM tiler 202. Figure 4 is a diagram illustrating FM tiling.

[0047] The original image data of a four-dimensional tensor is input to the first encoder filter 201. The first encoder filter 201 prepares a first pre-encoder filter image with channel count = C1 in memory 63C. The "C1 first pre-encoder filter image" may be a C1 channel image (C1 original image) represented by the original image data. Alternatively, the "C1 first pre-encoder filter image" may be a C1 image (each with a predetermined height and width) resulting from padding processing applied to each of the C1 channel images of the original image data, as shown in Figure 4(a). In the padding process, for example, if the division height is Th, the division width is Tw, the number of divisions vertically is Dh, and the number of divisions horizontally is Dw (i.e., the batch size B of FM = Dh × Dw), data padding is applied to the original image so that the size of the original image becomes a multiple of the division height Th × division width Tw. Hereafter, the size of the image after padding will be H1 vertically and W1 horizontally.

[0048] In Figure 4(b), the first encoder filter 201 performs the first encoder filter on image data representing C1 images before the first encoder filter. Specifically, the first encoder filter 201 takes C1 images before the first encoder filter as input and outputs a feature map (first FM) representing C2 images after the first encoder filter, as shown in Figure 4(c), without changing the dimensions and batch size of the image data. In other words, the image data representing C1 images before the first encoder filter is converted into an FM representing C2 tensors after the first encoder filter by the CNN of the first encoder filter 201. Typically, C2 is greater than C1, and the height H2 and width W2 of the first FM are smaller than the height H1 and width W1 of the images before the first encoder filter. For example, in Figure 4, the dimensions of the images before the first encoder filter are 1 × C1 × H1 × W1, and the dimensions of the first FM are 1 × C2 × (H1 / 2) × (W1 / 2).

[0049] The first FM is input to the FM tiler 202. Without changing the dimensions of the input image data, the FM tiler 202 divides each of the C2 first encoder-filtered images represented by the image data in the batch direction according to the division column number Dh and division column number Dw (i.e., the batch size of the FM B = Dh × Dw). In other words, each of the C2 first encoder-filtered tensors is divided into B images in a tile-like manner (for example, B is an integer greater than or equal to 2). Therefore, the FM output from the FM tiler 202 (tiled FM) has dimensions of B × C² × (Th / 2) × (Tw / 2), as shown in Figure 4(d).

[0050] In Figure 4(e), the second encoder filter 203 performs second encoder filtering on the tiled FM. Specifically, the second encoder filter 203 takes the tiled FM output from the FM tiler 202 as input and performs second encoder filtering in batches of B on the set of C2 tensors represented by the data, using multiple cores 62C in parallel. The data resulting from the second encoder filtering (multiple divided feature maps) is output from the second encoder filter 203. Subsequently, in the quantizer 204, entropy encoder 205, entropy estimator 210, entropy decoder 215, and second decoder filter 213, processing of B batches is performed in parallel using multiple cores 62C. In this example, the number of batches is described as being divided from a single original image, but as will be shown later in Figure 5, tiles can be collected from multiple images and processed in even more batches to improve the degree of parallelism.

[0051] The data output from the second encoder filter 203 is input to the quantizer 204. In Figure 4(f), the quantizer 204 generates a quantized feature map (FM) of the data. The quantizer 204 reduces the data by converting any floating-point numbers in the data to integers. In this way, a quantized feature map is generated. In other words, the data output from the quantizer 204 is, for example, an array of integer values. Furthermore, in the following description, the shape of the 4-dimensional tensor data of the quantized feature map in this embodiment will be represented as [B × C3 × H3 × W3]. For example, C3 = 32, H3 = Th / 16, and W3 = Tw / 16 may also be used. C3 may be increased to 64, C3 = 128, etc., or decreased depending on the required bitrate.

[0052] The data output from the quantizer 204 (quantized feature map) is input to the entropy estimator 210. The entropy estimator 210 uses the feature map to predict the probability distribution f of each symbol, which is the data unit for compression and expansion, using predictions by autoregressive models, other information stored for probability prediction, or a combination of these, and calculates the cumulative distribution function (CDF) of the probability distribution f. The probability distribution f and the cumulative distribution function CDF are called the predicted probability (CDF,f).

[0053] The entropy encoder 205 calculates a function that compares probabilities such as cross-entropy based on the predicted probability of each symbol and the actual occurrence probability of each symbol calculated directly from the feature map. The cross-entropy between the predicted probability of a symbol and the actual occurrence probability of a symbol becomes the amount of information encoded by the entropy encoder 205. Therefore, the prediction accuracy of the entropy estimator 210 affects the compression ratio.

[0054] The data output from the quantizer 204 and the data output from the entropy estimator 210 are input to the entropy encoder 205. The entropy encoder 205 encodes symbols based on the symbols derived from the input feature map and the predicted probabilities from the entropy estimator 210. The encoded data (bitstream data) is combined, excluding any unused portions, and output as compressed data 300. The output compressed data 300 is output from the parallel processing device 61C to the processor 53C. The compressed data 300 may be stored in the persistent memory 54C by the processor 53C, or it may be transferred to the computer 101D and stored in the persistent memory 54D by the processor 53D.

[0055] Decompression is performed in the reverse order of compression of the original image data. That is, the compressed data 300 output from the entropy encoder 205 and the predicted probability output from the entropy estimator 210 are input to the entropy decoder 215. Based on the compressed data 300 and the predicted probability, the entropy decoder 215 outputs multiple decoded data streams R=(B×C3). The decoded data is also input to the entropy estimator 210 and used for symbol prediction. This is because, in the Simple Lined Causal CNN described later in this embodiment, decoding iterations are performed for each symbol group of a predetermined unit (number of divisions Dh × number of divisions Dw × number of FM channels C3 × width of the quantized feature map W3), and the entropy estimator 210 sequentially generates the predicted probability of the current symbol group using the previously decoded symbol group as input.

[0056] Multiple decoded data, decoded by the entropy decoder 215, are input to the second decoder filter 213. The second decoder filter 213 performs a second decoder filter process on the multiple decoded data. Specifically, the second decoder filter 213 takes a tensor of size (B × C3 × H3 × W3) output from the entropy decoder 215 as input, performs a batch size B filtering (second decoder filter process) in parallel using multiple cores 62C, and converts it into a tensor of size (B × C2 × (Th / 2) × (Tw / 2)).

[0057] The FM untyler 212 unifies the batch dimension of the tiled data according to the division column number Dh and division row number Dw (i.e., the batch size B of the FM = Dh × Dw). In other words, for each of the C2 second decoded tensors, it unifies the B batches of tiled tensors into a single tensor. Therefore, the size of the tensor data output from the FM untyler 212 is, for example, 1 × C2 × H2 × W2. This will be referred to as untiled FM from now on.

[0058] The first decoder filter 211 performs the first decoder filtering process on the untile FM. Specifically, the first decoder filter 211 takes the untile FM as input and converts it into output image data representing C1 images after the first decoder filtering process, and outputs it.

[0059] In this embodiment, since batch splitting is performed by the FM tyra 202 after the first encoder filter processing is performed by the first encoder filter 201, the receptive fields of the neural network are not completely separated compared to conventional techniques that perform compression processing after splitting the image into batches. Therefore, it is expected that artifacts at the batch splitting locations will be reduced.

[0060] In this embodiment, the tiling size (division width) is fixed, but it may be optimized by providing several types of division widths, changing the division width according to the characteristics of each region of the input image, and parallelizing the processing for each type of division width. In that case, the division width is associated with the image region and recorded in the compressed data.

[0061] Figure 5 is a flowchart illustrating an example of compression processing according to Embodiment 1. The following describes an example where computer 101C performs the compression processing, but computer 101D can also perform the same compression processing by equipping itself with a compressor 70. The compression processing is executed by processor 53C, parallel processing device 61C, and compressor 70 implemented on parallel processing device 61C, for example, when data to be compressed is input to computer 101C.

[0062] First, in step S101, the processor 53C determines whether or not it has processed all the images to be compressed. If the processor 53C has processed all the images to be compressed (step S101 YES), it terminates the compression process; if it has not processed all the images to be compressed (step S101 NO), it moves the process to step S102.

[0063] In step S102, the processor 53C acquires the next image data to be compressed and stores it in the memory 63C of the parallel processing device 61C. Next, in step S103, the core 62C of the parallel processing device 61C performs padding (see Figure 4) on the image data to be compressed stored in the memory 63C. Padding is optional.

[0064] Next, in step S104, the first encoder filter 201 performs the first encoder filter processing on the image data to be compressed stored in memory 63C to generate FM. Next, in step S105, the FM tiler 202 performs FM tiling, dividing the FM generated in step S104 into tiles (division height Th, division width Tw) of a predetermined batch size B (number of divisions vertically Dh × number of divisions horizontally Dw), and stores the generated tiles (multiple divided feature maps) in a queue in the compressor 70.

[0065] Next, in step S106, the processor 53C determines whether there are tiles on the queue that are equal to or greater than a predetermined threshold, or whether all images to be compressed have been processed. If there are tiles on the queue that are equal to or greater than a predetermined threshold, or if all images to be compressed have been processed (step S106YES), the processor 53C moves the process to step S107. If there are no tiles on the queue that are equal to or greater than a predetermined threshold, and if all images to be compressed have not been processed (step S106NO), the process returns to step S101.

[0066] From this point onward, steps S107 to S111 are processed in parallel, with the maximum number of parallel processes for the data stream being Bm (batch size) × C (channel size).

[0067] In step S107, the second encoder filter 203 takes tiles from the queue and creates batches of batch size Bm based on the predetermined threshold mentioned above. Here, the processing of multiple input images can be grouped together as batches and executed in parallel. Next, in step S108, the second encoder filter 203 applies the second encoder filter processing to the batches created in step S107. Next, in step S109, the quantizer 204 performs quantization processing (during inference) on the batches that have undergone the second encoder filter processing in step S108. Details of step S109 will be described later with reference to Figure 6.

[0068] Next, in step S110, the entropy encoder 205 performs entropy coding on the batch that underwent quantization in step S110. Details of step S110 will be described later with reference to Figure 8.

[0069] Next, in step S111, the entropy encoder 205 performs the compressed data creation process. Details of the compressed data creation process will be described later with reference to Figure 12. When step S111 is completed, the processor 53C returns to step S106.

[0070] Figure 6 is a flowchart showing a detailed example of the quantization process (during inference) in step S109 of Figure 5. Generally, if a quantizer simply rounds the values, the range of the output values ​​becomes indeterminate. If the range of the output values ​​is indeterminate, it becomes difficult to calculate the CDF using tensor operations in the subsequent entropy estimator 210. Therefore, the quantizer 204 clamps its output to restrict it to 32 symbols in a predetermined range of -15 to 16, for example.

[0071] First, in step S109a, the quantizer 204 clamps the FM value to a predetermined range, for example, -15 to 16. Next, in step S109b, the quantizer 204 rounds the FM value.

[0072] Figure 7 is a flowchart showing an example of quantization processing (during training). The quantizer 204 clamps the output of the compressor 70 from the training stage of the model to limit it to 32 symbols in a predetermined range, for example, -15 to 16. The predetermined range of clamping during training and the predetermined range of clamping during inference are the same.

[0073] First, in step S201, the core 62C of the parallel processing device 61C clamps the value of the FM to be learned to a predetermined range, for example, -15 to 16. Next, in step S202, the core 62C generates a random number between -0.5 and 0.5. Then, in step S203, the core 62C adds the random number generated in step S202 to the value of the FM to be learned.

[0074] Figure 8 is a flowchart showing a detailed example of the entropy coding process in step S110 of Figure 5. In the entropy coding process, multiple images are taken as input, and the tiles to be processed are queued, allowing the process to be performed with a predetermined multiplicity even for small images, and also allowing processing of large images in predetermined units.

[0075] First, in step S110a, the entropy estimator 210 performs entropy estimator processing. Details of the entropy estimator processing will be described later with reference to Figure 9. Next, in step S110b, the entropy estimator 210 performs CDF creation processing. Details of the CDF creation processing will be described later with reference to Figure 10.

[0076] Next, in step S110c, the entropy encoder 205 performs the entropy encoder step processing. Details of the entropy encoder step processing will be described later with reference to Figure 11. Also, since this embodiment uses Tensor ANS, which will be described later, encoding is performed in the reverse order of the entropy decoding process shown in Figure 14. For this reason, in steps S110a and S110b, unlike the sequential decoding process shown in Figure 14, the CDFs to be encoded are calculated all at once in advance.

[0077] Next, in step S110d, the entropy encoder 205 determines whether all symbols have been encoded or not. If all symbols have been encoded (step S110d YES), the entropy encoder 205 moves the process to step S110e, and if not all symbols have been encoded (step S110d NO), it returns the process to step S110c. In step S110c, to which the process has returned, the entropy encoder 205 selects the next group of symbols and executes the entropy encoder step process.

[0078] In step S110e, the entropy encoder 205 performs a renormalization process a predetermined number of times to renormalize the data remaining in the queue.

[0079] Figure 9 shows an overview of an example of the entropy estimator processing in step S110a of Figure 8. Figure 9 shows a comparison between the entropy estimator processing of this embodiment using Simple Lined Causal CNN and a conventional method using Pixel CNN. In this embodiment, the entropy prediction by CNN for encoding is performed in parallel horizontally by appropriately determining the weights of the CNN kernel, thereby eliminating computational redundancy and achieving high-speed processing.

[0080] As shown in Figure 9(a), in conventional entropy estimator processing (Pixel CNN), the weights of adjacent pixels in both the vertical and horizontal directions of the target pixel are used for prediction within the CNN kernel, and the previous prediction result is used for the current prediction. Therefore, it is necessary to repeat the prediction iteration for each process.

[0081] On the other hand, as shown in Figure 9(b), in the entropy estimator processing (Simple Lined Causal CNN) according to this embodiment, the CNN kernel is configured with a size of 1, for example, 1x3, with a vertical size of 1. When inputting tensors to the CNN kernel, a predetermined given initial value (for example, zero) is inserted into the top row in particular. By shifting the input tensor vertically downward by one row before input, it is possible to limit the prediction to the pixels at the top of the target pixel. In other words, the entropy of the target pixel is predicted based only on the pixels perpendicular to the direction of the pixel arrangement. This eliminates the dependency on horizontal predictions and allows for batch processing as the current prediction does not require the previous prediction result. That is, during decoding, a [Bm×C3×1×W3] prediction can be performed simultaneously for the quantized feature map. Therefore, the number of iterations required for prediction during decoding can be reduced to only H3. On the other hand, during encoding, the entire quantized feature map can be processed simultaneously. Furthermore, because it does not require invalid parameters, it can reduce unnecessary zero multiplication and addition operations compared to Pixel CNNs.

[0082] Furthermore, conventional techniques generated context predictors using multiple layers. However, in this embodiment, the 2D CNN kernel used is a single layer, allowing for a simplified configuration of the context predictor.

[0083] Figure 10 shows an example algorithm for the CDF creation process in step S110b of Figure 8. This algorithm takes the parameters of the probability distribution, which are the output results of the entropy estimator process, as input and calculates the cumulative distribution function CDF for each symbol and the predicted probability f for each symbol. In this specification, the example algorithm is described using Python®-like notation.

[0084] The predicted probability f for each symbol is inherently a continuous function, but it needs to be converted to an integer representation for entropy coding. Integer representation of the predicted probability f means that, given the denominator coefficients d, the predicted probability for each symbol is generated such that the sum of the coefficients is d (which corresponds to 1 in the normal probability representation) and the coefficients are integers greater than or equal to 1. The cumulative distribution function CDF is the cumulative representation of f.

[0085] In this embodiment, the probability distribution is determined using a predetermined probability distribution function for symbol prediction, with the output result of the entropy estimator process as a parameter. For example, a Gaussian distribution is used as the probability distribution function for symbol prediction. However, it is not limited to the Gaussian distribution, and other probability distribution functions such as the logistic distribution can be used. In the case of the Gaussian distribution, the mean corresponds to the parameter mean, and the variance corresponds to the parameter scale. In the case of the logistic distribution, the position corresponds to the parameter mean, and the scale corresponds to the parameter scale. Furthermore, the output result of the entropy estimator process may be used for only some of the distribution parameters. For example, in the case of the Gaussian distribution, the mean may be fixed to a constant (e.g., 0), and the output result of the entropy estimator process may be used only for the scale, thereby simplifying and speeding up the process. For example, in this embodiment, assuming the use of two parameters, mean and scale, the shape of the tensor output of the entropy estimator process is [Bm × (2 * C3) × H3 × W3], and the shape of the predicted probability f and the cumulative distribution function CDF is 5-dimensional, [Bm × C3 × H3 × W3 × S], where S is the total number of symbols.

[0086] In Figure 10, symbolnum is the total number of symbols S. For example, if the possible values ​​of the symbols are integers from -15 to 16, then symbolnum = 32. Furthermore, calc_cdf is a function that takes two tensors, for example the mean and scale parameters of a Gaussian distribution, as input and calculates the cumulative probability of the random variable in the probability distribution of the input parameters from -∞ to 0. In the case of a Gaussian distribution, for example, it can be calculated using an error function.

[0087] In the first line of the process in Figure 10, we pre-prepare symbolnum variations of the parameter mean, and by calculating f in parallel for each parameter mean, it becomes possible to perform multiple calculations of the CDF using tensor operations, as shown in the fourth line.

[0088] In the second row of Figure 10, integer values ​​of f are calculated for each of the symbolnum parameter values ​​of mean. Here, since the minimum probability is 1, 1 is added at the end.

[0089] In the third row of Figure 10, since the sum of the num symbols f is denoted as d, the probability of the last symbol is calculated by subtracting the sum of the f values ​​up to the second-to-last symbol from the denominator coefficient d.

[0090] In the fourth row of Figure 10, the CDF is calculated using matmul (matrix operation). The matrix used here, matrix `matrix`, is a strictly triangular matrix of symbolnum × symbolnum, which allows the CDF to be generated in parallel from f using tensor operations.

[0091] In the fifth line of Figure 10, the generated f and CDF are output.

[0092] Figure 11 shows an example algorithm for the entropy encoder step processing in step S110c of Figure 8. In this embodiment, Tensor ANS is used to implement ANS (Asymmetric Numeral Systems) using tensor operations. In Figure 11, d is the denominator coefficient, L is the maximum code length, k is the number of iterations, and n is the normalization coefficient, as in Figure 10. Furthermore, the variables representing tensors will be explained below. The parentheses after the variable names indicate the shape of the tensor in this embodiment. In the first row of Figure 11, the argument x[Bm×C3] is the state variable during encoding, s[Bm×C3] is the target symbol, f[Bm×C3×S] is the predicted probability by the entropy estimator 210 for each symbol, CDF[Bm×C3×S] is the cumulative probability density distribution for each symbol, and enc[B m ∇ ×C3×N] represents the data stream to be encoded, and p[Bm×C3] represents a pointer (the current position in the encoding process and the amount of valid data). In the definition of enc, N is the length of the data stream and is assumed to be sufficiently long. One element of the data stream is log2(n)*N bits, and there are up to N such elements. For example, N = H3 × W3. In ANS, in general, an upper limit is set for the state variable x in order to improve the efficiency of the sign calculation. By writing data of a predetermined size (log2(n) bits for the normalization coefficient n) to the data stream and removing it from x, the range of x's value is controlled so that (n*L)>x≧L. This process is generally called renormalize. x_u[Bm×C3] in the algorithm is the upper limit of x in this control and can be calculated by n*(L / d)*f[s].

[0093] The first row of Figure 11 shows that rows 2 through 5 are repeated k times.

[0094] Conventional techniques require conditional branching to determine whether a value should be renormalized. In contrast, the proposed method of this embodiment (Tensor ANS), as shown in the second row of Figure 11, determines whether a value should be renormalized by using the existence value `exist` generated by `clamp`. Then, as shown in the third row of Figure 11, the update of `p` can be processed by a simple sum with `exist`. Furthermore, as shown in the fourth row of Figure 11, the product of the existence value `exist` and the remainder when the state `x` is divided by the normalization coefficient `n` allows processing, including whether renormalization is necessary, to be performed using element-wise sum-of-products operations on the tensor. This eliminates the need for conditional branching to determine whether a value should be renormalized, enabling high-speed multiplexing with tensor operations. Specifically, Tensor ANS can process Bm × C3 data streams simultaneously, and the degree of parallelism can be improved in stages by the aforementioned FM typer, enabling faster processing.

[0095] In this context, "clamp" refers to a process that restricts the target value to a range of 0 to 1. Specifically, values ​​less than 0 become 0, and values ​​greater than 1 are converted to 1. In particular, within the range used in the proposed method of this embodiment, the values ​​are integers, so the result of clamping will be either 0 or 1.

[0096] By using the above method, conditional branching for renormalization is unnecessary, enabling high-speed multiple execution of entropy coding on tensor operations.

[0097] Furthermore, in the fourth row of Figure 11, 'p' is specified as the index for 'enc', and `scatter_add` can be used as an implementation for this. `scatter_add` is a process that creates a tensor by adding a predetermined tensor with a specified index.

[0098] Additionally, for accessing data using other indexes such as f or CDF, the `gather` method can be used. `gather` is a process that uses a specified index to collect tensors at the specified index and create a tensor from them. These processes can also be performed in parallel for Bm × C3 data streams. Furthermore, another method involves creating one-hot vectors of these indices and performing tensor operations by calculating the product and sum of each element.

[0099] For the symbol operation, which searches for a corresponding symbol based on the CDF using a given probability y, the symbol (CDF index) was calculated by using clamp to restrict the range of the value obtained by subtracting y from the CDF to 0 to 1, and then calculating the sum.

[0100] As described above, by implementing all operations using tensor operations, it is possible to achieve operation multiplexing in dimensions such as batches and channels.

[0101] Figure 12 shows an overview of an example of the compressed data creation process in step S111 of Figure 5.

[0102] For example, in the entropy encoder step processing shown in Figure 11, unused portion data c1~cm included in the tensor data enc obtained are pre-filled, and the encoded portion data b1~bm (compressed tiles) are concatenated to form encoded data (compressed data). In addition, the length p of each encoded portion data b1~bm is used as map information (see Figure 3).

[0103] When decoding the encoded data, the encoded sub-data b1-bm of each encoded data is extracted from the map information, and unused sub-data c1-cm is added to obtain the tensor data enc.

[0104] Figure 13 is a flowchart illustrating an example of decompression processing according to Embodiment 1. The following describes an example where computer 101D performs decompression processing, but computer 101C can also perform decompression processing by similarly equipping itself with a decompressor 71. The decompression processing is executed by the processor 53D, the parallel processing device 61D, and the decompressor 71 implemented on the parallel processing device 61D, for example, when the data to be decompressed is input to computer 101D.

[0105] First, in step S301, the processor 53D determines whether or not it has processed all of the compressed data to be decompressed. If the processor 53D has processed all of the compressed data to be decompressed (step S301 YES), it terminates the decompression process. If it has not processed all of the compressed data to be decompressed (step S301 NO), it moves the process to step S302.

[0106] In step S302, the processor 53D acquires the next compressed data to be decompressed and stores it in the memory 63D of the parallel processing device 61D. Next, in step S303, the core 62D of the parallel processing device 61D decompresses the compressed data to be decompressed stored in the memory 63D based on the map information and breaks it down into compressed tiles (encoded batches). This process is the reverse of the compressed data creation process shown in Figure 12.

[0107] Next, in step S304, the processor 53D adds the compressed tiles obtained in step S304 to the queue in the decompressor 71.

[0108] Next, in step S305, the processor 53D determines whether there are compressed tiles on the queue that are equal to or greater than a predetermined threshold, or whether all images to be decompressed have been processed. If there are compressed tiles on the queue that are equal to or greater than a predetermined threshold, or if all images to be decompressed have been processed (step S305YES), the processor 53D moves to step S306. If there are no compressed tiles on the queue that are equal to or greater than a predetermined threshold, and if all images to be decompressed have not been processed (step S305NO), the processor 53D moves to step S310.

[0109] From this point onward, steps S306 to S309 are performed in parallel with respect to the data stream by a number of batches Bm × number of channels C.

[0110] In step S306, the entropy decoder 215 retrieves compressed tiles from the queue and creates batches of batch size Bm based on the predetermined threshold mentioned above. Here, as with the compression process, data from multiple input images can be processed simultaneously. Next, in step S307, the entropy decoder 215 performs entropy decoding on the batches created in step S306. Details of step S307 will be described later with reference to Figure 14.

[0111] Next, in step S308, the second decoder filter 213 performs the second decoder filtering process on the batch that underwent entropy decoding in step S308. The second decoder filtering process takes a quantized feature map as input and is performed in parallel by multiple cores 62D.

[0112] Next, in step S309, the FM untyler 212 stores the unfolded tile from step S309 into the tile queue (a queue for tiles).

[0113] Next, in step S310, the FM anti-tiller 212 determines whether all the division data for one image has been collected in the tile queue. If all the division data for one image has been collected in the tile queue (step S310 YES), the FM anti-tiller 212 proceeds to step S311, and if not all of the data has been collected (step S310 NO), it returns to step S305.

[0114] In step S311, the FM untyler 212 takes one unfolded tile from the tile queue and combines the FMs (FM untiling). Next in step S312, the first decoder filter 211 performs the first decoder filtering process on the tensor data restored to its pre-split state by FM untiling in step S311 to obtain image data. Next in step S313, if padding has been applied to the original image data before compression, the core 62D of the parallel processing device 61D removes the padding from the image data obtained in the first decoder filtering process in step S312.

[0115] Next, in step S314, the core 62D of the parallel processing device 61D generates and outputs the decompressed image data. When step S314 is completed, the core 62D of the parallel processing device 61D returns processing to step S310.

[0116] Figure 14 is a flowchart showing a detailed example of the entropy decoding process.

[0117] First, in step S307a, the entropy decoder 215 performs a predetermined number of renormalization operations during encoding in order to initialize the encoded data to argument x (similar to that shown in Figure 11).

[0118] Next, in step S307b, the entropy estimator 210 performs entropy estimator processing. The entropy estimator processing in step S307b is the same as the processing in step S110a in Figure 8 and in Figure 9.

[0119] Next, in step S307c, the entropy estimator 210 performs the CDF creation process. The CDF creation process in step S307c is the same as the process in steps S110b and Figure 10 in Figure 8.

[0120] Next, in step S307d, the entropy decoder 215 performs the entropy decoder step process. Details of the entropy decoder process will be described later with reference to Figure 15.

[0121] Next, in step S307e, the entropy decoder 215 determines whether the prediction unit of the entropy estimator 210 has been decoded or not. If the prediction unit of the entropy estimator 210 has been decoded (step S307e), the entropy decoder 215 moves the process to step S307f, and if the prediction unit of the entropy estimator 210 has not been decoded (step S307eNO), it returns the process to step S307d to decode the next group of symbols. In this embodiment, as shown in Figure 9, the prediction units of the entropy estimator 210 are processed in parallel, one column at a time, so the loop of steps S307d and S307e is repeated.

[0122] In step S307f, the entropy decoder 215 determines whether all symbols have been decoded. If all symbols have been decoded (step S307fYES), the entropy decoder 215 proceeds to step S308 in Figure 13; otherwise, if all symbols have not been decoded (step S307fNO), it returns to step S307b to perform the prediction for the next prediction unit.

[0123] Figure 15 shows an example algorithm for the entropy decoder step processing in step S307d of Figure 14.

[0124] The first line of Figure 15 indicates that lines 2-4 are repeated k times. The `fill` variable in the second line of Figure 15, similar to the encoding algorithm, indicates whether or not a renormalization process needs to be performed, and this is used to enable implementation using only tensor operations. Furthermore, `clamp(p)` in the second line prevents the renormalization process from being performed again even if p is zero, thereby suppressing improper renormalization processes and the updating of p to a negative value. In this way, by implementing all operations using tensor operations, just as in the encoding process, it is possible to achieve operation multiplexing in dimensions such as batches and channels. Moreover, since all operations are implemented using tensor operations, no special programming specific to parallel processing devices is required, and it can be easily implemented on machine learning middleware that implements tensor operations, allowing for easy and effective extraction of parallelism on parallel processing devices.

[0125] Furthermore, in order to achieve high speed in the proposed method in this embodiment, it is important to adjust the parameters denominator coefficient d, normalization coefficient n, and maximum code length L. This is because if the number of renormalization operations k increases due to the influence of the case where these values ​​are at their maximum, the computational load will increase. In general, k can be calculated by k = ceil(log(d) / log(n)). For example, d = 2 12 n=2 8 , L=2 16 This approach has minimal impact on the compression ratio and reduces the number of renormalization operations k. In this case, the renormalization flow only needs to be executed k=2 times, so the impact on processing speed is minimal.

[0126] Furthermore, in this embodiment, a CNN may be used in the first encoder filter, the second encoder filter, the first decoder filter, the second encoder filter, the entropy estimator, etc. In the training, the cross-entropy between the probability distribution calculated using the parameters output by the entropy estimator and the FM value output by the encoder is calculated to calculate the theoretical average bitrate in a differentiable form. The distortion between the input image of the encoder and the output image of the decoder (for example, PSNR, MS-SSIM, or other image quality metrics implemented in a differentiable form) is also calculated, and training can be performed to optimize the trade-off between the average bitrate and the distortion end-to-end using a Lagrangian optimizer or the like. In addition, in the training, it is desirable to use image data larger than the size of the original data corresponding to the tile (tile division width) as training data in order to reduce artifacts at the tile boundaries.

[0127] Furthermore, in this embodiment, the first encoder filter, the second encoder filter, the first decoder filter, and the second encoder filter, which have a large sum-of-accumulate operation, may be implemented using low-precision but high-speed calculations (for example, half-precision floating-point calculations), while the entropy estimator, which has a small sum-of-accumulate operation, may be implemented using high-precision calculations to ensure efficient processing.

[0128] [Embodiment 2] Embodiment 2 will now be described. In doing so, the differences from Embodiment 1 will be explained primarily, and the similarities with Embodiment 1 will be omitted or simplified.

[0129] Figure 16 shows a partial configuration example of the entropy estimator 210B and related compressor 70B and expander 71B according to Embodiment 2. Compared to the compressor 70 and expander 71 of Embodiment 1, the compressor 70B and expander 71B are equipped with an entropy estimator 210B instead of an entropy estimator 210. Furthermore, compared to the compressor 70 of Embodiment 1, the compressor 70B is equipped with a shuffler 801B1 between the quantizer 204 and the entropy encoder 205, and a shuffler 801B2 between the entropy estimator 210B and the entropy encoder 205 and entropy decoder 215. The shufflers 801B1 and 801B2 may be the same.

[0130] Furthermore, the extension unit 71B includes an unshuffler 802B between the entropy decoder 215 and the second decoder filter 213.

[0131] The entropy estimator 210B comprises an encoder filter 201B, a quantizer 204B, an entropy encoder 205B, an entropy estimator 210B1, an entropy decoder 215B, and a decoder filter 211B.

[0132] Unlike the Lined Causal CNN in Figure 9, encoder filter 201B predicts the predicted probability (CDF,f) for the quantized FM (input FM) input from quantizer 204 using a superprior distribution.

[0133] The encoder filter 201B performs encoder filtering on the input FM. The quantizer 204B, similar to the quantizer 204, performs quantization on the FM that has been encoder filtered by the encoder filter 201B. The entropy estimator 210B1 takes the FM quantized by the quantizer 204B as input and predicts the predicted probability (CDF,f). The entropy estimator 210B1 predicts the predicted probability (CDF,f) using the context model of the Lined causal CNN shown in Figure 9(b), or a simple static probability distribution.

[0134] The entropy encoder 205B obtains encoded information such as cross-entropy based on the probability of occurrence of each correct symbol, which is directly calculated from the FM quantized by the quantizer 204B, and the predicted probability (CDF,f) predicted by the entropy estimator 210C. The entropy encoder 205B outputs this encoded information as ultra-prior distribution compressed data 300B.

[0135] The entropy decoder 215B takes the predicted probability (CDF,f) from the entropy estimator 210B1 as input and performs entropy decoding on the ultra-prior distribution compressed data 300B to obtain decoded data. The decoded data of the previously decoded symbol is input to the entropy estimator 210B1 for estimating the predicted probability (CDF,f) of the currently decoded symbol.

[0136] The decoder filter 211B performs decoder filtering on the data decoded by the entropy decoder 215B and outputs a predicted probability (CDF, ​​f) corresponding to the input FM.

[0137] The shufflers 801B1 and 801B2 randomly shuffle the input FM and the corresponding predicted probability (CDF, ​​f) so that they are in the same order. Any shuffling method is acceptable as long as it is reproducible. Alternatively, the order may be changed using a method such as bin packing based on the predicted probability (CDF, ​​f).

[0138] The entropy encoder 205 performs entropy encoding on the quantized FM in the order of FM and predicted probabilities modified by the shufflers 801B1 and 801B2, generating compressed data 300. The entropy decoder 215 decodes the compressed data 300 in the order of predicted probabilities modified by the shuffler 801B2, generating decoded data.

[0139] The unshuffler 802B restores the order of the symbols in the decoded data, decoded by the entropy decoder 215, to the order before shuffling by shufflers 801B1 and 801B2.

[0140] Thus, when using a method to generate a superprior distribution, the predicted probabilities of multiple symbols are obtained at once. Random shuffling allows for the equalization of the average entropy of each batch, which is the division unit, thereby reducing the overhead in terms of the amount of information in the map data.

[0141] Furthermore, for example, by applying a predetermined amount of dropout to the FM during the training of a compression / decompression neural network, a model that is robust to some error in the FM value can be trained. This makes it possible to store only the average value of the pointer p of all streams as map information, and generate an FM with a predetermined value for data located above the average value, thereby enabling data decoding and significantly reducing the data size of the map information.

[0142] [Embodiment 3] Embodiment 3 will now be described. In this description, the differences from Embodiment 1 will be explained primarily, and the similarities with Embodiment 1 will be omitted or simplified. Embodiment 3 will describe the case where JPEG format image data is handled.

[0143] Figure 17 shows an example configuration of the compressor 70C and decompressor 71C according to Embodiment 3. Compared to the compressor 70 according to Embodiment 1, the compressor 70C includes a Huffman decoding / tensorization 201C instead of the first encoder filter 201. The decompressor 71C also includes a Zigzag encoding / Huffman coding 211C instead of the first decoder filter 211 compared to the decompressor 71 according to Embodiment 1. Furthermore, compared to the compressor 70 and decompressor 71 according to Embodiment 1, the compressor 70C and decompressor 71C include an entropy estimator 210C instead of the entropy estimator 210.

[0144] Huffman decoding / tensorization 201C decodes JPEG format image data using the Huffman decoding method up to just before the inverse discrete cosine transform (DCT), obtaining tensor data with the dimensions of the DCT (8x8 in the example in Figure 17) as the decoded data. In this embodiment, the tensor data is 6-dimensional, having six dimensions: batch, channel, DCT height, DCT width, height, and width. Huffman decoding / tensorization 201C then pads the tensor data so that the padding is a multiple of the division height (Th) × division width (Tw). In the example in Figure 17, the input image has 256 pixels, with Th and Tw being the sizes.

[0145] FM Tyler 202 expands the dimensionality of batches of padded tensor data. In the example in Figure 17, for an input FM of size (1×3×8×8×H2×W2), FM Tyler 202 divides it vertically and horizontally into tiles, using these divisions as the batch dimensions, to output an FM of size (((H2 / 32)×(W2 / 32))×3×8×8×32×32.

[0146] The second encoder filter 203C may use, for example, a 4D CNN kernel or a 2D+2D CNN kernel to process a four-dimensional structure. The quantizer 204 is the same as in Embodiment 1. The second encoder filter 203C and the quantizer 204 are omitted in the case of lossless compression formats such as PNG.

[0147] The entropy estimator 210C estimates entropy for a four-dimensional structure, so it uses a 4D CNN kernel, but a 2D+2D CNN kernel may also be used. Also, if the chroma format is not 4:4:4, the tensor sizes of Y, Cb, and Cr will differ. Therefore, channels of different sizes are divided, each divided channel is encoded, and information is exchanged between these channels. For example, the reconstruction result of one channel is used as auxiliary information during the decoding of the other. For example, the encoding and decoding of the luminance component Y channel, which has relatively small data, can be performed before the blue difference Cb and red difference Cr channels, which have relatively large data. By using the information obtained from the encoding or decoding of the luminance component Y channel as auxiliary information in the encoding or decoding of the blue difference Cb and red difference Cr channels, compression can be made more efficient.

[0148] The entropy encoder 205 and entropy decoder 215 are the same as in Embodiment 1. The second decoder filter 213C processes a four-dimensional structure, so for example, a 4D CNN kernel or a 2D+2D CNN kernel may be used. The second decoder filter 213C is omitted in the case of lossless compression formats such as PNG. The FM untyler 212 performs the reverse processing of the FM tyler 202 described above. The zigzag encoding / Huffman encoding 211C outputs a JPEG image format by zigzag encoding the input data and Huffman encoding it.

[0149] Note that if the file to be compressed is in a lossless format such as PNG, the "8x8" DCT unit does not exist, so it is processed using a 2D CNN kernel. Alternatively, the "8x8" DCT unit can be combined into the channel dimension instead of being set up as a separate dimension.

[0150] Figure 18 shows an example of compression processing according to Embodiment 3. In Figure 18, a JPEG format image data 400 with a chroma format of 4:2:0, height H1 = 512 pixels, and width W1 = 768 pixels is used as the input image, and an example of tiling with division height = division width = 256 pixels is shown.

[0151] In the compression process according to Embodiment 3, the compression process is performed by separating the channel for the luminance component Y and the channels for the blue difference Cb and red difference Cr.

[0152] First, in step S401, Huffman decoding / tensorization 201C performs Huffman decoding of JPEG format image data 400 up to just before the inverse discrete cosine transform (DCT), separating it into channels for luminance component Y, blue difference Cb, and red difference Cr, and then tensorizing them. Step S401 yields the feature map FM(Y)401a of the luminance component Y channel with order = 1 × 1 × (8 × 8) × (H1 / 8) × (W1 / 8). Also in step S401, the feature map FM(CbCr)401b of the blue difference Cb and red difference Cr channels with order = 1 × 2 × (8 × 8) × (H1 / 16) × (W1 / 16). Feature maps FM(Y) and FM(CbCr) have a dimension of (8 × 8) in DCT units.

[0153] Next, in step S402, Huffman decoding / tensorization 201C pads the feature maps FM(Y) and FM(CbCr) respectively so that the 5th and 6th dimensions (H1 / 8 and W1 / 8 in the case of feature map FM(Y), and H1 / 16 and W1 / 16 in the case of feature map FM(CbCr)) are divisible by the JPEG block size, which is tiling size 16 (split height Th = split width Tw). Step S402 yields the feature map FM(Y) 402a for the luminance component Y channel with order = 1 × 1 × (8 × 8) × (H2 / 8) × (W2 / 8). Also, step S401 yields the feature map FM(CbCr) 402b for the blue difference Cb and red difference Cr channels with order = 1 × 2 × (8 × 8) × (H2 / 16) × (W2 / 16).

[0154] Next, in step S403, the FM tiler 202 tiles to the batch dimension (FM batch size B=6), obtaining the feature map FM(Y)403a of the luminance component Y channel with order = 6 × 1 × (8 × 8) × 32 × 32, and the feature map FM(CbCr)403b of the blue difference Cb and red difference Cr channels with order = 6 × 2 × (8 × 8) × 16 × 16.

[0155] Furthermore, when performing lossy compression on JPEG format image data 400, as shown in Figure 17, the feature map FM(Y) 403a and the feature map FM(CbCr) 403b may be subjected to a second encoder filter process by the second encoder filter 203 and a quantization process by the quantizer 204.

[0156] Next, in step S404, the entropy encoder 205 entropy encodes the feature map FM(CbCr) 403b to create a data stream 404b.

[0157] Next, in step S405, the entropy encoder 205 upsamples the 5th and 6th dimensions of the entropy-encoded feature map FM(CbCr)403b by, for example, 2 times to create auxiliary information for entropy-encoded feature map FM(Y)403a.

[0158] Next, in step S406, the entropy encoder 205 entropy encodes the feature map FM(Y)403a using the auxiliary information created in step S405 to create a data stream 406a.

[0159] Since the feature map FM(CbCr)403b has a smaller data size than the feature map FM(Y)403a, entropy encoding can be performed on the feature map FM(Y)403a before the feature map FM(Y)403a. By using this encoded information as auxiliary information for the entropy encoding of the feature map FM(Y)403a, the compression ratio of the feature map FM(Y)403a can be improved. By encoding the smaller feature map FM(CbCr)403b first and completing the encoding quickly, the auxiliary information can be quickly used when encoding the feature map (Y)403a of the other channels.

[0160] Next, in step S407, the entropy encoder 205 performs a compressed data creation process based on the predicted probability (CDF,f) from the entropy estimator 210C, and creates compressed data 300C from data streams 406a and 404b.

[0161] Although Figure 18 only illustrates the encoding process for each channel, it is not limited to this. That is, the encoding and decoding processes for channels with relatively smaller data may be performed before those for channels with relatively larger data, and the information obtained from the encoding or decoding of the channels performed earlier may be used as auxiliary information in the encoding or decoding of other channels.

[0162] Figure 19 is a diagram illustrating an example of processing by the entropy estimator 210C according to Embodiment 3.

[0163] To improve processing speed, the entropy estimator 210C performs 4D (four-dimensional) convolutional neural network processing by first performing 2D convolution on two of the four dimensions using the kernel of the first two-dimensional neural network. Then, it performs 2D convolution on the remaining two dimensions using the kernel of the second neural network.

[0164] For example, if the input tensor has dimensions B×C×8×8×32×32, first the entropy estimator 210C performs 2D CNN processing S210C1 for the third and fourth DCT dimensions (8×8), for example, with a kernel size of 3×3×1×1. Next, the entropy estimator 210C performs 2D CNN processing S210C2 for the fifth and sixth spatial dimensions (32×32), for example, with a kernel size of 1×1×3×3. In this way, the entropy estimator 210C takes an input tensor of dimensions B×C×8×8×32×32 as input and outputs a parameter tensor of dimensions B×(C*2)×8×8×32×32, with two parameters, mean and scale, for each channel.

[0165] In this way, the entropy estimator 210C can improve the processing speed of entropy prediction by dividing the 4D (four-dimensional) convolution process into two 2D convolution processes and executing them in stages.

[0166] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them. In addition, some or all of the configurations and processes of one embodiment may be combined with other embodiments without departing from the spirit of the invention or being inconsistent.

[0167] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a recording medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the embodiment described above, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc. The program code that realizes the functions described in this embodiment can be implemented in any programming language or scripting language.

[0168] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected. [Explanation of Symbols]

[0169] S: Data compression / decompression system, 101, 101C, 101D: Computer, 61C, 61D: Parallel processing device, 70, 70B, 70C: Compressor, 71, 71B, 71C: Decompressor, 201: First encoder filter, 201C: Huffman decoding / tensorization, 202: FM tyra (coupler), 203: Second encoder filter, 204: Quantizer, 205: Entropy encoder (encoder), 210, 210B, 210B1, 210C: Entropy estimator (estimator), 211: First decoder filter, 211C: Zigzag encoding / Huffman coding, 211: First decoder filter, 212: FM untyra (coupler), 213: Second decoder filter, 215: Entropy decoder (decoder)

Claims

1. A data compression / decompression system having a parallel processing device that executes multiple processes in parallel, The parallel processing device has memory and multiple cores, The parallel processing device is The original data is divided into multiple data units in predetermined units, and the multiple data units are stored in the memory. An entropy estimater using a neural network that predicts the probability distribution of the multiple data sets generates the probability distribution for the multiple data sets in parallel. The cumulative distribution function of the aforementioned probability distribution is generated in parallel for the probability distribution corresponding to the plurality of data, Based on the probability distribution and the cumulative distribution function, the entropy coding process is performed in parallel on the plurality of data to generate the plurality of coded data. Compressed data is created by compressing the original data from the multiple encoded data mentioned above. A data compression and decompression system characterized by the following:

2. A data compression / decompression system according to claim 1, The information regarding the correspondence between the plurality of encoded data and the compressed data is recorded together with the compressed data. A data compression and decompression system characterized by the following:

3. A data compression / decompression system according to claim 1, The parallel processing device is A first encoder filter that performs filtering on the original data to create a feature map, A divider that creates a plurality of divided data by dividing the feature map created by the first encoder filter into a plurality of feature maps in predetermined units, A second encoder filter that performs filtering on the multiple divided data created by the divider, The entropy estimater predicts the entropy of the divided data sets, An entropy encoder that performs the entropy coding process in parallel on the divided data based on the entropy prediction result to generate the multiple coded data, and creates the compressed data from the multiple coded data, A compressor was constructed that had A data compression and decompression system characterized by the following:

4. A data compression / decompression system according to claim 1, The parallel processing device is The compressed data is expanded into the plurality of encoded data, Entropy decoding is performed in parallel on the aforementioned multiple encoded data to generate multiple decoded data. The above-mentioned multiple decoded data are combined to create decompressed data from the compressed data. A data compression and decompression system characterized by the following:

5. A data compression / decompression system according to claim 4, Based on the information regarding the correspondence between the multiple encoded data recorded together with the compressed data and the compressed data, the compressed data is decompressed into the multiple encoded data. A data compression and decompression system characterized by the following:

6. A data compression / decompression system according to claim 4, The parallel processing device is The aforementioned entropy estimater, An entropy decoder that decompresses the compressed data into the plurality of encoded data, and performs the entropy decoding process in parallel on the plurality of encoded data to generate the plurality of decoded data, A second decoder filter that performs filtering on the plurality of decoded data, A coupler that combines the plurality of decoded data filtered by the second decoder filter, A first decoder filter performs filtering on the plurality of decoded data combined by the coupler to create decompressed data of the compressed data, which is created from the plurality of encoded data generated from the divided data obtained by dividing the original data, A stretcher having was constructed. A data compression and decompression system characterized by the following:

7. A data compression / decompression system according to claim 1, The entropy encoding processor that performs the aforementioned entropy encoding process is configured using Tensor ANS (Asymmetric Numeral Systems), which enables parallel execution by replacing predetermined processes in ANS with tensor operations. A data compression and decompression system characterized by the following:

8. A data compression / decompression system according to claim 6, The aforementioned entropy decoder is, A tensor is constructed from the divided data and the information relating the correspondence between the multiple encoded data recorded together with the compressed data and the compressed data, and the entropy decoding process is executed in parallel on the multiple encoded data based on this tensor. A data compression and decompression system characterized by the following:

9. A data compression / decompression system according to claim 3, The aforementioned entropy estimator is, By predicting the entropy of each target pixel based only on the pixels perpendicular to the direction of arrangement of the target pixels in the divided data sets, the entropy prediction process for each target pixel is executed in parallel. A data compression and decompression system characterized by the following:

10. A data compression / decompression system according to claim 1, An entropy encoder that performs the aforementioned entropy coding process in parallel, An entropy decoder that decompresses the compressed data into the plurality of encoded data, and performs entropy decoding on the plurality of encoded data in parallel to generate a plurality of decoded data, A shuffler that randomly changes the order of the divided data sets and the probability distribution and cumulative distribution function of the divided data sets predicted by the entropy estimater, The system includes an unshuffler that restores the order of the divided data, which has been altered by the shuffler, to its original state. The entropy estimator generates the predicted probabilities of the divided data sets, The entropy encoder creates the plurality of coded data based on the plurality of divided data whose order has been changed by the shuffler and the predicted probability. The entropy decoder uses the predicted probabilities, which have been reordered by the shuffler. Based on this, the aforementioned multiple decoded data are generated, The unshuffler restores the order of the multiple decoded data to its original state. A data compression and decompression system characterized by the following:

11. A data compression / decompression system according to claim 4, The parallel processing device is If the source data includes multiple channels, these channels are separated, and the entropy coding and decoding processes are performed independently on each separated channel. The information obtained from the entropy coding or decoding process of one channel is then used as auxiliary information in the entropy coding or decoding process of the other channels. A data compression and decompression system characterized by the following:

12. A data compression / decompression system according to claim 11, The parallel processing device is If the data sizes of the multiple channels differ, the entropy coding and decoding processes for the channel with a data size smaller than a predetermined value among the original data are performed before those of the other channels, and the information obtained from the entropy coding or decoding process is used as auxiliary information in the entropy coding or decoding process of the other channels. A data compression and decompression system characterized by the following:

13. A data compression / decompression system according to claim 3, The entropy estimater predicts the entropy of a four-dimensional neural network by using a kernel of a first two-dimensional neural network for two of the four dimensions and a kernel of a second two-dimensional neural network for the remaining two dimensions. A data compression and decompression system characterized by the following:

14. A data compression and decompression method performed by a data compression and decompression system having a parallel processing device that performs multiple processes in parallel, The parallel processing device has memory and multiple cores, The aforementioned parallel processing device The original data is divided into multiple data units in predetermined units, and the multiple data units are stored in the memory. An entropy estimater using a neural network that predicts the probability distribution of the multiple data sets generates the probability distribution for the multiple data sets in parallel. The cumulative distribution function of the aforementioned probability distribution is generated in parallel for the probability distribution corresponding to the plurality of data, Based on the probability distribution and the cumulative distribution function, the entropy coding process is performed in parallel on the plurality of data to generate the plurality of coded data. Compressed data is created by compressing the original data from the multiple encoded data mentioned above. A data compression and decompression method characterized by having each of the following processes.

15. A data compression and decompression method according to claim 14, The aforementioned parallel processing device The compressed data is expanded into the plurality of encoded data, Entropy decoding is performed in parallel on the aforementioned multiple encoded data to generate multiple decoded data. The above-mentioned multiple decoded data are combined to create decompressed data from the compressed data. A data compression and decompression method characterized by having each of the following processes.

16. A data compression and decompression program for causing a computer to function as a data compression and decompression system according to any one of claims 1 to 13.

Images