Data compression system, data compression method, and data compression program

Abstract

To improve a processing speed of a compression processing.SOLUTION: A calculator system 1 for compressing data includes a parallel processing device 61B. The parallel processing device 61B divides the compression target data into a plurality of pieces of partial data, performs compression processing on each of the pieces of partial data in parallel, calculates an appearance probability for each predetermined data unit of the partial data using a neural network in the compression processing, and outputs an encoded bit string which is a bit string entropy-encoded on each of the data units based on the data unit and the appearance probability. A process for realizing the neural network includes a first conversion process for executing a matrix product operation process, and a second conversion process subsequent to the first conversion process for inputting a processing result of the first conversion process and converting each element of the matrix resulting from the processing result into a 1-bit integer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for compressing data. [Background technology]

[0002] With the advancement of IoT (Internet of Things), the amount of data generated has increased dramatically, and the amount of data stored in hybrid clouds has also increased. Although storage costs have fallen, they are not enough to cover the growth in data volume. Therefore, it is becoming increasingly important to reduce the amount of data stored in storage. This is also important from the perspective of reducing data communication volume.

[0003] As related technologies, for example, Patent Document 1 discloses a technology that can reduce the amount of data in storage by using a neural network, and Patent Document 2 discloses a technology that can reduce the amount of data by using an integer neural network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-95913 [Patent Document 2] International Publication No. 2020 / 068498 Summary of the Invention [Problem to be solved by the invention]

[0005] Although compression can reduce the amount of data, it takes a long time to complete the compression process, which can increase the cost of computational resources and affect other processes.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique that can improve the processing speed of compression processing. [Means for solving the problem]

[0007] In order to achieve the above object, one aspect of a data compression system is a data compression system that compresses data, and includes a parallel processing device. The parallel processing device divides data to be compressed into a plurality of partial data and performs compression processing on each of the partial data in parallel. In the compression processing, the parallel processing device uses a neural network to calculate the occurrence probability of each predetermined data unit of the partial data, and for each of the data units, outputs an encoded bit string that is an entropy-encoded bit string based on the data unit and the occurrence probability. The processing that realizes the neural network includes a first conversion processing that performs a matrix multiplication operation, and a second conversion processing that follows the first conversion processing, inputs the processing result of the first conversion processing, and converts each element of the matrix resulting from the processing result into a 1-bit integer. [Effects of the Invention]

[0008] According to the present invention, the processing speed of the compression process can be improved. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a computer system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the logical-to-physical conversion table according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing the configuration of block data storage information according to the first embodiment. [Figure 4] FIG. 4 is a configuration diagram of history information according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an overview of backup processing and restore processing in the computer system according to the first embodiment. [Figure 6]FIG. 6 is a diagram illustrating the compressor according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating the expander according to the first embodiment. [Figure 8] FIG. 8 is a flowchart of the compression process by the compressor according to the first embodiment. [Figure 9] FIG. 9 is a flowchart of the decompression process by the decompressor according to the first embodiment. [Figure 10] FIG. 10 is a flowchart of the backup process according to the first embodiment. [Figure 11] FIG. 11 is a flowchart of the restore process according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating the prediction model processing according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating the data flow of the prediction model processing and the integrated model processing according to the first embodiment. [Figure 14] FIG. 14 is a diagram illustrating an algorithm for the prediction model processing according to the first embodiment. [Figure 15] FIG. 15 is a configuration diagram of a cache information table according to the first embodiment. [Figure 16] FIG. 16 is a configuration diagram of a parallel processing device according to the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating the matrix multiplication operation process and the activation function process according to the first embodiment. [Figure 18] FIG. 18 is a flowchart of the weighting matrix quantization process according to the first embodiment. [Figure 19] FIG. 19 is a flowchart of the matrix multiplication process according to the first embodiment. [Figure 20] FIG. 20 is a flowchart of the activation function process according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating the data flow of the matrix multiplication process and the activation function process in the parallel processing device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following description of the embodiments will be given with reference to the drawings. Note that the embodiments described below do not limit the scope of the invention as claimed, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0011] In the following explanation, information may be described using the expression "AAA table", but the information may be expressed in any data structure. In other words, to show that the information does not depend on the data structure, the "AAA table" can be called "AAA information".

[0012] FIG. 1 is a diagram showing the overall configuration of a computer system according to the first embodiment.

[0013] The computer system 1 includes an input device 40, a user terminal 80, and computers 101A and 101B. The computers 101A and 101B are each an example of a data compression system. The computer system 1 can also be said to be an example of a data compression system. The computers 101A and 101B are connected via a network 150. The network 150 is, for example, a communication path such as a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), or a dedicated line. The computer 101A is connected to the input device 40. The computer 101B is connected to the user terminal 80. The computers 101A and 101B may be virtual computing resources (for example, virtual machines, containers, etc.) in a cloud environment.

[0014] The input device 40 is configured by a computer such as a PC (Personal Computer), and receives input from a user and executes various processes using data. The input device 40 stores data used for processing in the computer 101A and acquires the data from the computer 101A. The input device 40 may be a virtual computing resource (for example, a virtual machine or a container) in a cloud environment.

[0015] The user terminal 80 is configured by a computer such as a PC, and receives instructions from a user to configure the computer 101B, etc. The user terminal 80 may be a virtual computing resource (e.g., a virtual machine or a container) in a cloud environment.

[0016] The computer 101A stores and manages data used by the input device 40. The computer 101A performs a process of backing up data to the computer 101B and a process of restoring data from the computer 101B. The computer 101A is configured by a computer such as a PC or a server device, and includes a processor 53A, a memory 52A, IFs (interfaces) 5A1 and 5A2, a parallel processing device 61A, a persistent storage device 54A, and a bus 10A that connects these components.

[0017] The IFs 5A1 and 5A2 are interfaces such as a wired LAN card or a wireless LAN card, and communicate with other devices via the network 150 or a communication line.

[0018] The processor 53A executes various processes in accordance with the programs stored in the memory 52A. The processor 53A configures a compressor 72A and a decompressor 73A by executing the programs. The processor 53A also configures a first block corresponding security code creation unit and a second block corresponding security code creation unit by executing the programs. The compressor 72A performs compression processing on data, for example. The compression processing algorithm used by the compressor 72A may be gzip or LZMA (Lempel-Ziv-Markov chain algorithm). The decompressor 73A decompresses compressed data. The decompression processing algorithm used by the decompressor 73A may be an algorithm corresponding to the compression processing of the compressor 72A.

[0019] The memory 52A is, for example, a RAM (Random Access Memory), and stores programs executed by the processor 53A and necessary information. In this embodiment, the memory 52A stores, for example, a logical-physical conversion table 110 (see FIG. 2).

[0020] The persistent storage device 54A is, for example, a hard disk or flash memory, and stores programs that are read into the memory 52A to be executed by the processor 53A, as well as various data used by the processor 53A. In this embodiment, the persistent storage device 54A stores a data compression program as a program, and stores block data storage information 120 (see FIG. 3) and a logical-to-physical conversion table 110 (see FIG. 2) as data. Note that, instead of or in addition to the persistent storage device 54A, cloud storage connected via a network may be used. Note that the persistent storage device 54A and the cloud storage may be configured using block storage, file storage, object storage, a database, or the like.

[0021] The parallel processing device 61A is a device capable of executing processes in parallel, such as a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a multi-core central processing unit (CPU), and includes multiple cores 62A and a memory 63A. The cores 62A execute processes according to programs stored in the memory 63A. The memory 63A is an example of a storage unit and stores programs executed by the cores 62A and data used by the cores 62A. The memory 63A stores, for example, history information 130 (see FIG. 4 ). The multiple cores 62A can execute various processes in parallel using the memory 63A. In this embodiment, the parallel processing device 61A executes a program to configure a compressor 70A and a decompressor 71A. In this embodiment, the compression performed by the compressor 70A can achieve higher compression than the compression performed by the compressor 72A.

[0022] The computer 101B stores and manages data managed by the computer 101A. The computer 101B is configured as a computer such as a PC or a server device, and includes a processor 53B, a memory 52B, interfaces 5B1 and 5B2, a parallel processing device 61B, a persistent storage device 54B, and a bus 10B connecting these components. The processor 53B configures a compressor 72B and a decompressor 73B as an example of a decompression unit. The parallel processing device 61B has multiple cores 62B and a memory 63B, and configures a compressor 70B and a decompressor 71B. Each component of the computer 101B has the same configuration as the component in the computer 101A with the same first half of the reference numeral. Note that the configurations of the computers 101A and 101B are not limited to those described above, and some components may be omitted depending on the processing to be executed.

[0023] Next, the logical-physical conversion table 110 will be described.

[0024] FIG. 2 is a diagram showing the configuration of the logical-to-physical conversion table according to the first embodiment.

[0025] The logical-physical conversion table 110 is a table for managing physical addresses corresponding to logical addresses that are data storage destinations, and stores entries for each area of ​​a predetermined size (e.g., block). An entry in the logical-physical conversion table 110 includes fields for a logical address 110a, a physical address 110b, and a block size 110c. The logical address 110a stores a logical address indicating the area corresponding to the entry. The physical address 110b stores a physical address corresponding to the area corresponding to the entry. The block size 110c stores the size of the block of the area corresponding to the entry.

[0026] Next, the block data storage information 120 will be described.

[0027] FIG. 3 is a diagram showing the configuration of block data storage information according to the first embodiment.

[0028] The block data storage information 120 is information for managing block data and includes an entry for each block data. The entry in the block data storage information 120 includes fields for compressed data 120a, a guarantee code 120b, and a post-compression size 120c. The post-compression data 120a stores the compressed data after the block data has been compressed. The guarantee code 120b stores the guarantee code for the block data. The post-compression size 120c stores the data size of the compressed data. Data to be compressed is divided into multiple partial data and then compressed. Therefore, by assigning a post-compression size to each partial data, partial recompression or decompression can be performed without compounding partial data that is not included in the data position to be partially recompressed or decompressed. The model information 120d stores the fully trained model (a model that serves as the initial state of a prediction model or an integrated model). Alternatively, reference information for other corresponding models (models corresponding to other block data that have already been trained) may be stored to reduce the amount of data in the block data storage information. The model information 120d may also store the integer precision of the model layer.

[0029] Next, the history information 130 will be described.

[0030] FIG. 4 is a configuration diagram of history information according to the first embodiment.

[0031] The history information 130 is information for managing a probability distribution corresponding to data (calculation-used data) used when calculating the occurrence probability of a data unit, and includes an entry for each calculation-used data. An entry of the history information 130 includes fields for a code 130a and probability distribution information 130b. The code 130a stores a predetermined code (identification information) for the calculation-used data corresponding to the entry. The code may be a hash value for the calculation-used data or LSH (Locality Sensitive Hashing). The probability distribution information 130b stores information about the probability distribution calculated using the calculation-used data corresponding to the entry. The information about the probability distribution may be the occurrence frequency of the target data unit or a learning code obtained by a DNN (Deep Neural Network). The history information 130 is stored, for example, by the core 62B executing a program. The function of the core 62B corresponds to a history management unit.

[0032] Next, an overview of the backup process and restore process in the computer system 1 will be explained.

[0033] FIG. 5 is a diagram illustrating an overview of backup processing and restore processing in the computer system according to the first embodiment.

[0034] First, the backup process will be described. In the backup process, the data to be backed up is compressed by the compressor 72A of the computer 101A, a security code is created and associated in units of a predetermined block size (first size), and the data is stored in the memory 52A or the persistent storage device 54A.

[0035] The computer 101A refers to the logical-physical conversion table 110A, identifies the storage location in the memory 52A or persistent storage device 54A of the data to be backed up, acquires the data to be backed up and the security code, and transmits them to the computer 101B that is the backup destination.

[0036] The computer 101B stores the received backup target data and security code in memory 52B or persistent storage device 54B, and uses a decompressor 73B to decompress the backup target data. During this decompression process, the security code is used to detect and correct errors in the backup target data. Next, the compressor 70B compresses the decompressed backup target data, creates security codes for blocks of a predetermined block size (second size), stores the codes in memory 52B or persistent storage device 54B, and registers the storage destination in the logical-physical conversion table 110B. However, for partial updates (compressing only the part to be updated without decompressing the entire block), a security code that allows partial updates, such as XOR parity, may be used for each block. Alternatively, a security code may be created for each partial data to accommodate partial updates.

[0037] According to this backup process, the data to be backed up can be stored in a highly compressed state by the compressor 70B. Furthermore, if the second size is set to be larger than the first size and a security code is created using the second size as a data unit, the overall amount of data can be reduced. Note that the decompression process by the decompressor 73B of the computer 101B may be performed by the decompressor 73A of the computer 101A, and the decompressed data may be sent to the computer 101B.

[0038] Next, the restore process will be described. In the restore process, a case will be described in which the computer 101B is instructed to restore data that has been backed up by the backup process from the computer 101A. First, the decompressor 71B of the computer 101B references the logical-physical conversion table 110B, identifies the storage location of the backup data in the memory 52B or the persistent storage device 54B, acquires the data to be restored and the security code, and performs the decompression process. In this decompression process, the security code is used to perform error detection and correction on the data to be restored.

[0039] Next, the compressor 72B compresses the decompressed data to be restored using an algorithm that can be decompressed by the decompressor 73A of the computer 101A, creates a guarantee code in units of the first size, stores it in the memory 52B or the persistent storage device 54B, and sends the compressed data to be restored and the guarantee code to the computer 101A that is the destination of the restore.

[0040] The computer 101A stores the received data to be restored and the security code in the memory 52A or the persistent storage device 54A, and registers the storage destination in the logical-to-physical conversion table 110A. Next, the decompressor 73A performs a decompression process on the data to be restored. In this decompression process, the security code is used to perform error detection and correction on the data to be restored. This makes it possible for the computer 101A to use the data to be restored.

[0041] Next, the configuration and processing of the compressor 70 (70A, 70B) will be described.

[0042] FIG. 6 is a diagram illustrating the compressor according to the first embodiment.

[0043] The compressor 70 has a data dividing unit 701 and multiple compression processing units 702 (702A, 702B, etc.). Each compression processing unit 702 is configured by each core 62 (62A, 62B) of the parallel processing device 61 (61A, 61B) executing a program in a memory 63 (63A, 63B). Here, the learning unit and the model selection unit are configured by the compression processing units 702 executing a program.

[0044] The data dividing unit 701 divides the data to be compressed into a plurality of partial data (e.g., 1 KB). In this embodiment, the data dividing unit 701 divides the data into the same number of partial data as the number B of batches to be executed in parallel. In the compressor 70, the same number of compression processing units 702 as the number B of batches are used for processing, and each compression processing unit 702 performs compression processing on its own partial data. In addition, the data to be compressed can be divided into a further number of partial data by simultaneously processing a plurality of block data together, thereby increasing the degree of parallelism and speeding up processing on the parallel processing device 61B, for example.

[0045] The compression processing unit 702 includes a pre-encoding unit 703 as an example of a conversion unit, a probability calculation unit 704, and an entropy coding unit 705.

[0046] When the partial data to be compressed is binary data, the pre-encoding unit 703 performs one-hot encoding on each data unit of a predetermined size (u bits) included in the partial data. The pre-encoding unit 703 converts one data unit into 2u bits. The data encoded by the pre-encoding unit 703 becomes the number of channels (C) input to the probability calculation unit 704. Note that when one-hot encoding is performed, a predetermined bit of the encoded data for one data unit has the property that its value can be determined based on the value of the remaining bits. Therefore, the predetermined bit of the encoded data may be deleted, and the number of channels C may be set to 2u-1. Reducing the number of channels in this way reduces the amount of data input to the probability calculation unit 704. Note that the data used as input to the multiple probability calculation units 704 is an array of the number of batches of partial data (B) × the number of channels (C) × length (N). Note that additional auxiliary information may be added to the channel dimension of the data used for input. For example, by bit-encoding and adding the bit-wise logical address 110a corresponding to the partial data (adding bits corresponding to the maximum value of the logical address in addition to the 2u bits), the accuracy of the prediction model for the target data can be improved, thereby improving the compression rate. Furthermore, if the partial data to be compressed is numerical data or binary data that can be expressed numerically (such as sensor data or image data), the pre-encoding unit 703 may input the data as numerical data to the probability calculation unit 704. Generally, data on storage is expected to contain a mixture of numerical data files and text or binary data files. Therefore, for more efficient compression, multiple models may be prepared, such as a prediction model 704a for numerical data and a prediction model 704a for text or binary data.

[0047] Furthermore, the pre-encoding unit 703 calculates a hash value of the partial data to be compressed, and by comparing the hash values, detects duplication with partial data processed in the past, and for duplicated data, stores only reference information to the partial data processed in the past in the logical-physical conversion table 110 and the block data storage information 120. This eliminates the need to re-compress and store the same data, thereby increasing the compression rate. In this case, subsequent compression processes may be omitted, speeding up the processing.

[0048] The probability calculation unit 704 has a plurality of types of prediction models 704a and an integrated model 704b.

[0049] The prediction model 704a receives data (prediction data) used to predict the data unit (data unit to be predicted) whose occurrence probability is to be predicted and probability distribution information from the history information 130, and outputs prediction information Qb,i for the data unit to be predicted. Here, for example, if the data unit to be predicted is Db,i (i is the position of the prediction data unit in the partial data), the prediction data may be the data units Db,ir to Db,i-1, i.e., the r data units immediately preceding the data unit to be predicted. Note that if there are no r data units immediately preceding Db,i, the non-existent data units are assumed to have a predetermined value and are processed accordingly. The prediction model 704a may be a model configured with multiple trainable weights, such as a neural network, or may be a model configured to execute a predefined specific process. The prediction model 704a may output the occurrence probability as the prediction information Qb,i, or may output a feature map or the like at a position corresponding to the data unit to be predicted. Furthermore, when outputting the occurrence probability as prediction information Qb,i, a discrete probability (such as a categorical distribution) may be output as the occurrence probability for each symbol, or parameters (such as the mean and variance) of a continuous probability density function such as a Gaussian distribution may be output, or a result of calculating a discrete probability corresponding to the occurrence probability for each symbol from the cumulative distribution function of the continuous probability density function may be output. An example of the configuration of prediction model 704a will be described later with reference to Figures 12 to 14.

[0050] The integrated model 704b receives the prediction information output from the multiple types of prediction models 704a and the probability distribution information of the history information 130, and outputs (calculates) the occurrence probability Pb,i of the data unit to be predicted. The occurrence probability Pb,i is the distribution of the occurrence probability of each symbol of the data unit. A discrete probability (such as a categorical distribution) may be output. Alternatively, parameters (such as the mean and variance) of a continuous probability density function such as a Gaussian distribution may be output as the occurrence probability, and the result of calculating the discrete probability corresponding to the occurrence probability of each symbol from the cumulative distribution function during entropy encoding may be used. In other words, when one-hot encoding is performed, the integrated model 704b has the same number of elements as the number of channels (C=2u or 2u-1) input to the probability calculation unit 704. The integrated model 704b is a model such as a neural network. For example, it may be a gated linear network that integrates values ​​input from each prediction model 704a using trained weights, or it may be configured as a perceptron. The integrated model 704b may also include attention processing based on the output results of the prediction model (multiplying the data to be processed in the integrated model 704b by a value corresponding to the importance calculated from the output results of the prediction model). In this embodiment, the probability calculation unit 704 selects a prediction model to be used for processing from among multiple prediction models 704a based on the weights assigned to each prediction model 704a in the integrated model 704b. Note that the history information 130 may be provided for each compression unit 702 or may be shared by multiple compression units 702. In particular, when locality exists for each data unit, sharing the history information 130 can increase the number of statistical samples of the probability distribution, thereby improving prediction accuracy and the compression rate, and further reducing the amount of memory 63B used, for example. An example of the configuration of the integrated model 704b will be described later with reference to FIG. 13.

[0051] The entropy coding unit 705 receives the occurrence probability Pb,i output from the probability calculation unit 704 and the data unit Db,i as input, and outputs a bit string (encoded bit string) obtained by entropy coding the data unit Db,i.

[0052] Each compression processing unit 702 entropy encodes the entire partial data by repeatedly performing processing on each data unit in the partial data, for example, sequentially changing the compression target starting from the first data unit. The processing by each compression processing unit 702 is performed in parallel, for example, in the dimension of the batch number (B), thereby improving the efficiency of the compression processing. Furthermore, since the results of multiple prediction models are integrated to calculate the occurrence probability, the occurrence probability can be calculated with high accuracy, improving compression efficiency.

[0053] The compressor 70 compiles the bit streams (groups of bit strings corresponding to second blocks) for the respective partial data obtained by the multiple compression processing units 702 in a state in which the data ranges corresponding to the respective partial data can be identified, and sets them as compressed data of the data to be compressed. Here, the storage processing unit is constituted by the compressor 70.

[0054] Next, the configuration and processing of the expander 71 (71A, 71B) will be described.

[0055] FIG. 7 is a diagram illustrating the expander according to the first embodiment.

[0056] The decompressor 71 decompresses the compressed data compressed by the compressor 70. The decompressor 71 has a data synthesis unit 711 and a plurality of decompression processing units 712 (712A, 712B, etc.). Each decompression processing unit 712 is configured by each core 62 (62A, 62B) of the parallel processing device 61 (61A, 61B) executing a program in the memory 63 (63A, 63B). Here, the first block corresponding security code creation unit is configured by the decompressor 71 executing a program.

[0057] The data synthesis unit 711 generates pre-compression data by synthesizing partial data made up of data units decompressed by each decompression processing unit 712. In this embodiment, the data synthesis unit 711 synthesizes partial data of batch number B when compressing the compressed data. Note that in the decompressor 71, the same number of decompression processing units 712 as the batch number B are used for processing, and each decompression processing unit 712 performs decompression processing on the respective partial data.

[0058] The decompression processing unit 712 includes a pre-encoding unit 713 , a probability calculation unit 714 as an example of a second probability calculation unit, and an entropy decoding unit 715 .

[0059] The pre-encoding unit 713 performs the same processing as the pre-encoding unit 703. The probability calculation unit 714 performs the same processing as the probability calculation unit 704.

[0060] The entropy decoding unit 715 receives the occurrence probability Pb,i output from the probability calculation unit 714 and a bit string corresponding to the data unit Db,i to be decoded in the compressed data, and decodes the bit string to output the data unit Db,i.

[0061] Each decompressor 712 entropy-decodes the entire bit string of the entropy-encoded partial data by repeatedly performing the process for each data unit in the partial data, for example, by sequentially changing the decompression target starting from the first data unit. The processes by each decompressor 712 are performed in parallel, for example, in the dimension of the batch number (B), which can improve the efficiency of the decompression process.

[0062] Next, the processing operation of the compression process by the compressor 70 will be described.

[0063] Fig. 8 is a flowchart of the compression process by the compressor according to the first embodiment. The compression process shown in Fig. 8 is executed by each compression processing unit 702 of the compressor 70, and can be realized by, for example, a thread running on the core 62B of the parallel processing device 61B.

[0064] The compression processing unit 702 of the compressor 70 determines whether or not global learning is necessary (S11). Whether or not global learning is necessary is determined based on the total compression effect, for example, combining the effect of highly compressing the data to be compressed by global learning of a new model and the overhead of the storage capacity for saving the newly learned model.

[0065] As a result, if it is determined that holistic learning of the prediction model 704a or the integrated model 704b is necessary (S11: Y), the compression processing unit 702 performs holistic learning processing (S12) and uses this model in the prediction model processing and integrated model processing used in subsequent processing. The holistic learning processing is a process of learning using data that is close to each other in the logical address space based on the hypothesis that data that is close to each other in the logical address space have similar characteristics in the occurrence probability of data symbols, and saving the learned model. This learning data may be partial data of the data to be compressed that is specified when this processing flow is started, or other data may be used. For example, the learning data may be randomly sampled from data stored in the storage device based on a predetermined rule. Furthermore, the learned model is saved as model information 120d in the block data storage information 120 in association with the compressed data. Alternatively, reference information of other corresponding models (models corresponding to other block data that have already been learned) may be stored (i.e., the already learned model may be reused) to reduce the processing time for holistic learning and the data volume of the block data storage information. Learning may be sped up using techniques such as transfer learning and meta-learning.

[0066] On the other hand, if it is determined that overall learning is not necessary and the model is not stored in memory 63B (S11:N-1), the compression processing unit 702 performs a model load process to load the model into memory 63B (S12-2), and if the model is stored in memory 63B (S11:N-2), the compression processing unit 702 proceeds to step S12-3.

[0067] Next, the compression processing unit 702 causes the processor 53B to execute a weighting matrix quantization process (see FIG. 18) for quantizing the weighting matrix used in the model (S12-3).

[0068] Next, the compression processing unit 702 performs one-hot encoding as necessary on the data used to calculate the occurrence probability of the data unit to be compressed (data unit to be compressed: first, the first data unit, and thereafter, the next data unit) of the partial data it is responsible for, inputs the data into multiple prediction models 704a, and performs prediction model processing to calculate multiple occurrence probabilities using the multiple prediction models (S13).

[0069] Next, the compression processing unit 702 inputs the plurality of pieces of prediction information calculated in the prediction model processing into the integrated model 704b, and performs integrated model processing to calculate the occurrence probability of the data unit to be compressed (S14).

[0070] Next, the compression processing unit 702 determines whether or not learning of the prediction model 704a and the integrated model 704b is necessary (S15). For example, it may be determined that learning is necessary when the determination of step S15 has been made a predetermined number of times.

[0071] As a result, if it is determined that learning of the integrated model 704b is necessary (S15: Y), the compression processing unit 702 performs learning processing of the prediction model 704a and the integrated model 704b (S16), and then proceeds to step S15. The data used for learning is data that will have been decoded at this point in time when decompression is assumed, and pre-prepared data and / or historical data may be used. This learning processing results in correction of the weights held by the multiple prediction models 704a and the integrated model 704b.

[0072] On the other hand, if it is determined that learning of the integrated model 704b is not necessary (S15: N), the compression processing unit 702 advances the process to step S17.

[0073] In step S17, the compression processing unit 702 determines whether the prediction model actually used among the multiple prediction models needs to be updated. For example, if the weights for each prediction model are changed as a result of learning the integrated model 704b, and the weights for the prediction models become equal to or smaller than a predetermined threshold, the compression processing unit 702 may determine that the prediction model to be used needs to be updated.

[0074] As a result, if it is determined that the prediction model to be used needs to be updated (S17: Y), the compression processing unit 702 executes an update process for the prediction model to be used (S18) in which prediction models whose corresponding weights are less than a predetermined value are set not to be used, and proceeds to step S19.

[0075] On the other hand, if it is determined that the prediction model to be used does not need to be updated (S17: N), the compression processing unit 702 advances the process to step S19.

[0076] In step S19, the compression processing unit 702 performs entropy encoding processing using the occurrence probability output in step S14 and the data unit to be compressed, thereby converting the data unit to be compressed into an entropy-encoded bit string.

[0077] Next, the compression processing unit 702 determines whether all data units of the partial data have been encoded (S20), and if all data units have not been encoded (S20:N), the processing proceeds to step S13 to process the next data unit.

[0078] On the other hand, if all data units have been encoded (S20: Y), the process ends. After this, the compressor 70 compiles bit strings obtained by encoding the partial data converted by each compression processing unit 702, and stores the resulting compressed data in a predetermined storage location.

[0079] In the above description, an example has been shown in which the predictive model processing, integrated model processing, learning processing, and entropy coding processing are iterated in order for each data unit. However, the execution unit and timing of these processes can be changed. For example, in the predictive model processing and integrated model processing, since all of the original data exists at the time of compression, the processing can be performed collectively in parallel in the length (N) direction, as shown in FIG. 12. Furthermore, when a method such as ANS (Asymmetric Numeral Systems) is used in the entropy coding processing (S19), the processing must be performed in the reverse order of the encoding at the time of decoding. Therefore, during compression, the occurrence probabilities of the results predicted in the predictive model processing and integrated model processing are stored in the length (N) direction. Once a series of calculations in the length (N) direction is completed, the entropy coding processing is performed collectively in reverse order for the stored occurrence probabilities. Furthermore, in the learning processing, in addition to the method of performing the processing in the length (N) direction (learning using data prior to the current iteration in the length (N) direction), for example, multiple data blocks to be compressed can be sampled and used as the data to be learned. In this case, the training data and schedule for compression may be determined by sampling using deterministic pseudorandom numbers or the like so that the training data can be uniquely identified and is not included in the data before decompression. Furthermore, the processing order of data units in the length (N) direction does not need to be the original order, and they may be processed in a state changed using a reversible algorithm. Alternatively, a method for restoring the original data from a noisy state, such as a diffusion model, may be used, with the iteration unit corresponding to one step of the diffusion model. In either case, when the execution unit and timing are changed as described above, the same execution unit and timing are used during decompression to ensure the same probability of occurrence of the predicted result, thereby making the data decodable.

[0080] Next, the processing operation of the decompression process by the decompressor 71 will be described.

[0081] Fig. 9 is a flowchart of decompression processing by the decompressor according to the first embodiment. Note that the decompression processing shown in Fig. 9 is processing executed by each decompression processing unit 712 of the decompressor 71, and can be realized by, for example, a thread running on the core 62B of the parallel processing device 61B.

[0082] The decompression processing unit 712 of the decompressor 71 determines whether model loading is necessary (S21). Whether model loading is necessary is determined to be necessary, for example, if parameters and references of the model itself are stored in association with the compressed data as model information 120d in the block data storage information 120, and otherwise determined to be unnecessary.

[0083] As a result, if it is determined that model loading of the prediction model 704a or the integrated model 704b is necessary (S21: Y), the model is placed in the memory 63B of the parallel processing device 61B as model information 120d based on the parameters and references of the model itself corresponding to the compressed data (S22). Furthermore, the memory 63B of the parallel processing device 61B may be used like a cache. In that case, if it is determined in S21 that a model has already been loaded in the memory 63B used as a cache (frequently used models may be placed in the memory 63B like a cache), model loading can be omitted, so it can be determined that model loading is unnecessary, and the time required to load the model can be efficiently reduced.

[0084] On the other hand, if it is determined that model loading is not necessary (S21: N), the decompression processing unit 712 advances the process to step S22-2.

[0085] Next, the decompression processing unit 712 causes the processor 53B to execute a weighting matrix quantization process (see FIG. 18) for quantizing the weighting matrix used in the model (S22-2).

[0086] Next, the decompression processing unit 712 performs one-hot encoding on the data used to calculate the occurrence probability of the data unit to be decompressed (data unit to be decompressed: first, the first data unit, and thereafter, the next data unit) of the partial data it is responsible for, inputs the data to multiple prediction models 714a, and performs prediction model processing to calculate multiple pieces of prediction information using the multiple prediction models (S23).

[0087] Next, the decompression processing unit 712 inputs the plurality of pieces of prediction information calculated in the prediction model processing into the integrated model 714b, and performs integrated model processing to calculate the occurrence probability of the data unit to be decompressed (S24).

[0088] Next, the decompression processing unit 712 determines whether or not learning of the prediction model 714a and the integrated model 714b is necessary (S25). For example, it may be determined that learning is necessary when the determination in step S25 has been made a predetermined number of times.

[0089] As a result, if it is determined that learning of the prediction model 714a and the integrated model 714b is necessary (S25: Y), the decompression processor 712 performs learning processing of the prediction model 714a and the integrated model 714b (S26), and then proceeds to step S27. The data used in the learning processing is data that has been decoded at this point, and may be data prepared in advance and / or historical data. This learning processing results in correction of the weights held by the prediction model 714a and the integrated model 714b.

[0090] On the other hand, if it is determined that learning of the prediction model 714a and the integrated model 714b is not necessary (S25: N), the decompression processing unit 712 advances the process to step S27.

[0091] In step S27, the decompression processing unit 712 determines whether the prediction model actually used among the multiple prediction models needs to be updated. For example, if the weights for each prediction model are changed as a result of learning of the integrated model 714b, and the weights for the prediction models become equal to or smaller than a predetermined threshold, the decompression processing unit 712 may determine that the prediction model to be used needs to be updated.

[0092] As a result, if it is determined that the prediction model to be used needs to be updated (S27: Y), the decompression processing unit 712 executes an update process for the prediction model to be used (S28) in which prediction models whose corresponding weights are less than a predetermined value are set not to be used, and the process proceeds to step S29.

[0093] On the other hand, if it is determined that the prediction model to be used does not need to be updated (S27: N), the decompression processing unit 712 advances the process to step S29.

[0094] In step S29, the decompression processing unit 712 performs entropy decoding processing using the occurrence probability output in step S24 and the entropy-encoded bit string corresponding to the data unit to be decompressed, thereby decoding the entropy-encoded bit string into the data unit before encoding.

[0095] Next, the decompression processing unit 712 determines whether all data units of the partial data have been decoded (S30), and if all data units have not been decoded (S30:N), the process proceeds to step S23 to process the next data unit.

[0096] On the other hand, if all data units have been decoded (S30: Y), the decompression processing unit 712 ends the process. After this, the decompressor 71 collects the partial data converted by each decompression processing unit 712 and stores them as decompressed data in a predetermined storage location.

[0097] As described above, by consistently executing each process required for decompression of a plurality of data units by decompression processing unit 712 in a loop on a thread running on core 62B of parallel processing device 61B, for example, it is possible to reduce the processing time required for communication required for command transmission between processor 53B and core 62B of parallel processing device 61B and for thread startup, thereby enabling high-speed processing. Note that the calculation of the occurrence probability of data units, learning process, usage model update process, and other determination processes in steps S23 to S30 operate in the same way as the calculation of the occurrence probability of data units in compression processing, so the same occurrence probability can be calculated for the same data units and they can be decoded appropriately.

[0098] 8 and 9, when the learning and usage prediction model update processes of S16, S18, S24, and S26 are not performed, or when the states of each partial data are managed independently, recompression and decompression processes for updating data are possible for each partial data of the data to be compressed or decompressed, based on the compressed size 120c of the block data storage information 120. This makes it possible to partially update and decompress data in partial data units, without decompressing all the data in the block, even when the data has already been highly compressed as a large block (e.g., several MB to several GB), thereby reducing the amount of required computational resources.

[0099] Next, the processing operation of the backup process in the computer 101B will be described.

[0100] FIG. 10 is a flowchart of the backup process according to the first embodiment.

[0101] This backup process is executed when the processor 53B of the computer 101B receives from the computer 101A a compression command to which a group of data blocks to be backed up (first compressed data) compressed by the compressor 72A and a group of security codes are added. The processor 53B causes the decompressor 73B to execute decompression processing on one data block to be backed up (first block unit) (S101). Note that the compression by the compressor 72A is lower than the compression by the compressor 70B.

[0102] Next, the processor 53B checks the decompressed data using the added security code, and performs processing such as error detection and correction (S102).

[0103] Next, the processor 53B determines whether or not the decompression process has been performed on the number of blocks (number of target blocks) of a predetermined size (second size) to be compressed in the next compression process (S103).

[0104] As a result, if the decompression process has not been executed on the target number of blocks (S103: N), the processor 53B advances the process to step S101 and executes the process on the next data block.

[0105] On the other hand, if the decompression process is being performed on the blocks of the target number of blocks (S103: Y), the processor 53B causes the compressor 70B to perform compression process on the data of the blocks (second block units) of the target number of blocks (S104). The compression process by the compressor 70B is the compression process shown in FIG. 8, and the compression by the compressor 70B is more highly compressed than the compression by the compressor 72. Note that the throughput of the compression process can be improved by dividing the data of the target number of blocks into multiple data units and compressing them in parallel, for example, in the core 62B of the parallel processing device 61B. Furthermore, the size of the data to be compressed (second size) is larger than the size (first size) of the data compressed by the compressor 72, so that the compression efficiency is good.

[0106] Next, the processor 53B generates a guarantee code for data of blocks (second block units) equal to the number of target blocks (S105), stores the compressed data 120a, the guarantee code 120b, the compressed size 120c, and the model information 120d as block data storage information 120 in the persistent storage device 54B, and updates the logical-to-physical conversion table 110 according to the stored content (S106). Here, since the guarantee code is generated for data of the second size, the storage area required for storing the guarantee code can be reduced. Furthermore, when a plurality of data blocks of the first block unit are combined to form the second block unit, zero-padded data may be prepared for areas of the first block unit that were not originally assigned physical addresses, or data required to restore the logical-to-physical conversion table 110 of the first block unit may be stored together with the block data storage information. If the logical-to-physical conversion table 110 needs to be stored in the persistent storage device 54, the logical-to-physical conversion table 110 itself may also be compressed in a similar manner as a target of the compression process (S104), thereby reducing the capacity used by the persistent storage device 54 and improving efficiency. Furthermore, the guarantee code 120b and the model information 120d themselves may also be subjected to the compression process (S104) and compressed in a similar manner. The compressed size 120c is obtained as a result of the compression process (S104), so the compression efficiency may be improved by performing a process similar to the compression process (S104) again or by compressing it using another general compression method.

[0107] Next, the processor 53B determines whether or not all data blocks to be compressed have been processed (S107). As a result, if all data blocks to be compressed have not been processed (S107: N), the processor 53B proceeds to step S101 and executes processing on the remaining data blocks.

[0108] On the other hand, if all data blocks to be compressed have been processed (S107: Y), the processor 53B notifies the requesting computer 101A that the backup has ended, and ends the process.

[0109] Next, the processing operation of the restore process in the computer 101B will be explained.

[0110] FIG. 11 is a flowchart of the restore process according to the first embodiment.

[0111] This restore process is executed when processor 53B of computer 101B receives a restore command from computer 101A, to which is added information indicating a data block group to be restored. Note that the data blocks of this data block group are blocks in which data of the first size has been compressed.

[0112] First, the processor 53B causes the decompressor 71B to execute decompression processing on one data block of the second size that includes the data block group to be restored corresponding to the restore command (S111). As in the case of compression, the data of the target blocks is divided into multiple data units, and for example, the core 62B of the parallel processing device 61B performs decompression processing collectively in parallel, thereby improving the throughput of the decompression processing.

[0113] Next, the processor 53B checks the decompressed data using the added security code, and performs processing such as error detection and correction (S112).

[0114] Next, the processor 53B causes the compressor 72B to perform a compression process on the block data of the first size in the decompressed data (S113).

[0115] Next, the processor 53B generates a security code for the block data of the first size (S114), stores the block data and the security code in the persistent storage device 54B, and updates the logical-to-physical conversion table 110 for the block of the first size according to the stored contents (S115).

[0116] Next, the processor 53B determines whether or not compression processing has been performed on the number of blocks of the first size (the number of target blocks) included in the block data of the second size (S116).

[0117] As a result, if the compression process has not been performed on the target number of blocks (S116: N), the processor 53B advances the process to step S113 and continues the process on the remaining first size block data.

[0118] On the other hand, if compression processing has been performed on the number of blocks corresponding to the target number of blocks (S116: Y), the processor 53B determines whether or not processing has been completed on all data blocks including the data block group to be restored (S117).

[0119] As a result, if the process has not been completed for all data blocks including the data block group to be restored (S117: N), the processor 53B advances the process to step S111 and executes the process for the remaining data blocks.

[0120] On the other hand, if processing has been completed for all data blocks containing the data block group to be restored (S117: Y), the processor 53B sends a compressed data group consisting of compressed data for each block data of the first size (each first block unit) and a security code group consisting of the corresponding security codes to the requesting computer 101A, and terminates processing.

[0121] Next, the processing operation of the prediction model processing in the computer 101B will be described.

[0122] FIG. 12 is a diagram illustrating the prediction model processing according to the first embodiment.

[0123] This process is an example of prediction model 704a and prediction model 714a (executed by compression processing unit 702 and decompression processing unit 712) (simply referred to as prediction model). This prediction model process is an example of a case where prediction model processing is configured using a neural network, and by inputting data (prediction data) used to predict a data unit (prediction target data unit) whose occurrence probability is to be predicted, a feature map (data to be input to the integrated model) of a position corresponding to the prediction target data unit is output. This process can be executed in parallel as threads for each dimension of the batch number (B) running on core 62B of parallel processing device 61B, for example.

[0124] The data input here is an array of the number of batches (B) x number of channels (C) x length of partial data (N). Note that when training a prediction model or compressing, all data to be compressed is retained, so multiple pieces of data to be predicted (N directions) can be processed together in parallel after aligning their respective causal relationships. Figure 12 explains this case of collective processing. On the other hand, when decompressing (decoding), data that has not yet been decoded is not retained, and the operation in this case is explained in Figure 13.

[0125] This prediction model processing is mainly composed of functional components: input data shift processing (S121), downscale block (DB) (S122), residual block (RB) (S123), and upscale block (UB) (S124) (collectively referred to as Scale Causal Blocks (SCB) hereafter).

[0126] The input data shift process (S121) is a process of padding one unit at the beginning of the input data in the length (N) direction and deleting one unit at the end. By this process, the input data is shifted one unit back in the length (N) direction, and the subsequent process becomes a process of predicting the next input data from the past data.

[0127] The downscaling block (S122) processes the input data using a convolutional neural network, dividing the data in the channel dimension (C) and then reducing the data in the length (N) direction. Specifically, the data being processed is first padded at the beginning of the length (N) direction with a predetermined size (e.g., K-1 pieces) (indicated by black "0"s in Figure 12), and then processed using a convolutional neural network (Convolution in the figure). In this example, since the length (N) direction is one-dimensional, a one-dimensional convolutional neural network is applied. The kernel size of the convolutional neural network is K, and the stride width is 1. The kernel size can be set to, for example, K=2, from the perspective of the range of the receptive field relative to the number of weights in the neural network. Then, an activation function (Activation in the figure) is applied to the data being processed (the output result of the convolutional neural network). The activation function may be a Scaled Exponential Linear Unit (SELU) or a Rectified Linear Unit (ReLU). The data being processed is then split in the channel dimension (C direction) (Split in the figure). A portion of this split data is used as direct input to the corresponding upscale block (S124) without going through the downscale block (S122) process. This prevents the loss of fine-grained information due to the downscale block (S122) reducing the length (N) direction, thereby further improving the accuracy of the prediction model process. Furthermore, a memory model (Mem. in the figure) may be applied to a portion of this split data. The memory model is a process for improving the accuracy of probability prediction by storing historical information in the length direction. The memory model may be configured as a residual block, as shown in the downscale block (S122) diagram. Then, the data being processed is reduced in the length (N) direction (Down in the figure). This reduction process may be performed by, for example, replacing two consecutive data in the length (N) direction in the channel (C) direction, as shown in FIG.Alternatively, processing may be performed using a convolutional neural network with a wider stride width, or a method such as pooling. S122 in FIG. 12 shows an overview of the above processing, where the numbers in the figure represent numbers corresponding to the order of the length (N) of the input data units, and in the processing after convolution, the numbers represent the order of the input data and the causal relationship between the processing results corresponding to those numbers (the maximum value of the numbers corresponding to the information used in the processing). In the above processing, the causal relationship between the length (N) of the input data is preserved. In other words, since information after a certain target data unit is not used in the processing, it can be seen that this is a valid prediction process.

[0128] The residual block (S123) processes data output from the previous functional component using a dilated convolutional neural network and adds the processed data to the data before processing. Specifically, the data being processed is first padded at the beginning of the length (N) direction by a predetermined size (e.g., d*(K-1) pieces) before processing the data using the dilated convolutional neural network. In this example, since the length (N) direction is one-dimensional, a one-dimensional convolutional neural network is applied. Note that the kernel size of the dilated convolutional neural network is K (e.g., K=2), the stride width is 1, and the dilation size d is Kx-1. Here, x is the level of the residual block. When r residual blocks are configured as shown in Figure 12, it is specified as a natural number from 1 to r in the order of processing. Increasing the dilation size at each hierarchical level as described above allows for efficient expansion of the receptive field in the length (N) direction. Then, an activation function is applied, as in the downscale block (S122). The processed data and the unprocessed data are then added together to produce the output result of the residual block (S123). In addition to this addition process, attention processing (such as self-attention) may be introduced. As shown in S123 of FIG. 12, the above process preserves the causal relationship along the length (N) of the input data, similar to the downscale block (S122), and thus serves as a predictive process.

[0129] The upscale block (S124) scales the data output from the previous functional component in the length (N) direction, combines the data divided by the corresponding downscale block (S122) in the previous stage, and outputs the data processed by the convolutional neural network. Specifically, the data being processed is first padded at the beginning of the length (N) direction with a predetermined size (e.g., K-1 pieces) and then scaled in the length (N) direction (Up in the figure). For example, as shown in Figure 12, the process may be the opposite of Down in the downscale block (S122), by dividing the data into two pieces in the channel (C) direction and replacing them with two pieces of data consecutive in the length (N) direction. Alternatively, processing may be performed using a transposed convolutional neural network with an increased stride width. In addition, for the divided data in processing in the corresponding downscale block (DBx) having the same length in the length (N) direction, the data is padded at the beginning of the length (N) direction by a predetermined size (e.g., K-1 pieces), and data is created that is combined with the data in processing expanded in the length (N) direction and the channel (C) direction. This data is processed using a convolutional neural network. In this example, since the length (N) direction is one-dimensional, a one-dimensional convolutional neural network is applied. Note that the kernel size of the convolutional neural network is K (e.g., K=2), and the stride width is 1. Then, as with the downscale block (S122), an activation function is applied to the data in processing (the output result of the convolutional neural network) to obtain the resulting data. Note that in the above process, as shown in S124 of Figure 12, the causal relationship in the length (N) direction of the input data is preserved, just like with the downscale block (S122), and it can be seen that this is a valid prediction process.

[0130] As a method for configuring the functional components described above, for example, as shown in FIG. 12, multiple (s) downscaled blocks (S122) (DB1-DBs) may be processed first, followed by multiple (r) residual blocks (S123) (RB1-RBr), and then multiple (s) upscaled blocks (S124) (UB1-UBs). Processing in this manner reduces the number of elements in the length (N) direction from B × C × N input data to B × C1 × N / 2 and B × C2 × N / 4. This effectively increases the number of internal channels C1-Cs, thereby improving the efficiency and speed of predictive model processing. Note that C1-Cs may be increased in conjunction with the reduction in length (N) when processing the downscaled blocks (S122), and decreased in conjunction with the increase in length (N) when processing the upscaled blocks (S124). Furthermore, by combining the downscaled block (S122), residual block (S123), and upscaled block (S124), a receptive field with a length of Ks+r can be obtained. In other words, the amount of data (prediction data) used to predict the data unit (prediction target data unit) whose occurrence probability is to be predicted can be efficiently increased, thereby improving prediction accuracy for a given amount of processing. Furthermore, the residual block (S123) may be processed between multiple downscaled blocks (S124) or multiple upscaled blocks (S124).

[0131] Furthermore, in a configuration using s downscaled blocks (S122) and s upscaled blocks (S124), the convolutional layers of each of the w blocks (RBs-w to RBs and UBs-w to UBs) may be weight-shared (the convolutional layer of RBs-w to RBs is made the same as the parameters of the convolutional layer of RBs-w, and the convolutional layer of UBs-w to UBs is made the same as the parameters of the UBs-w convolutional layer). Reducing the number of parameters of the prediction model can reduce the amount of data in the model information 120d of the block data storage information 120, and therefore, by setting w appropriately, it is possible to improve compression efficiency.

[0132] Additionally, the kernel size (K) may be increased depending on the layer depth (s) of the downscale block (S122) or the upscale block (S124) to further widen the receptive field.

[0133] Additionally, for multidimensional data such as image data, the length (N) direction may be multidimensional. In this case, processing may be performed in the same way using a convolutional neural network of the corresponding dimension.

[0134] Next, the data flow of the prediction model processing and the integrated model processing in the computer 101B will be described.

[0135] FIG. 13 is a diagram illustrating the data flow of the prediction model processing and the integrated model processing according to the first embodiment.

[0136] The integrated processing model 1300 is an example of an implementation method for the prediction model 704a and the prediction model 714a (executed by the compression processing unit 702 and the decompression processing unit 712) (simply referred to as the prediction model) and the integrated model 704b and the integrated model 714b (executed by the compression processing unit 702 and the decompression processing unit 712) (simply referred to as the integrated model). The prediction model processing and the integrated model processing are the prediction model processing and the integrated model processing described in FIG. 12 , and an overall overview of how data is processed within them will be described. The initially input data is an array of the number of batches (B) × the number of channels (C) × the length of partial data (N), as in FIG. 12 . However, in this figure, the dimension of the number of batches (B) is omitted, and the horizontal axis represents the length of partial data (N), the vertical axis represents the order of each process, and the vertical axis of each process represents the number of channels (C). This processing can be executed in parallel as threads for each dimension of the number of batches (B) running on the core 62B of the parallel processing device 61B.

[0137] This example shows a process for predicting the eighth data item in the training data, starting from the beginning. In this case, the eighth data item is the data item whose occurrence probability is to be predicted (data item to be predicted), and the data used for this prediction (data for prediction) corresponds to the calculation portion 1301 and the reference portion 1302 in the figure. The reference portion 1302 also includes the results of calculations (for data items that have already been predicted) in the past (for example, stored as a cache).

[0138] In this example, the prediction model processing is composed of two downscale blocks (S122) (DB1, DB2), one residual block (S123) (RB1), and two upscale blocks (S124) (UB1, UB2), and the integrated model processing is composed of a linear neural network (similar to a convolution with K=1) and a Softmax function. The prediction model processing outputs a feature map (data to be input to the integrated model) of the position corresponding to the data unit to be predicted, and the integrated model processing ultimately outputs a prediction result of the occurrence probability of the data unit to be predicted.

[0139] The numbers in the figure represent the numbers corresponding to the order of the input data units in the length direction (N), and each process represents the number of the input data order and the causal relationship of the processing result corresponding to that number (the maximum value of the number corresponding to the information used in the process). As a result of the integrated model processing, the causal relationship in the length direction (N) of the input data is preserved. In other words, since information after a certain target data unit is not used in the processing, it can be seen that the prediction process is valid.

[0140] The prediction model processing proceeds in the same manner as the method already explained in Fig. 12, but in this example, in order to proceed with the calculation of calculation portion 1301 in the figure, the data of reference portion 1302 is cached with duplicates removed when calculating past occurrence probabilities (up to the seventh occurrence), and this is used to efficiently proceed with the calculation of calculation portion 1301. In this way, even during decoding, when it is difficult to perform calculations all at once in the length (N) direction, efficient iteration processing that prevents duplicate calculations enables high-speed processing.

[0141] Regarding the integrated model processing, this example explains the case where one prediction model is used, but multiple prediction models may be used in the same way. In that case, the input dimension of the neural network of the integrated model increases. Also, in this example, the Softmax function is used to reduce the input dimension of 1 bit (=2 1 However, for example, the output may be one in the channel direction, converted to a value between 0 and 1 using a Sigmoid function or the like, and treated as the probability of outputting the symbol "1." In that case, the value obtained by subtracting this probability from 1 is the probability of outputting the symbol "0."

[0142] Furthermore, as a method for training the neural network weights in the prediction model processing and the integrated model processing described above, since the prediction model processing and the integrated model processing are configured as differentiable processes, the prediction accuracy may be improved by calculating relative entropy (or negative log likelihood, etc.) between the occurrence probability of each symbol finally output by the integrated model and training data (e.g., a value corresponding to the occurrence probability of each symbol expressed in one-hot format), and then training the model end-to-end using an error backpropagation method or the like to minimize this value. Regarding the timing of this training, as shown in FIGS. 8 and 9, training may be performed in sync with the progress of the prediction processing for each data unit, or a pre-trained model may be used. Furthermore, as in the overall training process (S12), the prediction model and the integrated model may be trained periodically based on data stored in the memory 52B, memory 63B, persistent storage device 54B, etc. of the computer 101B. In this case, the trained model can be restored by saving it in the persistent storage device 54B or the like, and recording the correspondence between the model and the data compressed using it (model information 120d) together with the compressed data 120a in the block data storage information 120. Also, by distinguishing the model to be used based on identification information (code) that indicates the characteristics of the data, a trained model dedicated to similar data can be prepared, thereby improving compression efficiency.

[0143] Next, the algorithm for predictive model processing in computer 101B will be described.

[0144] FIG. 14 is a diagram illustrating an algorithm for the prediction model processing according to the first embodiment.

[0145] FIG. 12 describes the SCB components (downscale block, upscale block, etc.) that make up the prediction model. The module description in FIG. 12 uses a convolution kernel as the neural network, and learning processing is generally performed by parallelizing in the length (N) direction (dividing and processing data by kernel size). In this case, because a large amount of data is input to the neural network in the length (N) direction, the multiplicity in the batch (B) direction is set relatively small (e.g., 16). This is an appropriate processing method for the overall learning processing (S12) in compression processing. However, when operating as inference processing such as prediction model processing (S13 and S23), it is an iterative process in symbol units, so the batch multiplicity is set higher (e.g., several thousand or more). As described in FIG. 13, by using a cache of intermediate data output by each component, calculation processing can be performed partially in the length (N) direction, improving memory usage efficiency and significantly increasing the multiplicity in the batch (B) direction (e.g., several thousand or more). In the following, the inference process will be explained by specifically showing it as an algorithm procedure in FIG.

[0146] First, the overall SCB algorithm 1401 can be broadly divided into initialization processing (initialize in the figure) and inference processing (inference in the figure). The initialization processing is performed only the first time the prediction model processing (S13 and S23) is executed on the target data (B × C × N array) obtained by dividing the target data into multiple partial data. The inference processing is performed N times (N iterations) as the prediction model processing (S13 and S23) using a single piece of target data acquired in the length (N) direction (represented by x in the figure, a B × C array). Note that the notation [X, Y] in the figure represents the shape of the X × Y array (tensor), and [ ] represents an empty list structure. The initialization processing is performed on each component (block in the figure) in the overall component configuration of the SCB (represented by blocks in the figure, with each component, such as DB1, DB2, UB2, and UB1, listed in processing order). Also, in the inference process, as shown in algorithm 1401 in the figure, the inference process is executed for each component in order.

[0147] Next, we will explain the algorithm 1402 of the downscale block. This diagram illustrates an example of the algorithm when kernel size K=2. The process is broadly divided into initialization processing (initialize in the diagram) and inference processing (inference in the diagram). The initialization processing sets initial values ​​for the variables (xp and sp) used as two types of caches. Note that the bold "0" in the diagram indicates an array initialized to zero. The inference processing takes as input the array (x) of data to be processed (B × C_in) and a list (s) of data arrays to be input to each subsequent upscale block, and outputs an array (B × C_out) (the values ​​of C_in and C_out may be different for each component). The basic operation of the inference processing is as shown in the algorithm in Figure 14. Since the outline of the operation has already been explained in Figure 13, we will omit the details. Only the distinctive features of the inference processing will be explained below. Linear_from_conv in the figure is a process in which the convolution process described in FIG. 12 is replaced with a fully connected layer. Specifically, the convolution process can be considered as a fully connected layer with K×C_in as input and C_out as output, so the parameters of the convolution kernel are converted into parameters of the fully connected layer before processing. This allows for omitting intermediate processes such as array dimension conversion, enabling high-speed processing. Also, as shown in algorithm 1402, memory model processing (memory in the figure) may be performed.

[0148] Next, the algorithm 1403 for the upscale block will be explained. In this diagram, an algorithm for the case where kernel size K=2 is described as an example. There are two main processes: initialization processing (initialize in the diagram) and inference processing (inference in the diagram). The initialization processing sets initial values ​​for the variables (xp1, xp2, and sp) used as two types of cache. Furthermore, the inference processing takes as input the array (x) of data to be processed, B×C_in, and a list (s) of arrays of data to be input to each upscale block, and outputs the array B×C_out.

[0149] In this way, by using a cache of intermediate data output by each component, calculation processing can be carried out partially in the length (N) direction (processing in array units of B × C), improving memory usage efficiency and significantly increasing the multiplicity in the batch (B) direction (e.g., several thousand or more), enabling high-speed processing.

[0150] Next, the configuration of the cache information table used for the prediction model processing in the computer 101B will be described.

[0151] FIG. 15 is a configuration diagram of a cache information table according to the first embodiment.

[0152] As shown in FIG. 14, the downscale block algorithm 1402 has internal variables x_p and s_p. Also, as shown in FIG. 14, the upscale block algorithm 1403 has internal variables x_p1, x_p2, and s_p. These internal variables are used in the prediction model processing within an iteration loop consisting of steps S13 to S20 in FIG. 8, which perform encoding processing on all data units of the partial data, and steps S23 to S30 in FIG. 9, which perform decoding processing on all data units of the partial data. Hereinafter, to perform efficient prediction model processing, this information (collectively referred to as cache information) is managed in a cache information table. FIG. 15 shows a cache information table when the kernel size is set to K=2. Also, hereinafter, the processing corresponding to linear_from_conv in the downscale block algorithm 1402 and the upscale block algorithm 1403 in FIG. 14 will be referred to as matrix multiplication processing, the processing corresponding to activation will be referred to as activation function processing, and the weight data used in linear_from_conv will be referred to as weight matrix. Here, the configuration of the neural network that performs the matrix multiplication operation corresponds to the conversion process (first conversion process (e.g., linear_from_conv in Figure 14)), and the configuration of the neural network that performs the activation function operation corresponds to the conversion process (second conversion process (e.g., activation in Figure 14)).

[0153] Here, the size of an N-row by M-column matrix is ​​expressed as [N, M]. The cache information table 1500 is a table for managing cache information, and includes an entry for each input matrix (matrix size: [B, C_in]) input to each layer. That is, one entry (one row) of the cache information table 1500 stores input information required to output one calculation part 1301 calculated in the layer in FIG. 13. An entry of the cache information table 1500 includes fields for an input layer number 1500a, a cache index 1500b, and input matrices 1500c, 1500d, 1500e, and 1500f (each matrix size: [B, C_in / 4]). The input layer number 1500a stores the number of the layer to which the input matrix corresponding to the entry is input. The cache index 1500b stores the ID of the row corresponding to the entry. Input matrices 1500c, 1500d, 1500e, and 1500f store the portions of the input matrix divided into four in the C_in direction.

[0154] The cache information table 1500 also includes a selector 1500g that specifies the row to be input next for each layer. During prediction model processing, each time processing of a layer corresponding to a specified entry is executed, the selector 1500g selects a row corresponding to the same layer so that the value of the cache index 1500b of the specified row cycles from 0 to 1 to 2 to 0 to...

[0155] Next, we will explain the process of writing an input matrix to an entry in the cache information table 1500. Here, we will explain an example of the process of writing the calculation part 1301 corresponding to the seventh symbol in the length direction in layer DB1 in the prediction model processing and integration processing model 1300 in Figure 13 to the cache information table 1500.

[0156] The calculation portion 1301 of DB1 corresponding to the seventh symbol in the length direction is used as the reference portion 1302 (input) of the four subsequent calculation portions 1301: input x of calculation portion 1301 of DB2 corresponding to the seventh symbol in the length direction, input x_p of calculation portion 1301 of DB2 corresponding to the ninth symbol in the length direction, input r of calculation portion 1301 of UB1 corresponding to the seventh symbol in the length direction, and input s_p of calculation portion 1301 of UB1 corresponding to the ninth symbol in the length direction.

[0157] First, the input to DB2 after splitting the calculation portion 1301 of DB1 corresponding to the seventh symbol in the length direction will be described. As described above, this calculation portion 1301 becomes the input x of the calculation portion 1301 of DB2 corresponding to the seventh symbol in the length direction and the input x_p of the calculation portion 1301 of DB2 corresponding to the ninth symbol in the length direction. In the prediction processing model and the integration processing model 1300, when processing the seventh symbol in the length direction, the data of the calculation portion 1301 of DB1 corresponding to the seventh symbol becomes the last partial data when the input matrix of DB2 is divided into four in the C_in direction. Therefore, this data is overwritten in the row of the cache information table 1500 that will be selected by the selector of DB2 when the seventh symbol in the length direction is reached; in this example, the fourth input matrix 1500f stores the last partial data of the row currently selected by the selector of DB2. Furthermore, in the prediction processing model and integration processing model 1300, when processing the ninth symbol in the length direction, the data in the calculation portion 1301 of DB1 corresponding to the seventh symbol becomes the second partial data from the front when the input matrix of DB2 is divided into four in the C_in direction. Therefore, this data is also overwritten in the row of the cache information table 1500 that the selector of DB2 selects when it reaches the ninth symbol in the length direction, in this example, the second input matrix 1500d of the row that the selector of DB2 will select next.

[0158] Next, the input to UB1 after splitting calculation portion 1301 of DB1 corresponding to the seventh symbol in the length direction will be described. This calculation portion 1301 becomes s_p or r in algorithm 1403 of the downscale block in Fig. 14. The data of this calculation portion 1301 is overwritten in input matrix 1500f of the row selected by the selector of UB1 when the seventh symbol in the length direction is reached, and input matrix 1500d of the row selected by the selector of UB1 when the ninth symbol in the length direction is reached, in the prediction processing model and integration processing model 1300.

[0159] When reading an input matrix from this cache information table 1500 in a matrix column multiplication operation, the values ​​of input matrices 1500c, 1500d, 1500e, and 1500f in the row specified by selector 1500g are simply combined and read as the input matrix.

[0160] In the cache information table 1500, the storage areas for the input matrices 1500c, 1500d, 1500e, and 1500f of the same row may be arranged at consecutive addresses in the memory 63B. When reading, the storage areas for the input matrices 1500c, 1500d, 1500e, and 1500f of the row designated by the selector 1500g may be read by specifying the leading area, thereby enabling sequential reading. For example, in the area in the memory 63B where the input matrices 1500c, 1500d, 1500e, and 1500f of the same row are stored, the input matrices may be stored as a fifth-order tensor type whose first and second dimension elements are the size divided and processed by the arithmetic processor 1603 (e.g., (B / 128) × 4 × (C_in / 4 / 128) × 128 × 128). This is expected to reduce memory access time and enable efficient prediction model processing. Furthermore, the cache information table 1500 is preferably stored in the memory 63B in order to process the prediction processing model consistently on the parallel processing device 61B, but may also be stored in the memory 52B.

[0161] Next, the configuration of the parallel processing device 61 in the computer 101B will be described.

[0162] FIG. 16 is a configuration diagram of a parallel processing device according to the first embodiment.

[0163] The parallel processing device 61 includes a parallel processing chip 1601 and a shared memory 1602. The parallel processing chip 1601 is, for example, a GPU chip. The shared memory 1602 is an example of a second memory, for example, a DRAM (Dynamic Random Access Memory).

[0164] The parallel processing chip 1601 has a control processor 1611, one or more arithmetic processors 1612 (1612a, 1612b, . . .), one or more high-speed memories 1613 (1613a, 1613b, . . .), and an intra-chip shared memory 1614. The parallel processing chip 1601 processes data loaded from the shared memory 1602 in parallel and stores the data in the shared memory 1602.

[0165] The control processor 1611 is, for example, a massively large number of thread management engine, which is a physical configuration. The control processor 1611 manages the processing executed by the parallel processing chip 1601. The control processor 1611 realizes parallel processing by dividing the processing among pairs of each arithmetic processor 1612 and high-speed memory 1613. The arithmetic processor 1612 is, for example, a computing unit cluster, which is a physical configuration. The arithmetic processor 1612 processes data loaded from the high-speed memory 1613 using a matrix calculation core group 1621 and an integer calculation core group 1622, which will be described later, and stores the calculation results in the high-speed memory 1613.

[0166] The high-speed memory 1613 is an example of a first memory, such as a register. The high-speed memory 1613 is connected to one arithmetic processor 1612 and can be accessed exclusively by this arithmetic processor 1612. The high-speed memory 1613 stores data required for processing by the arithmetic processor 1612, the results of operations by the arithmetic processor 1612, and the like. The intra-chip shared memory 1614 is, for example, an L2 (level 2) cache. The intra-chip shared memory 1614 is accessible by each arithmetic processor 1612. The intra-chip shared memory 1614 may be used as an intermediary to efficiently transfer data from the shared memory 1602 to each high-speed memory 1613.

[0167] The arithmetic processor 1612 has a matrix operation core group 1621 including multiple matrix operation cores, and an integer operation core group 1622 including multiple alignment operation cores. The matrix operation core group 1621 is, for example, a matrix multiplication specialized operation unit having a physical configuration. The integer operation core group 1622 is, for example, a general-purpose integer operation unit having a physical configuration.

[0168] The shared memory 1602 stores data and calculation results required for parallel processing.

[0169] Here, the matrix calculation core of the matrix calculation core group 1621 and / or the integer calculation core of the integer calculation core group 1622 and / or the control processor 1611 correspond to the core 62B, and the shared memory 1602, the intra-chip shared memory 1614, and the high-speed memory 1613 collectively correspond to the memory 63B.

[0170] The high-speed memory 1613, the on-chip shared memory 1614, and the shared memory 1602 may have different data read / write speeds and capacities. For example, the order of memory speed from fastest to slowest may be high-speed memory 1613, on-chip shared memory 1614, and shared memory 1602. Furthermore, the order of memory capacity from largest to smallest may be shared memory 1602, on-chip shared memory 1614, and high-speed memory 1613.

[0171] Next, the matrix multiplication process and activation function process executed in the prediction model process will be described.

[0172] 17 is a diagram illustrating the matrix multiplication process and the activation function process according to the first embodiment. In FIG. 17, the process performed by the entire parallel processing device 61 will be described.

[0173] First, the matrix multiplication process will be described. The input matrix 1801 input to the layer that performs the matrix multiplication process is [B, K×C_in], and each element is a 1-bit integer of 0 or 1. The input matrix 1801 is loaded from the input matrices 1500c, 1500d, 1500e, and 1500f of the entry corresponding to the layer in the cache information table 1500. In the matrix multiplication process, the quantized weight matrix 1802-1 (matrix for calculation) used for calculation of the input matrix 1801 is [K×C_in, C_out]. Here, the integer N_int is the integer precision of the layer, and each element of the quantized weight matrix 1802-1 is N_int bits. In this embodiment, the matrix calculation cores of the matrix calculation core group 1621 that perform matrix calculation processing are configured to perform calculations between matrices with 1-bit elements, so if the quantized weight matrix 1802-1 is considered to be a matrix with each bit position of each element being a 1-bit element, the quantized weight matrix 1802-1 becomes a quantized weight matrix 1802-2 (virtual matrix) of [K×C_in, N_int×C_out] with each element being 1 bit.

[0174] In the matrix multiplication operation, the matrix operation core group 162 performs an XOR (exclusive OR) operation for each element on all combinations of each row of the input matrix 1801 and each column of the quantized weight matrix 1802-2, and then performs a popcnt operation (an operation to count the number of elements that are 1) (S201) to obtain a matrix multiplication operation result 1803 (operation result), which is stored in the high-speed memory 1613. The matrix multiplication operation result 1803 is [B, N_int × C_out], and each element is, for example, 32 bits.

[0175] Next, the activation function processing will be described. The activation function processing uses a matrix multiplication result 1803, coefficient data 1804, and bias data 1807. The coefficient data 1804 is data generated when a weight matrix quantization processing (see FIG. 18) is executed, and is data indicating how many times a quantum bit is multiplied to obtain a quantized value. In other words, it is data that reflects the order of the elements of a quantized weight matrix 1802-1 for the data of each column of the matrix multiplication result 1803, and is a matrix of [1, N_int × C_out]. For example, the quantized weight matrix 1802-1 is determined in step S216, which will be described later. The bias data 1807 is data for biasing, and is a matrix of [1, C_out].

[0176] In the activation function processing, a Broadcast_multiplication process (S202) is performed on the matrix multiplication result 1803 and coefficient data 1804 to obtain coefficient-processed data 1805. In the Broadcast_multiplication process (S202), each row of the matrix multiplication result 1803 is multiplied by each column-wise element of the coefficient data 1804 to obtain each row of the coefficient-processed data 1805. Here, the coefficient-processed data 1805 is [B, N_int × C_out], and each element is, for example, 32 bits. This coefficient-processed data 1805 is a matrix resulting from the matrix multiplication result 1803.

[0177] Next, adjacent summation (S203) is performed on the coefficient-processed data 1805, taking the sum for every N_int columns in the column direction, to obtain summed data 1806 (a summed matrix). In the adjacent summation, for example, when N_int=2, the first and second columns of the coefficient-processed data 1805 are added together for each element in the row direction to obtain the first column of the summed data 1806. Similarly, the second column of the summed data 1806 is obtained from the third and fourth columns of the coefficient-processed data 1805, and similar processing is performed on the other columns. The summed data 1806 is a matrix of [B, C_out], and each element is, for example, 32 bits. This summed data 1806 corresponds to the result of performing an XOR operation and a popcnt operation on the input matrix 1801 and the quantized weight matrix 1802-1. Therefore, the processing results of the XOR operation and the popcnt operation between the configuration input matrix 1801 and the quantized weight matrix 1802-1 can be obtained by using a matrix operation core 1621a or the like that performs operations between matrices with 1-bit elements.

[0178] Next, Broadcast_summation processing (S204) is performed on the summation-processed data 1806 and bias data 1807. In Broadcast_summation processing (S204), each element of each row of the summation-processed data 1806 is added to the element of each row of the bias data 1807 to obtain each row. Thereafter, 1-bit processing (S205) is performed to convert each element of the result of Broadcast_summation processing (S204) to a 1-bit integer, thereby obtaining calculation result 1808 (processing result) and storing it in high-speed memory 1613. The calculation result 1808 is a matrix of [B, C_out], and each element is a 1-bit integer. In the 1-bit processing (S205), for example, if the processing result of Broadcast_summation processing (S204) is greater than K×C_in / 2, it may be set to 0, and if not, it may be set to 1.

[0179] According to this matrix multiplication operation and activation function operation, the operation result 1808 is a matrix [B, C_out], and each element is a 1-bit integer. Therefore, the total amount of data can be reduced and stored in the high-speed memory 1613. Furthermore, the write time required to write data from the high-speed memory 1613 to the shared memory 1602 or the like for subsequent processing can be reduced.

[0180] Next, the weighting matrix quantization process (S12-3, S22-2) will be described.

[0181] 18 is a flowchart of the weighting matrix quantization process according to the first embodiment. In the following description, each step of the weighting matrix quantization process will be described as being performed by the processor 53B, but at least a part of each step may be executed by the parallel processing device 61B.

[0182] First, the processor 53B executes a process for creating a look-up table (LUT) (S213). The LUT is a table that manages the calculation results of a layer for all input candidates that can be inputs to the layer to be managed. The LUT is created by calculating the calculation results for all input candidates that can be inputs to the layer to be managed by performing a matrix multiplication operation and an activation function operation on the layer. Note that the LUT may manage only the input candidates and calculation results for some of the operations on the layer, for example, the matrix multiplication operation.

[0183] The weight matrix used in the matrix multiplication process when creating the LUT may be executed using a weight matrix with elements of higher bits than the weight matrix used in the matrix multiplication process shown in FIG. 17. For example, a weight matrix with 8-bit elements may be used. For example, the LUT for DB1 may have an input value of 2 per batch. K Since there are various types, it can be created by executing DB1 processing for all of these input values ​​and saving the calculation results as a table. At this time, it is also possible to process each input value in batch direction so that they can be processed in parallel.

[0184] Furthermore, whether or not to create an LUT may be determined for each layer. Because creating LUTs for more layers increases the amount of memory 63B used and the processing time required to create the LUTs, the layer for which an LUT is to be created may be determined by comprehensively considering the amount of memory 63B used and the processing time. For example, LUTs may be created for DB1, DB2, DB3, and DB4, and LUTs may be created only for a portion of the matrix multiplication operations for UB1, UB2, UB3, and UB4. Creating an LUT reduces the number of matrix multiplication operations and activation function operations in the prediction model processing, thereby enabling high-speed prediction model processing. Furthermore, creating an LUT using a weight matrix with a high bit count reduces the reduction in compression rate due to quantization errors compared to when an LUT is not used.

[0185] In step S214, the processor 53B determines whether the integer precision for each layer is undetermined. Regarding whether the integer precision for each layer is undetermined, for example, if the integer precision for each layer is not stored in the model information 120d of the block data storage information 120, the processor 53B determines that the integer precision for each layer is undetermined, and if the integer precision is stored, the processor 53B determines that the integer precision has been determined.

[0186] As a result, if it is determined that the integer precision for each layer has not been determined (S214: Y), the processor 53B executes an integer precision determination process to determine the integer precision for each layer (S215). While adopting a large integer precision value can suppress deterioration in the compression ratio due to quantization error, it also increases the processing time. Therefore, the integer precision determination process comprehensively considers both the compression ratio and the processing time when determining the integer precision. In the integer precision determination process, the integer precision is determined using, for example, a greedy algorithm based on the inference accuracy improvement efficiency of each layer, or a method using a fixed value of N_int = 1 for all layers.

[0187] An example of the integer precision determination process for each layer will be described below. The processor 53B creates random sample data by randomly extracting a specific number (e.g., 1024) of partial data units from the data to be compressed. The processor 53B then compresses the random sample data using a compressor 70B (here, compressor 70Ba) with N_int=1 for all layers and a compressor 70B (here, compressor 70Bb) with N_int=2 for a specific layer, and calculates the compression ratio. Multiple types of compressors 70Bb may be provided. The compressors 70Ba and 70Bb may also be configured to process in parallel on the parallel processing device 61. The processor 53B derives the inference accuracy improvement efficiency based on the compression ratios obtained when multiple compressors 70B are used. Here, the inference accuracy improvement efficiency may be calculated as ((compression rate when compressor 70Ba is used) - (compression rate when compressor 70Bb is used)) / ((processing time when compressor 70Bb is used) - (processing time when compressor 70Ba is used)). Note that each term used to calculate the inference accuracy improvement efficiency may be an estimated value. Based on the derived inference accuracy improvement efficiency, for example, a greedy method may be used to comprehensively consider both the compression rate and the processing time, and processor 53B may determine the accuracy of each layer so as to meet the required compression rate or processing time criteria.

[0188] On the other hand, if it is determined that the integer precision for each layer is not undetermined (S214: N), the integer precision has already been determined, so the determined integer precision will be used from now on, and the processor 53B proceeds to step S216.

[0189] In step S216, the processor 53B performs a quantization process in which the weighting matrix is ​​quantized to the determined integer precision, and calculates a quantized weighting matrix 1802-1 and coefficient data 1804. For example, the processor 53B generates the quantized weighting matrix 1802-1 by applying a sign function to each element of the weighting matrix, and calculates the coefficient data 1804 so as to minimize the square error between the product of the quantized weighting matrix 1802-1 and the coefficient data 1804 and the weighting matrix. For example, if each element of the quantized weighting matrix 1802-1 is 2 bits, the coefficient data 1804 is 1, 2, 1, 2, . . .

[0190] In step S217, the processor 53B performs a process of adding model data including the result of the quantization process to the model information 120d. The content added to the model information 120d may include, for example, a quantized weight matrix 1802-1, integer precision of each layer, coefficient data 1804, etc.

[0191] Next, the matrix multiplication process will be described.

[0192] 19 is a flowchart of the matrix multiplication process according to the first embodiment. The matrix multiplication process is started by the control processor 1611 in the prediction model.

[0193] First, the control processor 1611 determines whether it is necessary to divide the matrix multiplication result 1803 (S221). The control processor 1611 determines whether division is necessary by referring to the storage capacity of the high-speed memory 1613, the number of matrix calculation core groups 1621, etc. For example, if the matrix multiplication result 1803 is too large to be stored in one high-speed memory 1613, the control processor 1611 determines that division is necessary.

[0194] As a result, if it is determined that division is necessary (S221: Y), the control processor 1611 divides the matrix to be input to the matrix multiplication operation (S222). The size of the matrix after division is determined based on whether the size allows the prediction processing model process to operate efficiently overall from the standpoints of, for example, specifications for the matrix operation core group 1621, such as the row and column sizes after division having to be predetermined values ​​(e.g., multiples of 8), the size that can be stored in the high-speed memory 1613, and whether sequential read / write is possible when accessing the shared memory 1602. For example, the input matrix may be divided so that the matrix multiplication operation result 1803 is converted into a block matrix to share a matrix multiplication operation of 128 × 128 size.

[0195] On the other hand, if it is determined that division is not necessary (S221: N), the control processor 1611 proceeds to step S223, with the processing to be executed by one combination of the arithmetic processor 1612 and the high-speed memory 1613 thereafter.

[0196] In step S223, the control processor 1611 executes a process of loading the input matrix 1801 and the quantized weight matrix 1802-2 (if the input matrix 1801 was divided in step S222, the divided portion of the input matrix 1801 and the necessary portion of the quantized weight matrix 1802-2) from the shared memory 1602 to the high-speed memory 1613. For example, when a matrix multiplication operation of 128 × 128 size is to be shared, the control processor 1611 loads 128 rows of the input matrix 1801 and 128 columns of the quantized weight matrix 1802-2 into the high-speed memory 1613a of the operation processor 1612 that is to execute the operation. At this time, for efficient data transfer, the input matrix 1801 and the quantized weight matrix 1802-2 may be transferred from the shared memory 1602 to the intra-chip shared memory 1614, and the necessary portions may be loaded from the intra-chip shared memory 1614 to each high-speed memory 1613.

[0197] In step S224, the control processor 1611 loads the input matrix 1801 (or a part thereof) and the quantized weight matrix 1802-2 (or a part thereof) from the high-speed memory 1613 into the matrix calculation core group 1621, and causes them to perform a matrix multiplication calculation.

[0198] Next, the matrix calculation core group 1621 performs processing to store the matrix multiplication calculation result 1803 in the high-speed memory 1613 .

[0199] Next, the activation function processing will be described.

[0200] 20 is a flowchart of the activation function processing according to the first embodiment. The activation function processing is started by the control processor 1611 after the matrix multiplication operation is performed within the prediction model.

[0201] The control processor 1611 performs a process of loading the coefficient data 1804 into the high-speed memory 1613 (S231).

[0202] Next, the control processor 1611 causes the integer processing core group 1622 to load the matrix multiplication operation result 1803, the coefficient data 1804, and the bias data 1807 from the high-speed memory 1613, and execute the processes of steps S202, S203, S204, and S205 in Fig. 17 (S232). At this time, for example, a sign function may be adopted as the nonlinear function used when converting a multi-bit element to a single bit in the 1-bit conversion process of step S205, and the sign function may be, for example, a function that returns 0 if the operation result is larger than half of KC_in, and 1 if not.

[0203] Next, the integer processing core group 1622 performs processing to store the processing result 1808 in the high-speed memory 1613 (S233).

[0204] Next, the control processor 1611 executes a sorting process to sort the operation results 1808 (S234). The sorting process is performed to efficiently process the next operation result store process (S235) in which the operation results are stored in the shared memory 1602. The sorting process is performed to sort the data so that the data layout at the time of output corresponds to the data layout at the time of input, since the data layout in the high-speed memory 1613 at the time of input by the matrix operation core group 1621 generally differs from the data layout in the high-speed memory 1613 at the time of output by the integer operation core group 1622. For example, in the parallel processing device 61, threads managed by consecutive numbers can generally be efficiently stored in the shared memory 1602 when accessed by consecutive addresses, so the data held by each thread of the operation results 1808 is sorted to consecutive addresses. The sorting process may be performed on the high-speed memory 1613 for efficient processing.

[0205] Next, the control processor 1611 executes a process of storing the calculation result 1809 obtained by the sorting process in the shared memory 1602. The calculation result 1809 may be managed in the cache information table 1500.

[0206] Next, the data flow of the matrix multiplication process and activation function process in the parallel processing device 61 will be described.

[0207] Fig. 21 is a diagram illustrating the data flow of the matrix multiplication process and the activation function process in the parallel processing device according to the first embodiment. Fig. 21 shows the data flow in a set of a certain calculation processor 1612a and a high-speed memory 1613a when the control processor 1611 divides the process into sets of each calculation processor 1612 and a high-speed memory 1613 and executes them in parallel in the matrix multiplication process and the activation function process.

[0208] The divided input matrix 1801a is a matrix obtained by dividing the input matrix 1801, and the partial quantized weight matrix 1802a is a matrix obtained by dividing the quantized weight matrix 1802. The matrix calculation core group 1621 reads out the divided input matrix 1801a and the divided quantized weight matrix 1802a stored in the high-speed memory 1613a, and executes a matrix multiplication process (S224). Thereafter, the matrix calculation core group 1621 executes a process (S225) of storing the divided matrix multiplication result 1803a in the high-speed memory 1613a.

[0209] Thereafter, the integer processing core group 1622a reads the partitioned matrix multiplication operation result 1803a and executes activation function processing (S232). Thereafter, the integer processing core group 1622a executes processing to store the partitioned operation result 1808a in the high-speed memory 1613a (S233). Thereafter, if necessary, the control processor 1611 executes processing to rearrange the data in the high-speed memory 1613a (S234) to obtain a partitioned and rearranged operation result 1809a, which is the operation result after rearrangement. Next, the control processor 1611 executes processing to store the partitioned and rearranged operation result 1809a in the high-speed memory 1613a in the shared memory 1602 (S235).

[0210] 21, each element of the divided input matrix 1801a, the divided quantized weight matrix 1802a, the divided calculation result 1808a, and the divided rearranged calculation result 1809a is a 1-bit integer. On the other hand, each element of the divided matrix multiplication result 1803a is an integer of a required number of bits, such as 32 bits. In this embodiment, by making the divided rearranged calculation result 1809a 1-bit, the amount of access to the low-speed shared memory 1602 can be reduced, and the processing time required for step S235 can be shortened. Furthermore, by making everything other than the divided matrix multiplication result 1803a 1-bit integers, processing can be performed on the low-capacity high-speed memory 1613a, and the matrix multiplication calculation and activation function calculation can be performed efficiently.

[0211] The present invention is not limited to the above-described embodiment, and can be modified appropriately without departing from the spirit of the present invention.

[0212] For example, in the above embodiment, an example was shown in which the parallel processing device was different from the processor 53 (53A, 53B), but if the processor 53 has multiple cores, the processor 53 may be used as a parallel processing device.

[0213] In addition, in the above-described embodiments, some or all of the processing performed by the processor may be performed by a hardware circuit. Also, the programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]

[0214] 1...computer system, 40...input device, 53A, 53B...processor, 61A, 61B ...Parallel processing devices 70A, 70B, 72A, 72B...Compressors 71A, 71B, 73 A, 73B...expander, 80...user terminal, 101, 101A, 101B...computer

Claims

1. 1. A data compression system for compressing data, comprising: a parallel processing device; the parallel processing device Divide the data to be compressed into multiple partial data, Compressing each of the partial data in parallel; In the compression process, a neural network is used to calculate an occurrence probability for each predetermined data unit of the partial data; outputting an encoded bit string that is an entropy-encoded bit string for each of the data units based on the data unit and the occurrence probability; The process for realizing the neural network includes: a first transformation process for executing a matrix multiplication operation; a second conversion process subsequent to the first conversion process, which inputs a result of the first conversion process and converts each element of the matrix resulting from the result of the first conversion process into a 1-bit integer. Data compression system.

2. 2. The data compression system of claim 1, the parallel processing device includes one or more processors including a plurality of matrix operation cores and a plurality of integer operation cores; the parallel processing device performing the first transformation process using the matrix calculation core; The integer processing core is used to perform the second conversion process. Data compression system.

3. 3. The data compression system of claim 2, the arithmetic processor further includes a first memory connected to the matrix arithmetic core and the integer arithmetic core; the parallel processing device storing a processing result of the first conversion processing in the first memory; The processing result of the first conversion processing stored in the first memory is input to the second conversion processing, and the processing result of the second conversion processing is stored in the first memory. Data compression system.

4. 4. The data compression system of claim 3, The parallel processing device rearranges the operation results stored in the first memory into a layout in the first memory that matches the layout of input data input to the first conversion process. Data compression system.

5. 5. The data compression system of claim 4, The parallel processing device stores the operation results after rearranging the layout on the first memory in a second memory that has a larger capacity and a slower speed than the first memory. Data compression system.

6. 4. The data compression system of claim 3, an input matrix, each element of which is 1 bit, and an operation matrix, each element of which is a multi-bit matrix, for performing a matrix multiplication operation with the input matrix, are stored in the first memory; the matrix operation core is capable of performing operations between matrices, each element of which is 1 bit; the matrix operation core executes an operation between the input matrix and a virtual matrix in which each digit of each multi-bit element of the operation matrix is ​​a one-bit element, and stores a matrix resulting from the operation in a first memory; The integer calculation core multiplies each element of the matrix of the calculation result by a coefficient corresponding to each element of the calculation matrix, generates a sum matrix for the matrix obtained by multiplication, each element of which is a sum of values ​​of multiple elements based on the same element of the calculation matrix, and outputs a matrix in which each element of the sum matrix is ​​converted to 1 bit as a processing result of the second conversion process. Data compression system.

7. A data compression program that causes a computer to execute a process of compressing data, the computer has a parallel processing device; The parallel processing device The data to be compressed is divided into multiple partial data. performing compression processing on each of the partial data in parallel; In the compression process, a neural network is used to calculate an occurrence probability for each predetermined data unit of the partial data; outputting an encoded bit string that is an entropy-encoded bit string for each of the data units based on the data unit and the occurrence probability; The process for realizing the neural network includes: a first transformation process for executing a matrix multiplication operation; a second conversion process subsequent to the first conversion process, which inputs a result of the first conversion process and converts each element of the matrix resulting from the result of the first conversion process into a 1-bit integer. Data compression programs.

8. A data compression method by a data compression system that compresses data, comprising: the data compression system includes a parallel processing device; The parallel processing device The data to be compressed is divided into multiple partial data. performing compression processing on each of the partial data in parallel; In the compression process, a neural network is used to calculate an occurrence probability for each predetermined data unit of the partial data; outputting an encoded bit string that is an entropy-encoded bit string for each of the data units based on the data unit and the occurrence probability; The process for realizing the neural network includes: a first transformation process for executing a matrix multiplication operation; a second conversion process subsequent to the first conversion process, which inputs a result of the first conversion process and converts each element of the matrix resulting from the result of the first conversion process into a 1-bit integer. Data compression method.

Images