Data compression method and computing device

Abstract

The data compression method and the computing device disclosed by the application relate to the field of data compression, and solve the problem of how to reduce the memory overhead and time overhead of data compression using arithmetic coding. The method is applied to a computing device. The method comprises the following steps: a computing device receives to-be-compressed data, and identifies the data type of the to-be-compressed data; one or more data compression models are selected according to the data type, and the to-be-compressed data is compressed according to the selected one or more data compression models.

Description

[0001] This application claims priority to Chinese patent application filed on December 28, 2019, with application number 201911385235.6 and entitled "A Data Compression Method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data compression, and more particularly to data compression methods and computing devices. Background Technology

[0003] To reduce data storage costs, processors typically employ data compression algorithms to compress raw data. This involves reconstructing the raw data using specific methods to reduce redundancy and decrease the storage capacity occupied by the original data. Common data compression algorithms include dictionary coding and entropy coding. Among these, arithmetic coding offers the highest compression ratio. Arithmetic coding encodes the probability of each bit in the raw data as a decimal between 0 and 1. According to Shannon's information theory, the closer the predicted value of information is to the true probability distribution, the higher the compression ratio of arithmetic coding. Currently, processors use numerous data compression models to predict the probability of each bit in the raw data and then compress the data based on these probabilities. This results in significant memory overhead and long encoding time for processors using arithmetic coding for data compression. Therefore, reducing the memory and time overhead of data compression using arithmetic coding is a pressing issue that needs to be addressed. Summary of the Invention

[0004] The data compression method and computing device provided in this application solve the problem of how to reduce the memory and time overhead of data compression using arithmetic coding.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, this application provides a data compression method applied to a computing device, the method comprising: receiving data to be compressed; identifying the data type of the data to be compressed; selecting one or more data compression models according to the data type; and compressing the data to be compressed according to the selected one or more data compression models.

[0007] For example, data types include, but are not limited to, text, image, and video types. Each data type has its corresponding data compression model. For instance, the data compression model for text types is a text model, a word model, or a nested model; the data compression model for image types is a record model, an image model, or a JPEG image format model; and the data compression model for video types is a record model, a sound source model, or a sparse model. It is understood that each data type may have more than one corresponding data compression model.

[0008] This application avoids compressing the data to be compressed using all data compression models. Instead, it identifies the data type of the data to be compressed and selects an appropriate data compression model based on its data type, thereby compressing the data according to the selected model. Since the number of data compression models matching the data type is less than the number of data compression models used in the prior art for data compression using arithmetic coding, the memory and time overhead of data compression using arithmetic coding is reduced.

[0009] Optionally, the method provided in the first aspect can be executed by a first processor in a computing device, where the first processor mainly refers to a central processing unit (CPU). Alternatively, the method can be shared by a first processor and a second processor. The second processor, also located within the computing device, possesses computing capabilities and is used to perform data compression functions; it is referred to as a coprocessor. Utilizing a coprocessor to share the data compression operation can further reduce the computational burden on the first processor. Specifically, the first processor receives the data to be compressed, identifies the data type of the data to be compressed, and selects one or more data compression models based on the data type. The second processor compresses the data to be compressed according to the selected data compression model.

[0010] In one possible implementation, before compressing the data to be compressed using a data compression model, the method further includes: training the first sample data to generate several data compression models; storing the several data compression models or a subset of several data compression models in a computing device, so as to facilitate compressing the data to be compressed using the data compression models.

[0011] The subset of the data compression models refers to a selection of data compression models with basic compression functions, which can be superimposed to achieve the functions of other data compression models.

[0012] Optionally, the computing device may include an Artificial Intelligence (AI) chip, which trains the first sample data to generate several data compression models. Alternatively, the computing device may not include an AI chip; instead, a first processor or a second processor can train the first sample data to generate several data compression models. If an AI chip is included, and it trains the first sample data to generate several data compression models, the computational burden on the first or second processor within the computing device can be further reduced.

[0013] Specifically, at least three implementation methods are included here. (1) The plurality of data compression models or a subset of the plurality of data compression models are stored in the memory of the computing device, and the first processor and / or the second processor can obtain the data compression model that matches the data type of the data to be compressed from the memory; (2) The second processor has its own cache, and the plurality of data compression models or a subset of the plurality of data compression models are stored in the cache of the second processor, and the second processor can obtain the data compression model that matches the data type of the data to be compressed from its own cache; (3) Since the second processor is a processing chip, the plurality of data compression models or a subset of the plurality of data compression models can be burned into the processing chip. In the implementation of (2) or (3), the identifier of each data compression model can be stored in the memory of the computing device and the cache of the second processor (or burned into its own chip). Thus, after the first processor selects the data compression model that matches the data type, it can send the identifier of the data compression model to the second processor, and the second processor obtains the data compression model according to the identifier of the data compression model.

[0014] Optionally, the method also includes upgrading the data compression model stored in the computing device. Upgrading here refers to adding a new data compression model, decommissioning an unused or infrequently used data compression model, or updating one or more of the data compression models.

[0015] Optionally, the method further includes: training the second sample data to generate M data type recognition models. The training operation on the second sample data can be performed by the artificial intelligence chip described above, or by the first processor or the second processor.

[0016] In another possible implementation, the computing device stores M data type identification models, each used to identify the probability of P data types occurring. Identifying the data type of the data to be compressed includes: determining M probability sets corresponding to the data to be compressed based on the M data type identification models, each probability set containing P initial probability values, each of the P initial probability values ​​indicating the possibility that the data to be compressed is a data type, where M is an integer greater than or equal to 1 and P is an integer greater than or equal to 1; and determining the data type of the data to be compressed based on the M probability sets.

[0017] Therefore, the computing device can use the probability of multiple possible data types of the data to be compressed output by the data type identification model, and then determine the data type of the data to be compressed based on the probability of multiple possible data types. This makes it easier to determine the appropriate data compression model based on the data type of the data to be compressed, and to compress the data using the appropriate data compression model. Since the number of data compression models that match the data type is less than the number of data compression models used in the prior art for data compression using arithmetic coding, the memory and time overhead of data compression using arithmetic coding is reduced.

[0018] Furthermore, different data type identification models may have varying degrees of accuracy in recognizing different data types. This application utilizes multiple data type identification models to output the different probabilities of multiple possible data types of the data to be compressed, and determines the data type of the data to be compressed based on the different probabilities of these multiple possible data types, thereby improving the accuracy of identifying the data type of the data to be compressed.

[0019] Optionally, the data type of the data to be compressed is determined based on the M probability sets, including: obtaining P intermediate values ​​based on the weight ratios of the M probability sets and the M data type identification models; and determining the data type of the intermediate value that is greater than the accuracy threshold among the P intermediate values ​​as the data type of the data to be compressed.

[0020] This application determines the median value of the data type of the data to be compressed based on the weight ratio of the data type identification model and the different probabilities of multiple possible data types, and then uses an accuracy threshold and the median value to determine the data type of the data to be compressed. This improves the accuracy of identifying the data type of the data to be compressed.

[0021] Secondly, this application provides a computing device, which includes one or more processors, wherein the one or more processors are configured to receive data to be compressed, identify the data type of the data to be compressed, select one or more data compression models according to the data type, and compress the data to be compressed according to the selected one or more data compression models.

[0022] In one possible implementation, the one or more processors include a first processor and a second processor, wherein the first processor is used to receive data to be compressed, identify the data type of the data to be compressed, and select one or more data compression models according to the data type; the second processor is used to compress the data to be compressed according to the selected one or more data compression models. Thus, since the data compression process is performed by other processors, the compression performance of arithmetic coding is effectively improved.

[0023] Optionally, the first processor is a central processing unit (CPU), and the second processor is a coprocessor.

[0024] In another possible implementation, the computing device also includes an AI chip, which is used to train the first sample data to generate several data compression models; the first processor is also used to store the several data compression models or a subset of several data compression models in the computing device.

[0025] Optionally, the first or second processor may also be used to upgrade the data compression model stored in the computing device.

[0026] In another possible implementation, the AI ​​chip is also used to train the second sample data to generate M data type recognition models.

[0027] In another possible implementation, each data compression model has an identifier, and the first processor is also used to send the identifier of the selected data compression model to the second processor; the second processor is also used to obtain the corresponding data compression model based on the identifier.

[0028] In another possible implementation, the computing device stores M data type identification models, each model being used to identify the probability of P data types occurring. The first processor is specifically used to determine M probability sets corresponding to the data to be compressed based on the M data type identification models. Each probability set contains P initial probability values, and each of the P initial probability values ​​is used to indicate the possibility that the data to be compressed is a data type. M is an integer greater than or equal to 1, and P is an integer greater than or equal to 1. The data type of the data to be compressed is determined based on the M probability sets.

[0029] Optionally, the first processor is specifically used to obtain P intermediate values ​​based on the weight ratios of the M probability sets and the M data types of the identification model; and to determine the data type corresponding to the intermediate value that is greater than the accuracy threshold among the P intermediate values ​​as the data type of the data to be compressed.

[0030] Thirdly, this application also provides a data compression device, the beneficial effects of which are described in the first aspect and will not be repeated here. The data compression device has the function of implementing the behavior in the examples of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the data compression device includes: a first processing module and a second processing module. The first processing module is used to receive data to be compressed, identify the data type of the data to be compressed, and select one or more data compression models according to the data type; the second processing module is used to compress the data to be compressed according to the selected one or more data compression models. The first processing module can implement the function of a first processor. The second processing module can implement the function of a second processor. These units can perform the corresponding functions in the examples of the method in the first aspect, as detailed in the examples, and will not be repeated here.

[0031] Optionally, the data compression device further includes a storage module, and the first processing module is further configured to store the correspondence between the data type of the data to be compressed and the one or more data compression models in the storage module. The storage module may be a cache in a computing device, a cache of the second processor itself, or other components with storage functionality.

[0032] Optionally, the data compression device further includes an AI chip, which is used to train the first sample data to generate several data compression models; the first processing module is also used to store the several data compression models or a subset of the several data compression models in the computing device.

[0033] Optionally, the first processing module or the second processing module is further configured to upgrade the data compression model stored in the computing device.

[0034] Optionally, each data compression model has an identifier, and the first processing module is further configured to send the identifier of the selected data compression model to the second processing module; the second processing module is further configured to obtain the corresponding data compression model according to the identifier.

[0035] Optionally, the computing device stores M data type identification models, each used to identify the probability of P data types appearing. The first processing module is specifically used to determine M probability sets corresponding to the data to be compressed based on the M data type identification models. Each probability set contains P initial probability values, each of the P initial probability values ​​indicating the possibility that the data to be compressed is a data type, where M is an integer greater than or equal to 1 and P is an integer greater than or equal to 1; and determines the data type of the data to be compressed based on the M probability sets.

[0036] Optionally, the first processing module is specifically used to obtain P intermediate values ​​based on the weight ratios of the M probability sets and the M data type identification models; and to determine the data type corresponding to the intermediate value that is greater than the accuracy threshold among the P intermediate values ​​as the data type of the data to be compressed.

[0037] Fourthly, a data compression method is provided, which is applied to a computing device, the computing device including a first processor and a second processor. The first processor is used to receive data to be compressed, and the second processor is used to compress the data to be compressed.

[0038] Compared with the data compression method provided in the first aspect, the method provided in the fourth aspect does not filter the data compression model according to the data type of the data to be compressed. Instead, it sets up a second processor specifically for processing compression operations, thereby freeing up the first processor to process other data services and reducing the computational burden on the first processor.

[0039] The first processor and the second processor are consistent with those described in the first aspect, and will not be repeated here.

[0040] Fifthly, a computing device is provided for performing the data compression method provided in the fourth aspect.

[0041] In a sixth aspect, a computer program product is provided, the computer program product comprising: computer program code, which, when the computer program code is executed, causes the methods executed by the computing device in the above aspects to be performed.

[0042] Seventhly, this application provides a chip system including a processor for implementing the functions of the computing device in the methods of the above aspects. In one possible design, the chip system further includes a memory for storing program instructions and / or data. This chip system may be composed of chips or may include chips and other discrete devices.

[0043] Eighthly, this application provides a computer-readable storage medium storing a computer program that, when run, implements the methods executed by the computing device in the above aspects.

[0044] In this application, the names of the computing device and the data compression apparatus do not limit the devices themselves; in actual implementation, these devices may appear under other names. As long as the functions of each device are similar to those in this application, they fall within the scope of the claims of this application and their equivalents. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the composition of a computing device provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating a data compression method provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a data type identification model provided in an embodiment of this application;

[0048] Figure 4 A flowchart illustrating a data compression method provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the composition of a data compression device provided in an embodiment of this application. Detailed Implementation

[0050] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0051] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0052] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0053] Figure 1 This is a schematic diagram of the composition of a computing device provided in an embodiment of this application, as shown below. Figure 1 As shown, the computing device 100 may include a processor 101, a memory 102, a processor 103, and a communication bus 104.

[0054] The following is combined with Figure 1A detailed description of each component of the computing device 100 is provided below:

[0055] Computing device 100 may include multiple processors, such as Figure 1 The processors 101 and 103 shown are illustrated. Processor 101 is the control center of the computing device. Typically, processor 101 is a central processing unit (CPU), including one or more CPU cores, for example... Figure 1 CPU0 and CPU1 are shown in the diagram. Furthermore, processor 101 can also be an application-specific integrated circuit (ASIC), or configured as one or more integrated circuits, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Processor 101 can perform various functions of computing device 100 by running or executing software programs stored in memory 102 and by accessing data stored in memory 102.

[0056] Processor 103 can be a processor with the same physical form as processor 101, or it can be a processor with a different physical form. However, processor 103 is a processing chip with computing capabilities, used to perform data compression functions to reduce the computational burden on processor 101. In this embodiment, processor 103 assists processor 101 in compressing the data to be compressed. For example, processor 103 compresses the data to be compressed according to one or more data compression models indicated by processor 101. In practical applications, processor 103 can be an accelerator card, coprocessor, graphics processing unit (GPU), or neural network processing unit (NPU), etc. In this embodiment, one or more processors 101 and one or more processors 103 can be configured. However, it should be noted that processor 103 is an optional component in this embodiment. Even with only one processor 101, processor 101 can independently receive the data to be compressed, identify the data type of the data to be compressed, select one or more data compression models according to the data type, and then compress the data to be compressed using the one or more data compression models. When the computing device 100 includes both processor 101 and processor 103, the above operations can be performed jointly by processor 101 and processor 103. For example, processor 101 is mainly used to receive data to be compressed, identify the data type of the data to be compressed, select one or more data compression models according to the data type, then send the data to be compressed to processor 103, and instruct processor 103 to compress the data to be compressed according to the selected one or more data compression models.

[0057] Both processors 101 and 103 can access memory 102 via communication bus 104. Memory 102 stores a set of data compression models, which includes multiple data compression models. Additionally, memory 102 stores a correspondence between the data type of the data to be compressed and one or more data compression models. Understandably, this correspondence can be a correspondence between the data type of the data to be compressed and the entirety of one or more data compression models. Alternatively, the correspondence can be a correspondence between the data type of the data to be compressed and each of the one or more data compression models. Processor 101 can retrieve the correspondence from memory 102 and select one or more data compression models based on the data type and the correspondence.

[0058] Typically, different data compression models can be designed for different data types. Therefore, data compression models can be classified according to different data types. The memory 102 can store the correspondence between other data types and one or more data compression models. This application does not limit the number of correspondences between data types and data compression models.

[0059] In addition, the memory 102 is also used to store the software program that executes the solution of this application, and the execution is controlled by the processor 101.

[0060] In terms of physical form, memory 102 can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 102 can exist independently and be connected to processor 101 via communication bus 104. Memory 102 can also be integrated with processor 101, without limitation.

[0061] Regarding the source of the data compression model, this embodiment provides at least two implementation methods. One is to select a commonly used data compression model from all data compression models in conventional technologies, or to use other data compression models generated by superimposing commonly used data compression models as the data compression model. The so-called other data compression models generated by superimposing commonly used data compression models mainly refer to generating a high-order data compression model by superimposing low-order data compression models.

[0062] These commonly used data compression models include, but are not limited to: nested models, context models, indirect models, text models, sparse models, Extensible Markup Language (XML) models, match models, distance models, executable model (EXE) models, word models, record models, image models (PIC) models, prediction by partial matching (PPMD) models, dynamic Markov compression (DMCM) models, byte models, linear prediction models, adaptive prediction models, WAV models, and common models. Among these, nested models predict subsequent bytes based on information from nested symbols (such as []) appearing in the byte to be predicted. Context models predict subsequent bytes based on the context of consecutive bytes preceding the byte to be predicted. Indirect models predict subsequent bytes using the historical information of the first 1-2 bytes of the byte to be predicted and the surrounding context. Text models predict subsequent bytes using information such as words, sentences, and paragraphs. They are typically used to predict text-based data. Sparse models predict subsequent bytes by finding non-contiguous bytes preceding the byte to be predicted as context. For example, a model predicting the next byte based on the byte preceding it (e.g., 1 byte and 3 bytes). Extensible Markup Language (XML) models predict subsequent bytes based on features such as tags contained in the byte to be predicted. Matching models predict subsequent bytes by checking for matching information in the context preceding the byte to be predicted. Distance models predict subsequent bytes using the current byte to be predicted and certain special bytes. For example, the distance between the special byte and a space character. Executable program models predict subsequent bytes using specific instruction sets and opcodes. Word models predict the context of subsequent bytes based on the occurrence of words. Record models predict the context of subsequent bytes by looking at rows and columns in a file. In a table, a row is called a record, and this is often used in databases and tables. Image models predict the context of subsequent bytes using features of an image. For example, a model predicting the context of subsequent bytes using the grayscale or pixel count of an image.Partial matching prediction models predict the next bit by searching for a match within the byte to be predicted, based on multiple consecutive bytes preceding it. If no match is found, a portion of the bytes is removed, and the search continues until a match is found or the byte is recorded as a new byte. Dynamic Markov compression models predict the next bit using a variable-length bit-level context history table. Byte models predict subsequent bits based on bit history information. Linear prediction models predict the context of subsequent bytes using linear regression analysis. Adaptive models adjust calculated probabilities based on probabilities from other models and known context. Sound source models predict the context of subsequent bytes using feature information from audio files. General models predict probabilities for new or unrecognized data types. This general model can be generated by stacking multiple other models. Because the data compression model set is a filtered set of data compression models, its number is much smaller than the original set, resulting in lower storage space requirements. Therefore, the data compression model set can be stored on processor 103, which will then perform the data compression operation. This reduces the computational burden on processor 101, as well as memory and time overhead.

[0063] In some embodiments, the correspondence between data types and data compression models can be presented in tabular form, i.e., Table 1 presents the correspondence between data types and data compression models. After the processor 101 identifies the data type of the data to be compressed, it can query Table 1 to determine one or more data compression models corresponding to the data type of the data to be compressed.

[0064] Table 1

[0065]

[0066] As shown in Table 1, text types correspond to text models, word models, and nested models. If the processor 101 determines that the data to be compressed is of type text, the processor 101 can look up Table 1 to find the text model, word model, and nested model corresponding to the text type, and then use the text model, word model, and nested model to compress the data to be compressed.

[0067] The executable program type corresponds to the executable program model and the sparse model. If the processor 101 determines that the data type of the data to be compressed is an executable program type, the processor 101 can look up Table 1 to find out the executable program model and the sparse model corresponding to the executable program type, and then use the executable program model and the sparse model to compress the data to be compressed.

[0068] If the processor 101 does not recognize the data type of the data to be compressed, and determines that the data type of the data to be compressed is other (general) type, the processor 101 can look up Table 1 to find out the text model, sparse model and record model corresponding to other (general) types, and use the text model, sparse model and record model to compress the data to be compressed.

[0069] It should be noted that Table 1 is only a table illustrating the storage format of the correspondence between data types and data compression models in the storage device. It is not a limitation on the storage format of the correspondence between data types and data compression models in the storage device. Of course, the correspondence between data types and data compression models can also be stored in other forms in the storage device. This application embodiment does not limit this.

[0070] Specifically, the data compression model set can be stored on the processor 103 via software. For example, the data compression models contained in the data compression model set can be stored in a memory built into or coupled to the processor 103. Optionally, the data compression model set can also be stored on the processor 103 via hardware. For example, the data compression model set can be programmed onto the processor 103 in the form of a circuit structure designed for the processor 103.

[0071] With the development of science and technology, if new data types emerge, the aforementioned set of data compression models can be upgraded. For example, the data compression model corresponding to the new data type can be stored on processor 103. New data types include genetic data and big data types. Existing data compression models that are used less frequently can be phased out, and existing data compression models (one or more) can be modified.

[0072] In some embodiments, new data compression models can be generated by combining existing data compression models, and the set of data compression models stored on the processor 103 can be updated through software upgrades. For example, a high-order data compression model can be implemented using a low-order data compression model. Thus, no changes to the hardware circuitry are required, significantly reducing the upgrade cost of the processor 103.

[0073] In other embodiments, different data compression models can be stored by designing the circuit structure of processor 103. When upgrading the set of data compression models, the circuit structure of processor 103 can be changed to upgrade the set of data compression models. Since processor 103 stores commonly used data compression models, even if the set of data compression models is upgraded, the hardware circuit needs to be modified less, and the upgrade cost is low.

[0074] Another implementation involves placing an Artificial Intelligent (AI) chip 105 in the computing device 100. The AI ​​chip 105 periodically trains the received sample data to generate several data compression models. These data compression models are then stored in the memory 102 via the communication bus 104, or transferred to the processor 103 for storage. The AI ​​chip 105 can be a single chip or other physical component; for example, it could be a training chip for building neural network models, or an inference chip for using neural network models for reasoning. Neural network models can also be called artificial neural networks, neural networks (NNs), or connection models. An "artificial neural network" is an algorithmic mathematical model that uses a structure similar to the synaptic connections of the brain for distributed parallel information processing. This artificial neural network, depending on the complexity of the system, adjusts the relationships between a large number of interconnected nodes to achieve the purpose of processing information. Compared to the implementations listed above, periodically training the sample data with the AI ​​chip 105 to generate several data compression models allows for better adaptation to changes in the data compression models.

[0075] In addition, each data compression model has an identifier, and the processor 103 also stores the correspondence between the data compression model and its identifier. Therefore, the processor 103 can obtain the corresponding data compression model according to the identifier of the data compression model sent to it by the processor 101, and compress the data to be compressed.

[0076] The communication bus 104 can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into an address bus, a data bus, and a control bus. For ease of representation, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0077] Figure 1 The device structure shown does not constitute a limitation on the computing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] Next, combined Figure 2 This application provides a detailed description of a data compression method according to an embodiment of the present application, which is applied to... Figure 1 In the computing device 100 shown, processors 101 and 103 will be used as examples for explanation. Figure 2 As shown, the method may include the following steps.

[0079] S201, the processor 101 receives the data to be compressed and identifies the data type of the data to be compressed.

[0080] Before identifying the data type of the data to be compressed, a data type identification model can be obtained by training a large amount of sample data using machine learning or artificial neural networks (ANNs). Alternatively, the data type identification model can also be trained on sample data by the artificial intelligence chip 105. This data type identification model is used to indicate the mapping relationship between the data to be compressed and its data type. Then, the processor 101 can use the data type identification model to identify the data type of the data to be compressed.

[0081] Data type identification models include, but are not limited to: Naive Bayes ( Models such as Bayes (NB), Extreme Gradient Boosting Tree (XGBoost), Multilayer Perceptron (MLP), and combinations thereof.

[0082] Sample data can originate from data blocks or file fragments. Sample data includes, but is not limited to: text data, image data, video data, genetic data, executable programs, virtual hard disk data, and database data. Correspondingly, the data types of sample data include, but are not limited to: text, image, video, genetic, executable program, virtual hard disk, and database types.

[0083] Optionally, sample data can be obtained from a database. Alternatively, the sample data can be open-source data or test data.

[0084] In some embodiments, an identifier can be set for the sample data based on a known data type. The sample data and its identifier are then input into an artificial neural network for training to obtain a data type recognition model. For example, if the sample data is text data, an identifier can be set for the text data based on its file extension. The text data and its identifier are then input into an artificial neural network for training to obtain a mapping relationship between the text data and the text type.

[0085] For example, such as Figure 3The diagram shown is a schematic of a data type recognition model provided in an embodiment of this application. When the data to be compressed is input into the data type recognition model, it can identify text type, image type, video type, gene type, executable program type, virtual hard disk type, database type, and other types. "Other types" can refer to general types.

[0086] In this embodiment, the data type recognition model can be pre-trained by other devices and then imported into the processor 101 so that the processor 101 can use the data type recognition model to identify the data type of the data to be compressed.

[0087] Optionally, after optimizing the internal weights of the data type recognition model by training it with the aforementioned sample data, the model can be validated using partial sample data to verify its reliability and adjust parameters such as the number of iterations and the learning rate. The parameters of the data type recognition model include internal weights, weight ratios, and accuracy thresholds. Since each data type recognition model may have different predictive capabilities for different data types, this application can assign different weights to the prediction probability of each data type recognition model based on different accuracy rates.

[0088] When identifying the data type of the data to be compressed, the processor 101 can first initialize the parameters of the data type identification model. Then, the data to be compressed is input into the data type identification model to obtain the probabilities of multiple data types. The processor 101 can determine the data type corresponding to the probability greater than the accuracy threshold among the multiple data type probabilities as the data type of the data to be compressed.

[0089] For example, processor 101 inputs the data to be compressed into a data type recognition model to obtain the probabilities of text type, image type, video type, gene type, executable program type, virtual hard disk type, database type, and other types. If the probability of text type is greater than the accuracy threshold, processor 101 determines that the data type of the data to be compressed is text type.

[0090] Optionally, if the processor 101 does not recognize the data type of the data to be compressed, it can determine that the data type of the data to be compressed is another type.

[0091] In some embodiments, to improve the accuracy of identifying the data type of the data to be compressed, the processor 101 may utilize multiple data type identification models to identify the data type of the data to be compressed. For example, such as... Figure 4 As shown, identifying the data type of the data to be compressed can be achieved through the following steps. Here, we will use an M-data type identification model as an example to illustrate how to identify the data type of the data to be compressed.

[0092] S201a, Processor 101 receives the data to be compressed and determines the M probability sets corresponding to the data to be compressed based on the M data type identification models.

[0093] Understandably, processor 101 inputs the data to be compressed into M data type recognition models, and each data type recognition model outputs a probability set. Each of the M probability sets contains P initial probability values. Each of the P initial probability values ​​indicates the probability that the data to be compressed is a data type. P represents the number of data types that the data type recognition model can recognize. Here, M is an integer greater than or equal to 1, and P is an integer greater than or equal to 1.

[0094] Optionally, each of the M data type recognition models can recognize the same data type. Understandably, for the same data type, each of the M data type recognition models outputs an initial probability value for that data type, and the M data type recognition models can collectively output M initial probability values ​​for that data type.

[0095] Optionally, the M data type recognition models output a total of M multiplied by P initial probability values. For the same data type recognition model, the P initial probability values ​​can be the same or different. For different data type recognition models determining the same data type, the initial probability values ​​can be the same or different; there is no restriction.

[0096] For example, assuming M=3 and P=6, processor 101 inputs the data to be compressed into three data type recognition models, and each data type recognition model outputs a probability set, resulting in three probability sets. Each of the three probability sets contains six initial probability values. The three data type recognition models output a total of 18 (3*6) initial probability values.

[0097] S201b, the processor 101 determines the data type of the data to be compressed based on M probability sets.

[0098] First, the processor 101 obtains P intermediate values ​​based on the weight ratios of the M probability sets and the M data type recognition models.

[0099] In some embodiments, for the j-th data type among P data types, the processor 101 can determine the intermediate value of the j-th data type based on the M initial probability values ​​corresponding to the j-th data type and the weight ratio of the M data type recognition models. The intermediate value of the j-th data type satisfies formula (1).

[0100]

[0101] Where j is an integer, j∈[1,P]. j Let represent the median value of the j-th data type. Let Wi represent the weight ratio of the model for the ith data type. P Mi This represents the initial probability value of the j-th data type output by the model for recognizing the i-th data type.

[0102] Understandably, after determining the median value of each of the P data types, the processor 101 obtains the median value of the P data types. Optionally, the median values ​​of the P data types can be the same or different, and are not limited thereto.

[0103] Furthermore, the processor 101 determines the data type of the data to be compressed based on an accuracy threshold and the median value of P data types. For example, the processor determines the data type corresponding to the median value of the P data types that is greater than the accuracy threshold as the data type of the data to be compressed.

[0104] S202, the processor 101 selects one or more data compression models according to the data type of the data to be compressed.

[0105] Typically, different data compression models can be designed for different data types. Therefore, data compression models can be pre-classified for different data types. The correspondence between data types and data compression models can be pre-defined. In some embodiments, the correspondence between data types and data compression models is pre-stored in memory 102. After processor 101 determines the data type, it can first retrieve the correspondence between data types and data compression models from memory 102, and then obtain one or more data compression models from the correspondence based on the data type of the data to be compressed. The correspondence between data types and data compression models includes the correspondence between the data type of the data to be compressed and one or more data compression models.

[0106] For example, assume the data to be compressed is of type text. Processor 101 determines the corresponding data compression model based on the text type, including TextModel, WordModel, and NestModel.

[0107] S203, the processor 101 sends the data to be compressed to the processor 103 and instructs the processor 103 to compress the data according to one or more selected data compression models.

[0108] Processor 101 can instruct processor 103 to compress the data to be compressed according to one or more selected data compression models by sending instruction information.

[0109] In some embodiments, the data compression model has an identifier. The indication information includes the data to be compressed and the identifier of the selected data compression model. The processor 103 stores the correspondence between the data compression models and their identifiers. Upon receiving the indication information, the processor 103 can retrieve the data compression model based on its identifier.

[0110] In other embodiments, the one or more data compression models selected by the processor 101 based on the data type of the data to be compressed can be considered as a set of data compression models. The set of data compression models has an identifier. The indication information includes the identifier of the data to be compressed and the identifier of the data compression model set. The processor 103 stores the correspondence between the data compression model set and its identifier. After receiving the indication information, the processor 103 can obtain the data compression model set according to its identifier; that is, the processor 103 obtains one or more data compression models corresponding to the data type of the data to be compressed.

[0111] S204, the processor 103 receives the data to be compressed and compresses the data according to one or more selected data compression models.

[0112] Understandably, the processor 103 has several preset data compression models. After determining one or more data compression models based on the identifier of the data compression model, the processor 103 retrieves one or more data compression models from the preset data compression models.

[0113] The processor 103 first determines the probability of each bit in the data to be compressed based on one or more data compression models, and then encodes the data to be compressed based on the probability of each bit in the data to be compressed and an arithmetic coding algorithm to obtain compressed data.

[0114] In some embodiments, it is assumed that the processor 103 compresses the data to be compressed according to N data compression models. N is an integer greater than or equal to 1. The processor 103 can determine N probability sets corresponding to the data to be compressed according to the N data compression models, and determine the probability of each bit in the data to be compressed according to the N probability sets.

[0115] Processor 103 inputs the data to be compressed into N data compression models, and each data compression model outputs a probability set. Each of the N probability sets contains Q initial probability values. Each of the Q initial probability values ​​indicates the initial probability value of a bit in the data to be compressed. Q represents the number of bits in the data to be compressed. Q is an integer greater than or equal to 1.

[0116] Understandably, for the same bit, each of the N data compression models outputs an initial probability value for that bit, and the N data compression models can output a total of N initial probability values ​​for that bit.

[0117] It should be noted that the data compression model has pre-set initial probability values. For example, the initial probability value is the initial probability value predicted by the data compression model for a bit to be 1. For the first bit of the data to be compressed, the data compression model can use the initial probability value to predict the initial probability value of the first bit. For the second bit of the data to be compressed, the data compression model can use the initial probability value of the first bit to predict the initial probability value of the second bit. Understandably, for the (y+1)th bit of the data to be compressed, the data compression model can use the initial probability value of the yth bit to predict the initial probability value of the (y+1)th bit. And so on, obtaining the initial probability value for each bit of the data to be compressed. y is an integer, y∈[1,Q].

[0118] Optionally, N data compression models output a total of N multiplied by Q initial probability values. For the same data compression model, the Q initial probability values ​​can be the same or different. Different data compression models can also determine the same or different initial probability values ​​for the same bit; this is not restricted.

[0119] For example, assuming N=3 and Q=6, processor 103 inputs the data to be compressed into three data compression models, and each data compression model outputs a probability set, resulting in three probability sets. Each of the three probability sets contains six initial probability values. The three data compression models output a total of 18 (3*6) initial probability values.

[0120] It should be noted that the initial probability value of the y-th bit can be determined by the processor 103 based on the initial probability value of the (y-1)-th bit. Alternatively, the initial probability value of the y-th bit can be predicted by the processor 103 based on the initial probability value of the (y-1)-th bit. Or, the processor 103 can predict the initial probability value of the (y+1)-th bit based on the initial probability value of the y-th bit.

[0121] Furthermore, the processor 103 obtains the probability of each bit in the data to be compressed based on the N probability sets and the weight ratios of the N data compression models.

[0122] In some embodiments, for the y-th bit among Q bits, the processor 103 can determine the probability of the y-th bit based on the N initial probability values ​​corresponding to the y-th bit and the weight ratio of the N data compression models. The probability of the y-th bit satisfies formula (2).

[0123]

[0124] Where y is an integer, y∈[1,Q]. P y Let Sx represent the probability of the y-th bit. Let Sx represent the weight ratio of the x-th data compression model. P Nx This represents the initial probability value of the y-th bit output by the x-th data compression model.

[0125] Optionally, the probabilities of the Q bits can be the same or different, without limitation.

[0126] It should be noted that when predicting the probability of bits contained in the data to be compressed, the parameters of the data compression model can be continuously adjusted based on existing prediction values ​​to obtain a more accurate bit probability. For example, a neural network can be used to train the weight ratio of the data compression model to obtain a more accurate weight ratio, and thus a more accurate bit probability.

[0127] For example, the data compression model has pre-set initial weight ratios. For instance, the initial weight ratio is the weight ratio of the probability that the data compression model predicts a bit as 1. The data compression model adjusts its weight ratios after predicting each bit. For the first bit of the data to be compressed, the data compression model can determine the probability of that first bit based on N initial probability values ​​corresponding to that first bit and N initial weight ratios of the data compression model. For the second bit of the data to be compressed, the data compression model can determine the probability of that second bit based on N initial probability values ​​corresponding to that second bit and N adjusted weight ratios of the data compression model. Understandably, the weight ratio used by the data compression model to predict the probability of the (y+1)th bit can be the weight ratio adjusted after the data compression model predicts the probability of the yth bit.

[0128] Compared to traditional technologies, where the processor compresses the data to be compressed using all data compression models, resulting in extremely high memory and time overhead. The data compression method provided in this application involves the processor 101 selecting one or more data compression models based on the data type of the data to be compressed. This reduces the number of data compression models used in the data compression process, thereby reducing the time spent predicting the probability of the data to be compressed and lowering the memory and time overhead of data compression using arithmetic coding. Furthermore, by offloading the data compression process to the processor 103, the compression performance of arithmetic coding is effectively improved.

[0129] The following example uses text data to illustrate the data compression method provided in this application.

[0130] Here, we use sample data to train and validate a Naive Bayes model, a Limiting Gradient Boosting Decision Tree model, and a Multilayer Perceptron model, respectively, obtaining the internal weights of the three models, the weight ratios between models, and the accuracy threshold. We assume the weight ratio for Naive Bayes is 0.4, the weight ratio for Limiting Gradient Boosting Decision Tree is 0.1, the weight ratio for Multilayer Perceptron is 0.5, and the accuracy threshold is 0.8. The data types of the sample data include text, images, videos, gene data, database data, and others. For ease of description, we use M1 to represent the Naive Bayes model, M2 to represent the Limiting Gradient Boosting Decision Tree model, and M3 to represent the Multilayer Perceptron model. We use W1 to represent the weight ratio for Naive Bayes, W2 to represent the weight ratio for Limiting Gradient Boosting Decision Tree, and W3 to represent the weight ratio for Multilayer Perceptron.

[0131] When identifying the data type of text data, the processor 101 inputs the text data into the Naive Bayes model, the Limit Gradient Boosting Decision Tree model, and the Multilayer Perceptron model, respectively. Each model outputs initial probability values ​​for six data types, as shown below:

[0132] P M1 ={P 文本 =0.8, P 图片 =0.05, P 视频 =0.02, P 基因 =0.03, P 数据库 =0.01, P 其他 =0.09};

[0133] P M2 ={P 文本 =0.6, P 图片 =0.1, P 视频 =0.02, P 基因 =0.08, P 数据库 =0.0, P 其他 =0.2};

[0134] P M3 ={P 文本 =0.9, P 图片 =0.02, P 视频 =0.01, P 基因 =0.0, P 数据库 =0.02, P 其他 =0.05}.

[0135] Among them, P M1 P represents the set of probabilities output by the Naive Bayes model. M2 P represents the set of probabilities output by the limiting gradient boosting decision tree model. M3 P represents the set of probabilities output by the multilayer perceptron model. 文本P represents the initial probability value for the text type. 图片 P represents the initial probability value for the image type. 视频 P represents the initial probability value for the video type. 基因 P represents the initial probability value of the gene type. 数据库 P represents the initial probability value for the database type. 其他 This represents the initial probability value for other types.

[0136] The processor 101 calculates the median value for each data type based on the weight ratio of the data type identification model. The median value satisfies formula (3).

[0137] P = W1 * P M1 +W2*P M2 +W3*P M3 (3)

[0138] For the intermediate value of the text type, P1 = 0.4 * 0.8 + 0.1 * 0.6 + 0.5 * 0.9 = 0.83.

[0139] For the intermediate value of the image type, P2 = 0.4 * 0.05 + 0.1 * 0.1 + 0.5 * 0.02 = 0.04.

[0140] For the intermediate value of the video type, P3 = 0.4 * 0.02 + 0.1 * 0.02 + 0.5 * 0.01 = 0.015.

[0141] For the intermediate value of the gene type, P4 = 0.4 * 0.03 + 0.1 * 0.08 + 0.5 * 0.0 = 0.02.

[0142] For the intermediate value of the database type, P5 = 0.4 * 0.01 + 0.1 * 0.0 + 0.5 * 0.02 = 0.014.

[0143] For other types of intermediate values, P6 = 0.4 * 0.09 + 0.1 * 0.2 + 0.5 * 0.05 = 0.081.

[0144] Based on the accuracy threshold, the processor 101 determines that the median value of the text type among the above six median values ​​is greater than 0.8 (P1 = 0.85), therefore the data type of the input data is text.

[0145] In some embodiments, the data compression model corresponding to the text type can be obtained by using the text type lookup table 1. The data compression models corresponding to the text type include the text model, the word model, and the nest model.

[0146] After determining the text type, processor 101 sends the data to be compressed to processor 103 and instructs processor 103 to compress the data according to the data compression model corresponding to the text type. Assume the initial probability of the next bit being 1 for the text model is 0.5 for all models. The initial probability of the next bit being 1 for the word model is 0.5 for all models. The initial probability of the next bit being 1 for the nested model is 0.2 for all models. The initial weight ratio for the text model is 0.33. The initial weight ratio for the word model is 0.33. The initial weight ratio for the nested model is 0.34. The data to be compressed is 01001101. Understandably, the data to be compressed includes eight bits, i.e., Q = 8. y is an integer, y ∈ [1, 8]. When processor 103 reads the y-th bit, the probabilities of the y-th bit being 1 predicted by the three models are updated, and the updated bit probabilities are shown in Table 2 below.

[0147] Table 2

[0148]

[0149] Similarly, the weight ratios for each model are adjusted as shown in Table 3 below.

[0150] Table 3

[0151]

[0152] The processor 103 synthesizes the initial probability values ​​of the bits calculated by the three data compression models to obtain the probability of each bit of the data to be compressed. The probability of each bit satisfies formula (4).

[0153] P = S1 * P N1 +S2*P N2 +S3*P N3 (4)

[0154] The probability of the next bit being 1 is shown in Table 4 below.

[0155] Table 4

[0156]

[0157] Optionally, the processor 103 can use an artificial neural network to weight and calculate the probability from multiple initial probability values. The artificial neural network includes, but is not limited to, models using Long Short-Term Memory (LSTM), gated recurrent units (GRU), fully connected models (FC), or other neural network models. Therefore, synthesizing multiple initial probability values ​​using an artificial neural network can yield more accurate bit probabilities, thereby improving the compression ratio of the data to be compressed.

[0158] The processor 103 encodes the data to be compressed according to the probability of each bit of the data to be compressed and the arithmetic coding algorithm, and obtains the arithmetic-coded interval, as shown in Table 5 below.

[0159] Table 5

[0160]

[0161] The processor 103 determines the compressed data according to formula (5).

[0162] z1≤c*2^(-b)<(c+1)*2^(-c)≤z2 (5)

[0163] Where c is an integer and b is the minimum number of bits. z1 is the lower bound of the probability interval for the bit, and z2 is the upper bound of the probability interval for the bit. Assume the probability of predicting a bit as 1 is P. If the bit read is 1, then update the lower bound z1 = z1 + (1 - P) * (z2 - z1). If the bit read is 0, then update the upper bound z2 = z1 + (1 - P) * (z2 - z1).

[0164] At this point, the minimum value b is found to be 6, and the value of c is 23. The final output encoded value is the binary value of 23, which is the compressed data 010111.

[0165] When predicting contextual matching information, the parameters of the data compression model need to be continuously adjusted based on existing probabilities and updated contextual information in order to output more accurate probability predictions.

[0166] In one example, processor 103 predicts the next bit based on the currently occurring consecutive bits, requiring information about the current bit and the number of occurrences n. The number of occurrences n represents the number of times the current bit has appeared consecutively. The larger n is, the higher the probability that the predicted next bit will be the same as the current bit. Whenever a bit differs from the previous consecutive bits, the information about the current bit and the number of occurrences n are updated to 1.

[0167] Suppose the current bit y is 0, and there have been 3 consecutive bits of 0, i.e., n = 3. The processor 103 predicts the probability of the (y+1)th bit being 1 as 0.3. After writing this probability into the arithmetic encoder, when the probability of reading the (y+1)th bit is 0, the information of the current bit required in the data compression model remains unchanged, and the number of iterations n is updated to 4. The probability of predicting the (y+2)th bit being 1 then becomes 0.25, because the increased frequency of bit 0 has led to an increase in the probability of predicting a bit of 0, while the probability of predicting a bit of 1 has decreased accordingly. If the probability of reading the (y+1)th bit is 1, then the current bit is updated to 1, and n is also updated to 1. The probability of predicting the (y+2)th bit being 1 then increases to 0.55.

[0168] In some embodiments, the probability update module can continuously update the feature information required by the probability prediction module based on the currently read bits, ensuring the accuracy of the probability prediction module in predicting the next bit.

[0169] A probabilistic mixture model can synthesize the initial probability values ​​of a bit calculated by multiple probabilistic statistical models to obtain the most accurate bit probability. The probabilistic mixture module can use a neural network to weight and calculate the bit probability from multiple initial probability values. Neural networks include, but are not limited to, those using long short-term memory models, gated recurrent units, fully connected models, or other neural network models.

[0170] The arithmetic encoding module performs arithmetic encoding operations on the probabilities of each bit to achieve data compression. Arithmetic encoding is complex, has high computational overhead, and is tightly integrated with the probability and statistics module. To reduce computational overhead, arithmetic compression algorithms are all bit-wise encoded, making hardware execution of the arithmetic encoding module highly efficient. In contrast to the compression process, the decompression stage uses an arithmetic decoder to calculate the original data. Since the parameters of the data compression model are stored in the encoded data (e.g., the data compression model corresponding to the data type) during the compression stage, the decoding stage only needs to use the data compression model and does not require the data type identification model; therefore, decompression is faster than compression. The arithmetic encoding module can be implemented by an arithmetic encoder, which can be located in processor 103.

[0171] In other embodiments, after receiving the data to be compressed, the processor 101 identifies the data type of the data and selects one or more data compression models based on the data type. The processor 101 then compresses the data to be compressed according to the selected data compression models. The difference from the embodiments described above is that the processor 103 does not need to compress the data according to the selected data compression models. The specific methods for selecting one or more data compression models and compressing the data to be compressed can be found in the descriptions of the embodiments above and will not be repeated here.

[0172] In other embodiments, several data compression models can be stored in the processor 103. The processor 103 uses the several data compression models to obtain the probability of each bit of the data to be compressed, and encodes the data to be compressed according to the probability of each bit in the data to be compressed and an arithmetic coding algorithm to obtain compressed data. The several data compression models include commonly used data compression models among all data compression models in conventional technology and other data compression models generated by superimposing commonly used data compression models. The specific method for compressing the data to be compressed can be referred to the description in the above embodiments, and will not be repeated here.

[0173] It is understood that, in order to achieve the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0174] Figure 5 This is a schematic diagram of a possible data compression apparatus provided for embodiments of this application. These data compression apparatuses can be used to implement the functions of the computing device in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In the embodiments of this application, the data compression apparatus can be as follows: Figure 1 The computing device 100 shown can also be a module (such as a chip) applied to a computing device.

[0175] like Figure 5 As shown, the data compression device 500 includes a first processing module 510 and a second processing module 520. The data compression device 500 is used to implement the above-mentioned... Figure 2 or Figure 4 The method embodiment shown illustrates the functionality of the computing device.

[0176] When the data compression device 500 is used to achieve Figure 2When calculating the function of the computing device in the method embodiment shown: the first processing module 510 is used to execute S201 to S203; the second processing module 520 is used to execute S204.

[0177] When the data compression device 500 is used to achieve Figure 4 In the method embodiment shown, when calculating the function of the device: the first processing module 510 is used to execute S201a, S201b, S202 and S203; the second processing module 520 is used to execute S204.

[0178] For a more detailed description of the first processing module 510 and the second processing module 520 mentioned above, please refer to [link / reference]. Figure 2 or Figure 4 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0179] For example, in a specific product implementation, the first processing module 510 can be... Figure 1 The processor 101 shown may have a second processing module that can be... Figure 1 The processor 103 is shown. In other examples, the first processing module 510 and the second processing module 520 may be two components with computing capabilities, or they may be two computing units in the same component. This embodiment does not limit the organization between the first processing module 510 and the second processing module 520.

[0180] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0181] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects.

[0182] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A data compression method, characterized in that, The method is applied to a computing device; the computing device includes a first processor and a second processor, and the method includes: The first processor receives data to be compressed, identifies the data type of the data to be compressed, and selects one or more data compression models according to the data type. The data compression model is used to predict the probability of each bit in the data to be compressed. The second processor compresses the data to be compressed according to the selected one or more data compression models and arithmetic coding algorithms.

2. The method according to claim 1, characterized in that, The method further includes: The first sample data is used to train several data compression models; The plurality of data compression models or a subset of the plurality of data compression models are stored in the computing device.

3. The method according to claim 2, characterized in that, The method further includes: The data compression model stored in the computing device is upgraded.

4. The method according to claim 1, characterized in that, Each data compression model has an identifier, and the method further includes: The corresponding data compression model is obtained based on the identifier.

5. The method according to claim 1, characterized in that, The method further includes: The correspondence between the data type of the data to be compressed and the one or more data compression models is stored in the computing device.

6. The method according to any one of claims 1-5, characterized in that, The computing device stores M data type recognition models, each model used to identify the probability of P data types occurring. The identification of the data types to be compressed includes: Based on the M data type identification models, M probability sets corresponding to the data to be compressed are determined. Each probability set contains P initial probability values. Each of the P initial probability values ​​is used to indicate the possibility that the data to be compressed is a data type. M is an integer greater than or equal to 1, and P is an integer greater than or equal to 1. The data type of the data to be compressed is determined based on the M probability sets.

7. The method according to claim 6, characterized in that, Determining the data type of the data to be compressed based on the M probability sets includes: Based on the M probability sets and the weight ratios of the M data types, P intermediate values ​​are obtained for the identification model. The data type corresponding to the median value that is greater than the accuracy threshold among the P median values ​​is determined as the data type of the data to be compressed.

8. The method according to claim 6, characterized in that, The method further includes: The second sample data is used to train the model to generate the M data type recognition models.

9. A computing device, characterized in that, The computing device includes multiple processors, including a first processor and a second processor, wherein... The first processor is configured to receive data to be compressed, identify the data type of the data to be compressed, and select one or more data compression models according to the data type, wherein the data compression model is used to predict the probability of each bit in the data to be compressed; The second processor is used to compress the data to be compressed according to the selected one or more data compression models and arithmetic coding algorithms.

10. The computing device according to claim 9, characterized in that, The computing device further includes a memory, and the first processor is also configured to store the correspondence between the data type of the data to be compressed and the one or more data compression models in the memory.

11. The computing device according to claim 9, characterized in that, The computing device also includes an artificial intelligence (AI) chip. The AI ​​chip is used to train the first sample data to generate several data compression models; The first processor is further configured to store the plurality of data compression models or a subset of the plurality of data compression models in the computing device.

12. The computing device according to claim 11, characterized in that, The first processor or the second processor is further configured to upgrade the data compression model stored in the computing device.

13. The computing device according to claim 9, characterized in that, Each data compression model has an identifier. The first processor is further configured to send the identifier of the selected data compression model to the second processor; The second processor is also used to obtain the corresponding data compression model based on the identifier.

14. The computing device according to any one of claims 9-13, characterized in that, The computing device stores M data type recognition models, each model used to identify the probability of P data types appearing. The first processor is specifically used to determine M probability sets corresponding to the data to be compressed based on the M data type identification models. Each probability set contains P initial probability values. Each of the P initial probability values ​​is used to indicate the possibility that the data to be compressed is a data type. M is an integer greater than or equal to 1, and P is an integer greater than or equal to 1. The data type of the data to be compressed is determined based on the M probability sets.

15. The computing device according to claim 14, characterized in that, The first processor is specifically used to obtain P intermediate values ​​based on the weight ratios of the M probability sets and the M data types to identify the model; The data type corresponding to the median value that is greater than the accuracy threshold among the P median values ​​is determined as the data type of the data to be compressed.

16. The computing device according to claim 14, characterized in that, The computing device also includes an artificial intelligence (AI) chip, which is used to train the second sample data to generate the M data type recognition models.

17. The computing device according to any one of claims 9-13, characterized in that, The first processor is a central processing unit, and the second processor is a coprocessor.

18. A data compression device, characterized in that, The device is located in a computing device, and the device includes a first processing module and a second processing module; The first processing module is used to receive data to be compressed, identify the data type of the data to be compressed, and select one or more data compression models according to the data type, wherein the data compression model is used to predict the probability of each bit in the data to be compressed; The second processing module is used to compress the data to be compressed according to the selected one or more data compression models and arithmetic coding algorithms.

19. The apparatus according to claim 18, characterized in that, The data compression device further includes a storage module, and the first processing module is also used to store the correspondence between the data type of the data to be compressed and the one or more data compression models in the storage module.

20. The apparatus according to claim 18, characterized in that, The device also includes an artificial intelligence (AI) chip. The AI ​​chip is used to train the first sample data to generate several data compression models; The first processing module is further configured to store the plurality of data compression models or a subset of the plurality of data compression models in the computing device.

21. The apparatus according to claim 18, characterized in that, The first processing module or the second processing module is further configured to upgrade the data compression model stored in the computing device.

22. The apparatus according to claim 18, characterized in that, Each data compression model has an identifier. The first processing module is further configured to send the identifier of the selected data compression model to the second processing module; The second processing module is also used to obtain the corresponding data compression model based on the identifier.

23. The apparatus according to any one of claims 18-22, characterized in that, The computing device stores M data type recognition models, each model used to identify the probability of P data types appearing. The first processing module is specifically used to determine M probability sets corresponding to the data to be compressed based on the M data type identification models. Each probability set contains P initial probability values. Each of the P initial probability values ​​is used to indicate the possibility that the data to be compressed is a data type. M is an integer greater than or equal to 1, and P is an integer greater than or equal to 1. The data type of the data to be compressed is determined based on the M probability sets.

24. The apparatus according to claim 23, characterized in that, The first processing module is specifically used to obtain P intermediate values ​​based on the weight ratios of the M probability sets and the M data types to identify the model; The data type corresponding to the median value that is greater than the accuracy threshold among the P median values ​​is determined as the data type of the data to be compressed.

25. The apparatus according to claim 23, characterized in that, The device also includes an artificial intelligence (AI) chip, which is used to train the second sample data to generate the M data type recognition models.

26. The apparatus according to any one of claims 18-22, characterized in that, The first processing module is a central processing unit, and the second processing module is a coprocessor.

Images