Data processing method and related equipment

CN118923045AActive Publication Date: 2024-11-08HUAWEI TECH CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202280094057.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-11-08
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

Existing data compression methods based on neural networks have shortcomings in compression speed and efficiency, resulting in slow compression speed and difficulty in meeting the rapidly developing data processing needs.

Method used

By clustering the context bytes of each target byte in the data, a feature vector is generated, and a neural network model is used for probability estimation, thereby achieving efficient compression and decompression of data and reducing the amount of calculation and resource overhead.

Benefits of technology

While maintaining a good data compression rate, the data processing speed is significantly improved, and the resource overhead of data processing, transmission and storage is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118923045A_ABST
    Figure CN118923045A_ABST
Patent Text Reader

Abstract

In the data processing method provided by the present application, after obtaining input data (201), a feature vector corresponding to each target byte in a plurality of bytes of the input data can be obtained (202), the feature vector being obtained based on a clustering result of context bytes corresponding to the corresponding target byte. Then, based on a neural network model and the feature vector corresponding to each target byte, obtaining a probability estimation result corresponding to each target byte (203); and obtaining a processing result corresponding to the input data according to the probability estimation result corresponding to each target byte (204), the processing result being a compression result corresponding to the input data, or a decompression result corresponding to the input data. By means of the method, the good data compression rate can be kept while the good data processing speed is guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

A data processing method and related equipment Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and related equipment. Background Art

[0002] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0003] With the rapid development of AI, neural network models (e.g., deep neural network models) are being deployed in various devices to process and analyze a variety of information, including images, videos, text, and voice. Several neural network-based data compression methods have emerged in the field of data compression. These methods typically offer good compression rates, but because neural network operations are often complex and inefficient, these current methods typically offer slow compression speeds.

[0004] Summary of the Invention

[0005] The present application provides a data processing method to improve the processing speed when compressing and decompressing data. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0006] The first aspect of the present application provides a data processing method, which includes: obtaining input data, where the input data is data to be compressed or data to be decompressed, and the input data includes multiple bytes; obtaining a feature vector corresponding to each target byte in the multiple bytes, where the feature vector is obtained based on a clustering result of context bytes corresponding to the corresponding target byte; obtaining a probability estimation result corresponding to each target byte based on a neural network model and the feature vector corresponding to each target byte; and obtaining a processing result corresponding to the input data based on the probability estimation result corresponding to each target byte, where the processing result is a compression result corresponding to the input data, or a decompression result corresponding to the input data.

[0007] In the first aspect, after obtaining the input data, a feature vector corresponding to each target byte in the multiple bytes of the input data can be obtained. The feature vector is obtained based on the clustering results of the context bytes corresponding to the corresponding target byte. In this case, by clustering the context bytes, the context bytes can usually be grouped according to the clustering results, and then data processing is performed based on the grouping to obtain the feature vector of the target byte, thereby obtaining a feature vector with a smaller data dimension. Then, based on the neural network model and the feature vector corresponding to each target byte, a probability estimation result corresponding to each target byte is obtained; and based on the probability estimation result corresponding to each target byte, the processing result corresponding to the input data is obtained. In this case, due to the smaller data dimension of the feature vector, the data processing speed of the neural network model for the feature vector can be effectively improved, thereby improving the speed of data compression or data decompression. The data compression method or data decompression method for obtaining the processing result corresponding to the input data based on the probability estimation result corresponding to each target byte corresponds to a better data compression rate. Therefore, through the first aspect, it is possible to maintain a good data compression rate while ensuring a good data processing speed.

[0008] In a possible implementation of the first aspect, the input data is data to be compressed; based on the probability estimation result corresponding to each target byte, a processing result corresponding to the input data is obtained, including: based on the probability estimation result corresponding to each target byte, obtaining a first encoding result corresponding to the target byte; based on the first encoding result corresponding to the target byte, obtaining a processing result corresponding to the input data.

[0009] In this possible implementation, the compressed data packet does not need to include probability distribution information for decoding, but instead includes uncompressed bytes and a first encoding result obtained by compressing part of the bytes in the input data. During subsequent decompression, the probability distribution information for decompression is inferred based on the uncompressed bytes and the neural network model. This can reduce the amount of data in the compressed data packet, ensure a better compression ratio, and maintain a high compression rate.

[0010] In a possible implementation of the first aspect, the input data is data to be decompressed, and the input data includes multiple bytes and a second encoding result; based on the probability estimation result corresponding to each target byte, a processing result corresponding to the input data is obtained, including: based on the probability estimation result corresponding to the target byte, the second encoding result in the input data is decoded to obtain a decoding result; based on the decoding result and multiple bytes, a processing result corresponding to the input data is obtained.

[0011] In this possible implementation, based on the neural network model and the feature vector corresponding to each target byte, obtaining the probability estimation result corresponding to each target byte can be understood as inferring the probability distribution information applied in the encoding process corresponding to the second encoding result, that is, inferring the probability distribution information required for the second encoding result. Therefore, after obtaining the probability estimation result corresponding to the target byte, the probability distribution information required for decoding the second encoding result can be inferred based on the probability estimation results corresponding to each target byte, and then the second encoding result can be decoded. Since multiple bytes in the input data include uncompressed bytes, a complete decoding result can be obtained by combining these multiple bytes and the decoding result corresponding to the second encoding result.

[0012] In a possible implementation of the first aspect, obtaining a feature vector corresponding to each target byte in a plurality of bytes includes: obtaining a clustering result of context bytes corresponding to the target byte, the clustering result including g groups of bytes, each group of bytes in the g groups of bytes including at least two bytes, and g being a positive integer; mapping each byte in the context bytes corresponding to the target byte through a mapping matrix to obtain a mapping result corresponding to each byte in the context bytes; and merging multiple mapping results corresponding to each group of bytes to obtain a feature vector corresponding to the corresponding target byte.

[0013] In this possible implementation, the number of bytes in each group and the value of g can be predetermined. This way, the data dimensions in the clustering result are relatively fixed, making it easier to use as input for the neural network model. The dimensions of the mapping matrix can be determined based on g and the number of bytes in the context bytes.

[0014] Compared to existing solutions that directly map context bytes without clustering them, this possible implementation significantly reduces the dimension of the feature vector obtained. This demonstrates that clustering context bytes and then performing vector mapping based on the mapping matrix and clustering results to obtain the feature vector of the target byte significantly reduces the dimension of the target byte's feature vector, thereby significantly reducing the associated computational effort, lowering resource overhead for data processing, transmission, and storage, and significantly improving the processing speed of data compression and decompression.

[0015] In a possible implementation manner of the first aspect, the mapping matrix and the neural network model are obtained after joint training based on training data, and the type of the training data is the same as the type of the input data.

[0016] In a possible implementation of the first aspect, the method is applied to a computer device, which includes a first processor and a second processor; obtaining input data, including: obtaining input data through the first processor; obtaining a feature vector corresponding to each target byte in a plurality of bytes, including: obtaining a feature vector corresponding to each target byte in a plurality of bytes through the second processor; obtaining a probability estimation result corresponding to each target byte based on a neural network model and the feature vector corresponding to each target byte, including: obtaining a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte through the second processor; obtaining a processing result corresponding to the input data based on the probability estimation result corresponding to each target byte, including: obtaining a processing result corresponding to the input data based on the probability estimation result corresponding to each target byte through the first processor.

[0017] In this possible implementation, the first processor may be a general-purpose processor such as a CPU, and the second processor may be a dedicated processor such as a GPU or an NPU.

[0018] In a possible implementation of the first aspect, input data is stored in a first processor; a feature vector corresponding to each target byte in a plurality of bytes is obtained by a second processor, including: for each target byte, updating a first-in-first-out (FIFO) storage queue of the second processor according to a difference byte transmitted by the first processor, where the difference byte is a byte added to a context byte corresponding to the target byte relative to a context byte of a previous target byte of the target byte; obtaining, by the second processor, a context byte corresponding to the target byte from the FIFO storage queue; and obtaining, by the second processor, a feature vector corresponding to the target byte based on the context byte corresponding to the target byte.

[0019] In this possible implementation, updating the FIFO storage queue of the second processor based on the difference bytes means that the difference bytes are queued from the end of the FIFO storage queue, and at the same time, the bytes at the head of the FIFO storage queue that have the same number of bytes as the difference bytes are dequeued. At this time, the context bytes of the target byte are stored at a specific position in the FIFO storage queue. In this way, the second processor can directly read the data in the FIFO storage queue to obtain the context byte information of the target byte. In the process of transferring data to the second processor, the context bytes of the target byte and the context bytes of the previous target byte do not need to be repeatedly transmitted, thereby reducing the amount of data transferred between the first processor and the second processor, accelerating the entire data processing process, and improving data processing efficiency.

[0020] In a possible implementation of the first aspect, the probability estimation result corresponding to each target byte includes the probability upper bound value and the probability lower bound value corresponding to the corresponding target byte; the first processor obtains the processing result corresponding to the input data based on the probability estimation result corresponding to each target byte, including: obtaining the probability upper bound value and the probability lower bound value corresponding to each target byte from the second processor through the first processor; and obtaining the processing result corresponding to the input data based on the probability upper bound value and the probability lower bound value corresponding to each target byte through the first processor.

[0021] In this possible implementation method, only the probability upper limit value and the probability lower limit value corresponding to the target byte need to be transmitted to the first processor, without transmitting the probability estimation results corresponding to all target bytes. This can reduce the amount of information transmitted from the second processor to the first processor and shorten the information transmission time.

[0022] In a possible implementation of the first aspect, the first processor obtains a processing result corresponding to the input data based on the probability estimation result corresponding to each target byte, including: obtaining first information from the second processor through the first processor, the first information including probability estimation results corresponding to the H bytes with the highest probability among all target bytes, where H is a positive integer; obtaining the processing result corresponding to the input data through the first processor based on the first information.

[0023] In this possible implementation, the second processor transmits the probability estimation results corresponding to a portion of target bytes with higher probability to the first processor without transmitting the probability estimation results corresponding to all target bytes. This can reduce the amount of information transmitted from the second processor to the first processor and shorten the information transmission time.

[0024] In a possible implementation of the first aspect, a first processor obtains a processing result corresponding to the input data based on a probability estimation result corresponding to each target byte, including: obtaining, through the first processor, second information from a second processor, where the second information is used to indicate a fitting result obtained by fitting the probability estimation result corresponding to the target bytes in multiple bytes; and obtaining, through the first processor, a processing result corresponding to the input data based on the second information.

[0025] In this possible implementation, the probability distribution information can be fitted using a specified fitting function, and the obtained fitting result can include the parameter value of the specified parameter in the specified fitting function. Correspondingly, the second information can include the parameter value of the specified parameter in the fitting function obtained after fitting, so as to describe the fitting result using the parameter value. In this way, a second processor such as a GPU or NPU can transmit the second information to a first processor such as a CPU without having to transmit the complete probability distribution information, thereby greatly reducing the amount of information transmitted from the second processor to the first processor and shortening the information transmission time.

[0026] In a possible implementation manner of the first aspect, the context bytes corresponding to the target byte include first P bytes of the target byte among the multiple bytes, where P is an integer greater than 1.

[0027] In a possible implementation of the first aspect, the neural network model includes K fully connected layers, where K is a positive integer.

[0028] In this possible implementation, the neural network model implements probability estimation through at least two fully connected layers, and can achieve data compression or data decompression through a lightweight structure and with low overhead, thereby greatly improving the data processing speed.

[0029] A second aspect of the present application provides a data processing device having the functionality to implement the method of the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functionality, such as an acquisition module, a vector module, a probability estimation module, and a processing module.

[0030] A third aspect of the present application provides a computer device, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable by the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0031] The fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0032] The fifth aspect of the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0033] A sixth aspect of the present application provides a chip system, which includes a processor for supporting a computer device in implementing the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the computer device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0034] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application;

[0036] FIG2 is an exemplary schematic diagram of a data processing method provided in an embodiment of the present application;

[0037] FIG3 is an exemplary schematic diagram of a process for obtaining a feature vector of a target byte according to an embodiment of the present application;

[0038] FIG4 is an exemplary schematic diagram of an information processing flow provided in an embodiment of the present application;

[0039] FIG5a is an exemplary schematic diagram of input data and processing results of the input data provided by an embodiment of the present application;

[0040] FIG5 b is another exemplary schematic diagram of input data and processing results of the input data provided by an embodiment of the present application;

[0041] FIG6 is an exemplary schematic diagram of an information interaction process between a first processor and a second processor provided in an embodiment of the present application;

[0042] FIG7 is an exemplary schematic diagram of steps 2021-2023 provided in an embodiment of the present application;

[0043] FIG8 is a schematic diagram of an embodiment of a data processing device provided in an embodiment of the present application;

[0044] FIG9 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0046] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate. This is merely a way of distinguishing objects with the same properties when describing them in the embodiments of this application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0047] Since the embodiments of the present application involve the field of artificial intelligence, for ease of understanding, the following first introduces relevant terms and related concepts such as neural networks involved in the embodiments of the present application.

[0048] Please refer to Figure 1, which shows a schematic diagram of the main framework of artificial intelligence. The following explains this framework from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem.

[0049] (1) Infrastructure

[0050] The infrastructure provides computing power for AI systems, enabling communication with the outside world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips (central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field programmable gate array (FPGA), tensor processing unit (TPU), and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks, as well as related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to obtain data, which is then provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

[0051] (2) Data

[0052] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0053] (3) Data processing

[0054] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0055] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0056] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0057] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0058] (4) General ability

[0059] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0060] (5) Smart products and industry applications

[0061] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0062] (6) Neural Network

[0063] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0064]

[0065] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0066] There are many types of neural networks, such as deep neural networks (DNNs), also known as multi-layer neural networks, which are neural networks with multiple hidden layers; and convolutional neural networks (CNNs), which are deep neural networks with convolutional structures. This application does not limit the specific types of neural networks involved.

[0067] (7) Fully connected layer

[0068] Each node in the fully connected layer is connected to all nodes in the previous layer. In the probability density calculation process, which is a linear weighted summation process, each output of the fully connected layer can be regarded as each node in the previous layer multiplied by a weight coefficient W.

[0069] y=Wx

[0070] The weight matrix of the fully connected layer is fixed. Due to its fully connected nature, the fully connected layer generally has the most parameters.

[0071] Since Shannon proposed the theory of information entropy in 1950, the field of data compression has developed rapidly. One direction of data compression development has been the pursuit of high compression ratios. For example, the Cmix algorithm can achieve good compression ratios for a variety of data types, but the compression speed is slow, even less than 1KB / s in some cases. Another direction of development has been the pursuit of higher compression speeds. For example, the LZ4 and LZ77 algorithms can achieve compression speeds of hundreds of megabytes per second, but the compression ratio is relatively poor.

[0072] With the rapid development of AI, neural network models (e.g., deep neural network models) are being deployed in various devices to process and analyze a variety of information, including images, videos, text, and voice. Several neural network-based data compression methods have emerged in the field of data compression. These methods typically offer good compression rates, but because neural network operations are often complex and inefficient, current neural network-based data decompression methods typically offer slow compression speeds.

[0073] Based on the above problems, an embodiment of the present application provides a data processing method that can improve the compression speed while maintaining a good compression rate.

[0074] The data processing method of the embodiment of the present application is described in detail below.

[0075] The data processing method in the embodiment of the present application can be applied to a computer device, which can be a terminal device, or a server, a container, or a virtual machine.

[0076] Exemplarily, the terminal device may be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a wireless terminal in the Internet of Things (IoT), etc.

[0077] The computer device may be deployed with a neural network model that can output corresponding probability distribution information based on an input feature vector. The neural network model can be of various types, for example, a recurrent neural network (RNN) or other neural network including a fully connected layer.

[0078] As shown in FIG2 , the data processing method specifically includes steps 201 - 204 .

[0079] Step 201: Obtain input data.

[0080] The input data is data to be compressed or data to be decompressed, and the input data includes multiple bytes.

[0081] The embodiments of the present application can be applied to data compression scenarios as well as data decompression scenarios. In either data compression scenario or data decompression scenario, the corresponding input data includes multiple bytes.

[0082] The data type corresponding to the input data is not limited herein. For example, the input data may include one or more of text data, voice data, image data, or video data. In other words, the type of data described by the multiple bytes in the input data is not limited herein. The multiple bytes in the input data may form a byte stream.

[0083] The input data may come from a variety of sources. For example, the input data may be received by the computer device executing the embodiments of the present application from another device via a wireless communication connection or a wired communication connection, may be collected by a sensor device such as a camera in the computer device, or may be generated by a related program in the computer device.

[0084] The storage location of the input data in the computer device is not limited here.

[0085] In some examples, the input data may be stored in a first processor of a computer device, which may be a general-purpose processor such as a CPU. Steps 202-203 in the embodiment of the present application are performed by a second processor in the computer device, which may be a dedicated processor such as a GPU or NPU. In this case, the input data may be transferred from the first processor to the second processor.

[0086] Step 202: Obtain a feature vector corresponding to each target byte in the plurality of bytes.

[0087] The feature vector is obtained based on the clustering results of the context bytes corresponding to the corresponding target bytes.

[0088] After obtaining the input data, the computer device can obtain a feature vector corresponding to a target byte among multiple bytes.

[0089] The target byte among the plurality of bytes may be determined based on the scenario.

[0090] In the embodiment of the present application, the number of bytes in the target byte may be various. For example, each target byte may include one byte or multiple bytes.

[0091] In some examples, the target byte in the plurality of bytes may specifically be a byte in the plurality of bytes that has sufficient context bytes. The sufficient context bytes may refer to a number of context bytes corresponding to the corresponding byte reaching a predetermined number. For example, if the context bytes corresponding to a byte refer to the first P bytes of the corresponding byte in the byte stream, then if the number of bytes preceding a byte in the plurality of bytes is less than P, the byte is not the target byte. In other examples, the target byte in the plurality of bytes may be a byte in the plurality of bytes other than the first N bytes.

[0092] In embodiments of the present application, the number of context bytes corresponding to a target byte and the positional relationship between the context bytes and the corresponding target bytes can take various forms. For example, in some examples, the context bytes corresponding to the target byte include the first P bytes of the target byte among the multiple bytes, where P is a positive integer; while in other examples, the context bytes corresponding to the target byte also include the last G bytes of the target byte among the multiple bytes, where G is a positive integer.

[0093] In embodiments of the present application, there are various ways to cluster the context bytes corresponding to the target byte, which are not limited herein. For example, in the clustering results, identical or similar bytes in the context bytes of the target byte can be clustered into one group, while different bytes can be grouped into different groups. Thus, the clustering results can include multiple groups obtained by clustering the context bytes.

[0094] In this case, the bytes in a group of context bytes in the clustering result can be concatenated to obtain a byte vector corresponding to the group. This byte vector is then mapped based on a preset mapping rule to obtain a feature vector corresponding to the target byte. The preset mapping rule can be manually pre-set or obtained through joint training with a neural network model. The preset mapping rule can be represented in the form of a matrix.

[0095] In one embodiment, the above step 202 specifically includes the following steps:

[0096] Obtain a clustering result of context bytes corresponding to the target byte, the clustering result including g groups of bytes, each group of bytes in the F groups of bytes including at least two bytes, and g is a positive integer;

[0097] Map each byte in the context bytes corresponding to the target byte through the mapping matrix to obtain a mapping result corresponding to each byte in the context bytes;

[0098] The multiple mapping results corresponding to each group of bytes are merged to obtain the feature vector corresponding to the corresponding target bytes.

[0099] The number of bytes in each group and the value of g can be predetermined. This way, the data dimensions in the clustering result are relatively fixed, making it easier to use them as input to the neural network model. The dimensions of the mapping matrix can be determined based on g and the number of bytes in the context bytes.

[0100] A specific example is given below to illustrate a specific implementation of the embodiment of the present application.

[0101] FIG3 is a schematic diagram of the process of obtaining the feature vector of the target byte. In the example of FIG3 , the target byte x iThe context bytes are {x i-c ,…,x i-1}.

[0102] For {x i-c ,…,x i-1}Clustering is performed to obtain g groups of bytes, where each c / g bytes form a group, and each group of bytes forms a byte vector v of (g, c / g).

[0103] Each byte in the byte vector v is mapped through a mapping matrix of size (256, 256*g / c). The mapping matrix contains 256 vectors, each of which is 256*g / c long and corresponds to a state of the byte. After the mapping matrix maps each byte in the byte vector v, a tensor of (g, c / g, 256*g / c) is obtained. Then, based on the grouping of g groups of bytes, the vectors in the tensor are merged in the second dimension to obtain the target byte x i The feature vector q of , the size of q is (g, 256).

[0104] Generally speaking, if the context bytes are not clustered but directly mapped, the size of the feature vector of the target byte obtained is (c, 256), which is larger than the target byte x i The size of the feature vector q is (g, 256). It can be seen that by clustering the context bytes and then performing vector mapping based on the mapping matrix and the clustering results to obtain the feature vector of the target byte, the dimension of the feature vector of the target byte can be significantly reduced, thereby significantly reducing the related computational complexity, reducing the resource overhead of data processing, transmission, and storage, and greatly improving the processing speed of data compression and decompression.

[0105] It should be understood that the above example is merely an illustrative illustration of the embodiment of the present application, and is not a limitation to the embodiment of the present application.

[0106] Step 203: Based on the neural network model and the feature vector corresponding to each target byte, a probability estimation result corresponding to each target byte is obtained.

[0107] After obtaining the feature vector corresponding to the target byte, the feature vector can be input into the neural network model to obtain the probability estimation result of the neural network model for the feature vector.

[0108] The neural network model can realize probability estimation. The specific structure of the neural network model can be various, for example, it can be a neural network model such as RNN, or a neural network model of other structures and types.

[0109] In one embodiment, the neural network model may include at least two fully connected layers.

[0110] In an embodiment of the present application, the neural network model implements probability estimation through at least two fully connected layers, and can achieve data compression or data decompression through a lightweight structure and with low overhead, thereby greatly improving the data processing speed.

[0111] The following uses a specific example to illustrate an exemplary implementation of a neural network model.

[0112] Figure 4 is a schematic diagram of an information processing flow. In the example of Figure 4, the neural network model can be composed of three fully connected layers. The parameters of the three fully connected layers are 256x4096, 4096x256, and 256x256 respectively.

[0113] Process the target byte x through the three fully connected layers i The process of finding the eigenvectors consists of the following steps:

[0114] Target byte x i The feature vector q(g, 256) is input into the first fully connected layer (256x4096) to obtain the first intermediate layer output (g, 4096);

[0115] The output of the first intermediate layer (g, 4096) passes through the fully connected layer of the second layer (4096x256) to obtain the output of the second intermediate layer (g, 256);

[0116] The output of the second intermediate layer (g, 256) passes through the third layer (256x256) fully connected layer to obtain the output (g, 256). Take the last (256) vector in the output (g, 256) as x i The estimated output of

[0117] By normalizing the exponential function and x i The estimated output of x i The estimated probability distribution of .

[0118] x i The estimated probability distribution of can be a vector with a dimension of (256). By doing a cumulative sum on this vector, we can get x i The corresponding probability upper bound and probability lower bound, and x i The corresponding probability upper bound and probability lower bound are x i Of course, according to different scenarios, in some examples, x i The estimated probability distribution of x i The probability estimation results.

[0119] The normalized exponential function can transform a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that each element is between (0,1) and the sum of all elements is 1. The specific formula is:

[0120]

[0121] It can be seen that in the above example, vector mapping is performed based on the clustering results of the mapping matrix and context bytes, and the data dimension of the feature vector of the target byte obtained is smaller, which reduces the dimension of the input data of the neural network model, thereby reducing the relevant calculation amount in the neural network model. In addition, combined with a lightweight neural network model, the processing speed of data compression and data decompression can be greatly improved, and the resource overhead in the data processing process can be reduced.

[0122] Step 204: Obtain a processing result corresponding to the input data based on the probability estimation result corresponding to each target byte.

[0123] The processing result is the compression result corresponding to the input data, or the decompression result corresponding to the input data.

[0124] After obtaining the probability estimation result corresponding to each target byte, the processing result corresponding to the input data can be obtained based on the probability estimation result.

[0125] In an embodiment of the present application, different target bytes can be understood as different states of the byte. Then, based on the probability estimation result corresponding to each target byte, if the probability distribution information corresponding to the input data can be obtained, the probability distribution information can indicate the probability intervals corresponding to the different states of the byte.

[0126] For example, in one example, a byte can have 256 states, and all target bytes corresponding to the input data contain 256 different bytes. That is to say, based on all probability estimation results corresponding to all target bytes, the probability estimation results corresponding to the 256 states of the byte can be obtained. In this way, based on the probability estimation results corresponding to all target bytes, the probability distribution information (p0, p1,..., p255) corresponding to the input data can be obtained, where p0~p255 correspond one-to-one to the 256 states of the byte.

[0127] After obtaining the probability distribution information corresponding to the input data, the processing results corresponding to the input data can be obtained through algorithms such as Huffman coding and decoding and arithmetic coding and decoding.

[0128] In an embodiment of the present application, after obtaining the input data, a feature vector corresponding to each target byte in the multiple bytes of the input data can be obtained. The feature vector is obtained based on the clustering result of the context bytes corresponding to the corresponding target bytes. At this time, by clustering the context bytes, the context bytes can usually be grouped according to the clustering result, and then data processing is performed based on the grouping to obtain the feature vector of the target byte, thereby obtaining a feature vector with a smaller data dimension. Then, based on the neural network model and the feature vector corresponding to each target byte, the probability estimation result corresponding to each target byte is obtained; based on the probability estimation result corresponding to each target byte, the processing result corresponding to the input data is obtained. At this time, since the data dimension of the feature vector is small, the data processing speed of the neural network model for the feature vector can be effectively improved, thereby improving the speed of data compression or data decompression, and the data compression method or data decompression method of obtaining the processing result corresponding to the input data according to the probability estimation result corresponding to each target byte corresponds to a better data compression rate. Therefore, through the embodiment of the present application, while ensuring a better data processing speed, it can also maintain a better data compression rate.

[0129] In the embodiment of the present application, the specific process of step 204 in the data encoding scenario is different from the specific process of step 204 in the data decoding scenario. The following describes the data encoding scenario and the data decoding scenario in detail.

[0130] 1. Data encoding scenarios

[0131] In some embodiments, the input data is data to be compressed;

[0132] Based on the probability estimation results corresponding to each target byte, the processing results corresponding to the input data are obtained, including:

[0133] Obtaining a first encoding result corresponding to each target byte according to a probability estimation result corresponding to each target byte;

[0134] According to the first encoding result corresponding to the target byte, a processing result corresponding to the input data is obtained.

[0135] Let’s take arithmetic coding as an example.

[0136] In the arithmetic coding scenario, the probability estimation result corresponding to the target byte can be used to perform arithmetic coding on a portion of the multiple bytes to obtain a first coding result for the portion of bytes. Bytes other than the encoded portion of the multiple bytes of the input data can be directly encapsulated without being compressed. In this way, the compressed result corresponding to the input data can include the uncompressed bytes and the first coding result. The portion of bytes (i.e., the bytes to be arithmetically coded) can include a portion of the target bytes or all of the target bytes.

[0137] In this way, during the subsequent decoding process, the probability distribution information used in the data encoding process corresponding to the first encoding result can be inferred based on the uncompressed bytes and the neural network model, and the first encoding result can be decoded based on the inferred probability distribution information. The uncompressed bytes are combined with the decoding result corresponding to the first encoding result to obtain a complete decoding result.

[0138] In this method, the compressed data packet does not need to include probability distribution information for decoding, but instead includes uncompressed bytes and a first encoding result obtained by compressing part of the bytes in the input data. During subsequent decompression, the probability distribution information used for decompression is inferred based on the uncompressed bytes and the neural network model. This can reduce the amount of data in the compressed data packet, ensure a better compression ratio, and maintain a high compression rate.

[0139] In some embodiments, the target byte is a byte in the plurality of bytes excluding the first N bytes, where N is an integer greater than 1;

[0140] The processing result corresponding to the input data includes the first encoding result corresponding to the target byte and the first N bytes.

[0141] In the embodiment of the present application, in the compression result corresponding to the input data, the uncompressed bytes are the first N bytes of the multiple bytes of the input data, where the value of N can be determined based on the accuracy of the probability estimation, the compression ratio, and other requirements.

[0142] For example, as shown in Figure 5a, a schematic diagram of input data and the processing results of the input data is provided. In the example shown in Figure 5a, N can be 64. In this case, the input data is the data to be compressed. Among the multiple bytes of the data to be compressed, the first 64 bytes are the bytes that do not need to be compressed, and the remaining bytes other than the 64 bytes are the target bytes. In this case, the compression result corresponding to the data to be compressed can include the first 64 bytes and a first encoding result corresponding to the bytes other than the 64 bytes, where the first encoding result is a numerical value.

[0143] 2. Data decoding scenarios

[0144] In some embodiments, the input data is data to be decompressed, and the input data includes a plurality of bytes and a second encoding result;

[0145] Based on the probability estimation results corresponding to each target byte, the processing results corresponding to the input data are obtained, including:

[0146] Decoding the second encoding result in the input data according to the probability estimation result corresponding to the target byte to obtain a decoding result;

[0147] According to the decoding result and the multiple bytes, a processing result corresponding to the input data is obtained.

[0148] In an embodiment of the present application, the input data is data to be decompressed, multiple bytes in the data to be decompressed are uncompressed bytes, and the second encoding result in the data to be decompressed is the result obtained by compression. The second encoding result can be a numerical value, and the specific generation process of the second encoding result can refer to the first encoding result in the embodiment related to Figure 5a, which will not be repeated here.

[0149] At this time, based on the neural network model and the feature vector corresponding to each target byte, obtaining the probability estimation result corresponding to each target byte can be understood as inferring the probability distribution information applied in the encoding process corresponding to the second encoding result, that is, inferring the probability distribution information required for the second encoding result. Therefore, after obtaining the probability estimation result corresponding to the target byte, the probability distribution information required for decoding the second encoding result can be inferred based on the probability estimation results corresponding to each target byte, and then the second encoding result can be decoded. Since multiple bytes in the input data include uncompressed bytes, a complete decoding result can be obtained by combining these multiple bytes and the decoding result corresponding to the second encoding result.

[0150] For example, as shown in Figure 5b, a schematic diagram of input data and the processing results of the input data is provided. In the example shown in Figure 5b, the first 64 bytes of the data to be decompressed are uncompressed bytes. In addition, the data to be decompressed also includes a second encoding result. The decompression result corresponding to the data to be decompressed may include the first 64 bytes and a decoding result corresponding to the second encoding result, wherein the decoding result corresponding to the second encoding result is obtained by decoding the second encoding result based on the first 64 bytes and the probability estimation information obtained by the neural network model.

[0151] In some embodiments, steps 201 and 204 may be performed by a first processor such as a CPU in a computer device, where the first processor may be a general-purpose processor, while steps 202 and 203 may be performed by a second processor such as a GPU or NPU, where the second processor may be a dedicated processor.

[0152] FIG6 is a schematic diagram showing information interaction between the first processor and the second processor in an embodiment of the present application.

[0153] In Figure 6, the first processor executes step 201. Then, the second processor executes steps 202 and 203 based on the input data obtained by the first processor. The first processor then executes step 204 based on the probability estimation result obtained by the second processor. In this way, based on the characteristics of different processors, the first and second processors can be controlled to perform the tasks that their respective processors are better at, thereby improving processing efficiency.

[0154] In some embodiments, the above step 202 specifically includes steps 2021-2023 as shown in FIG7 :

[0155] Step 2021: For each target byte, update the first-in-first-out (FIFO) storage queue of the second processor according to the difference byte transmitted by the first processor, where the difference byte is the byte added to the context byte corresponding to the target byte relative to the context byte of the previous target byte.

[0156] Step 2022: Obtain, by the second processor, a context byte corresponding to the target byte from the FIFO storage queue;

[0157] Step 2023: Obtain, through the second processor, a feature vector corresponding to the target byte based on the context byte corresponding to the target byte.

[0158] In an embodiment of the present application, input data can be transferred from the first processor to the second processor, and then the second processor obtains a feature vector corresponding to each byte of the multiple bytes based on the acquired input data.

[0159] Among them, the first input first output (FIFO) storage queue can be located in a buffer corresponding to the second processor. The buffer corresponding to the second processor can be independent hardware, integrated into the second processor, or a storage space shared by the second processor and the first processor, for example, a random access memory (RAM) in a computer device.

[0160] Updating the FIFO storage queue of the second processor according to the difference byte means that the difference byte is queued from the tail of the FIFO storage queue, and at the same time, the byte with the same number of bytes as the difference byte at the head of the FIFO storage queue is dequeued. At this time, the context byte of the target byte is stored at a specific position in the FIFO storage queue.

[0161] For example, the length of the FIFO storage queue is c, for the target byte x c+1 , target byte x c+1 The context bytes are {x1, ..., x c}, xc+1 The previous target byte x c The context bytes are {x0, ..., x c-1} has been stored in the FIFO storage queue. Then, at this time, we only need to c} with respect to {x0,…,x c-1} the difference in bytes x c Enter the queue from the end of the FIFO storage queue, and dequeue the x0 at the head of the FIFO storage queue, and the FIFO storage queue can be updated to {x1, ..., x c}.

[0162] Thus, in this example, the second processor directly reads the data in the FIFO storage queue to obtain the target byte x c+1 The context bytes {x1, ..., x c}, and in the process of data transfer to the second processor, {x1, ..., x c-1} information does not need to be transmitted repeatedly, thereby reducing the amount of data carried between the first processor and the second processor, accelerating the entire data processing process, and improving data processing efficiency.

[0163] In an embodiment of the present application, the specific steps of obtaining the feature vector corresponding to the target byte according to the context byte corresponding to the target byte through the second processor can refer to the relevant steps of any of the above embodiments and will not be repeated here.

[0164] In some embodiments, the above step 203 is usually performed by a second processor such as a GPU, NPU, etc., and the above step 204 is usually performed by a first processor such as a CPU. Therefore, after obtaining the probability estimation result corresponding to each byte through the neural network model, the second processor needs to send probability distribution information for data decompression or data compression to the first processor.

[0165] There are many ways in which the second processor sends the probability distribution information for data decompression or data compression to the first processor, which are described below with examples.

[0166] 1. In one embodiment, the probability estimation result corresponding to each target byte includes an upper probability limit value and a lower probability limit value corresponding to the corresponding target byte;

[0167] Obtaining, by the first processor, a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte, including:

[0168] Obtaining, by the first processor, an upper probability limit value and a lower probability limit value corresponding to each target byte from the second processor;

[0169] The first processor obtains a processing result corresponding to the input data based on the probability upper bound value and the probability lower bound value corresponding to each target byte.

[0170] In the data compression scenario, taking arithmetic coding as an example, for the target byte c, the target byte c can be compressed by the probability upper bound value and probability lower bound value of the target byte c. The probability interval corresponding to the target byte c can be [L,H), where the probability lower bound value L = p0 + p1 + ... + p(c-1), and the probability upper bound value H = L + p(c). Therefore, it is only necessary to transmit the probability upper bound value and probability lower bound value corresponding to the byte c to be compressed to the first processor, and the first processor can perform the encoding operation on the byte c to be compressed. At this time, there is no need to transmit the probability estimation results corresponding to all target bytes, which can reduce the amount of information transmitted from the second processor to the first processor and shorten the information transmission time.

[0171] It can be seen that in the embodiment of the present application, only the probability upper limit value and the probability lower limit value corresponding to the target byte need to be transmitted to the first processor, without transmitting the probability estimation results corresponding to all target bytes. This can reduce the amount of information transmitted from the second processor to the first processor and shorten the information transmission time.

[0172] 2. In one embodiment, obtaining, by the first processor, a processing result corresponding to the input data based on a probability estimation result corresponding to each target byte includes:

[0173] Obtaining, by the first processor, first information from the second processor, where the first information includes probability estimation results corresponding to H bytes with the highest probabilities among all target bytes, where H is a positive integer;

[0174] A processing result corresponding to the input data is obtained based on the first information through the first processor.

[0175] Among them, the value of H can be determined based on the scenario requirements and is not limited here. Except for the H bytes, the probabilities corresponding to the other bytes are evenly distributed. For example, H bytes correspond to H states of the 256 states of the byte, and the probabilities corresponding to the H bytes are (p(i1), p(i2), ...p(iH-1)), and the probabilities corresponding to the remaining 256-H bytes are all (1-(p(i1)+p(i2)+...+p(iH-1))) / (256-H). In this way, the first processor can obtain the probability estimation results of all states of the byte.

[0176] In an embodiment of the present application, the second processor transmits the probability estimation results corresponding to a portion of target bytes with higher probability to the first processor without transmitting the probability estimation results corresponding to all target bytes. This can reduce the amount of information transmitted by the second processor to the first processor and shorten the information transmission time.

[0177] 3. In one embodiment, obtaining, by the first processor, a processing result corresponding to the input data based on a probability estimation result corresponding to each target byte includes:

[0178] Obtaining, by the first processor, second information from the second processor, where the second information is used to indicate a fitting result obtained by fitting a probability estimation result corresponding to a target byte among the multiple bytes;

[0179] A processing result corresponding to the input data is obtained based on the second information through the first processor.

[0180] In an embodiment of the present application, the probability distribution information can be fitted using a specified fitting function, and the obtained fitting result may include the parameter value of the specified parameter in the specified fitting function. Correspondingly, the second information may include the parameter value of the specified parameter in the fitting function obtained after fitting, so as to describe the fitting result through the parameter value.

[0181] For example, for the probability distribution information of characters in 256 states (p0, p1, ..., p255), the probability distribution information can be fitted by a specific function, and then the parameter values ​​obtained after fitting are transferred from the GPU running the neural network model to the CPU.

[0182] Taking the Gaussian function as an example, the parameters of the Gaussian function include the mean u and the variance σ. The expression of the Gaussian function is as follows:

[0183]

[0184] Here, the value range of x is [0,255], u is the mean, σ is the variance, and f(x) represents the probability of x.

[0185] Based on the probability distribution information (p0, p1, ..., p255) and the Gaussian function for fitting, the values ​​of u and σ can be determined. Then, the second processor can transmit the values ​​of u and σ to the first processor, so that the first processor obtains the probability distribution information (p0, p1, ..., p255) based on the values ​​of u and σ, thereby determining the probability distribution of all states of the byte.

[0186] It can be seen that in an embodiment of the present application, the second information may include the parameter value of the fitting function obtained after fitting, and the parameter value can represent the fitting result obtained by fitting the probability estimation result corresponding to the target byte based on the specified function. The second processor such as GPU or NPU can transmit the second information to the first processor such as CPU without transmitting the complete probability distribution information, thereby greatly reducing the amount of data transmitted by the second processor to the first processor and shortening the information transmission time.

[0187] In an embodiment of the present application, the mapping matrix and the neural network model used to obtain the feature vector corresponding to the target byte are obtained after joint training based on the training data, and the type of the training data is the same as the type of the input data.

[0188] For example, the type of the training data and the type of the input data may be the same, that is, the training data is image data and the input data is also image data. Alternatively, the training data is text data in a certain language and the input data is also text data in the same language. Alternatively, the training data is speech data and the input data is also speech data.

[0189] If any of the above embodiments are understood as data processing processes in an application scenario, before applying the neural network model to process the actual scenario, the reference neural network model corresponding to the neural network model can be trained. In addition, in some of the above embodiments, the applied mapping matrix and neural network model can be obtained through joint training based on training data.

[0190] The following introduces the training process of the neural network model and mapping matrix.

[0191] The training data includes multiple training bytes, a corresponding reference context byte for each training byte, and a corresponding one-hot code for each training byte. A one-hot code is a binary code that assigns an N-bit code to each state of a signal source with N states. Only one bit is 1, and all others are 0. The number of 1 bits in the one-hot code for different states is guaranteed to be different.

[0192] In this example, a byte can have 256 states, and each training byte corresponds to one state of the byte. Therefore, the one-hot code corresponding to a training byte has 256 bits, where the position of the bit that is 1 in the one-hot code corresponding to different training bytes is different.

[0193] The untrained matrix corresponding to the mapping matrix is ​​called the reference matrix, and the untrained model corresponding to the neural network model is called the reference neural network model.

[0194] During one iteration of training:

[0195] Obtaining the training feature vector corresponding to the training bytes in the training data through the reference matrix;

[0196] By referring to the neural network model, the training feature vector is processed to obtain the training probability estimation result corresponding to the training bytes.

[0197] A loss value is calculated using a loss function based on the training probability estimation result and the single-hot code corresponding to the training byte. For example, the loss function may be a cross-entropy function, and the loss value may be a cross-entropy error. After obtaining the cross-entropy error, the reference neural network model and the reference matrix may be updated via backpropagation. Below, the backpropagation process is described using an example where the reference neural network model includes three fully connected layers.

[0198] Starting with the last of the three fully connected layers, the input to that layer is multiplied by the corresponding return error to obtain the gradient of the weights, where the return error for that layer is the cross-entropy error. This gradient is multiplied by a specified ratio, negated, and then added to the weights corresponding to that layer. This specified ratio can affect the speed and effectiveness of the training process and is therefore also called the "learning rate" or "step size." The direction of the gradient indicates the direction of error expansion, so the gradient needs to be negated before updating the weights to reduce the error caused by the weights. The return error for that layer (i.e., the cross-entropy error) is multiplied by the updated weights to obtain the return error for the second fully connected layer. The weights of this second fully connected layer are then updated using a similar method to the last fully connected layer update. This update process continues until the weights corresponding to all three fully connected layers and the weights corresponding to the reference matrix are fully updated. Since only some parameters in the reference matrix are involved in the calculation, only those parameters in the reference matrix that are currently involved are updated in this update process.

[0199] When the number of training reaches a preset number of times or the training results converge to the expected state, the training can be determined to be complete. After the training is completed, the parameters of the reference neural network model and the reference matrix can be fixed, and the fixed reference neural network model can be used as the neural network model in any of the above embodiments, and the fixed reference matrix can be used as the mapping matrix in any of the above embodiments.

[0200] During the training process, the process of processing data with reference to the neural network model and the reference matrix can refer to the relevant processing process of the neural network model and the mapping matrix in the above-mentioned related embodiments.

[0201] The training process in the embodiment of the present application can be implemented on the computer device that executes the above steps 201-204; or, it can also be implemented on other devices, for example, it can be implemented on a cloud server, and then the information of the neural network model and mapping matrix obtained after training is sent to the computer device that executes the above steps 201-204.

[0202] As shown in FIG8 , an embodiment of the present application provides a data processing device 80 , which can be applied to the computer device in the above embodiment.

[0203] The device 80 comprises:

[0204] An acquisition module 801 is configured to acquire input data, where the input data is data to be compressed or data to be decompressed, and the input data includes multiple bytes.

[0205] A vector module 802 is configured to obtain a feature vector corresponding to each target byte in the plurality of bytes, where the feature vector is obtained based on a clustering result of context bytes corresponding to the target byte;

[0206] A probability estimation module 803 is configured to obtain a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte;

[0207] The processing module 804 is used to obtain a processing result corresponding to the input data according to the probability estimation result corresponding to each target byte. The processing result is a compression result corresponding to the input data, or a decompression result corresponding to the input data.

[0208] Optionally, the input data is data to be compressed;

[0209] The processing module 804 is used to:

[0210] Obtaining a first encoding result corresponding to each target byte according to a probability estimation result corresponding to each target byte;

[0211] According to the first encoding result corresponding to the target byte, a processing result corresponding to the input data is obtained.

[0212] Optionally, the target byte is a byte other than the first N bytes in the plurality of bytes, where N is an integer greater than 1;

[0213] The processing result corresponding to the input data includes the first encoding result corresponding to the target byte and the first N bytes.

[0214] Optionally, the input data is data to be decompressed, and the input data includes multiple bytes and a second encoding result;

[0215] The processing module 804 is used to:

[0216] Decoding the second encoding result in the input data according to the probability estimation result corresponding to the target byte to obtain a decoding result;

[0217] According to the decoding result and the multiple bytes, a processing result corresponding to the input data is obtained.

[0218] Optionally, the vector module 802 is used to:

[0219] Obtain a clustering result of context bytes corresponding to the target byte, the clustering result including g groups of bytes, each group of bytes in the g groups of bytes including at least two bytes, and g is a positive integer;

[0220] Map each byte in the context bytes corresponding to the target byte through the mapping matrix to obtain a mapping result corresponding to each byte in the context bytes;

[0221] The multiple mapping results corresponding to each group of bytes are merged to obtain the feature vector corresponding to the corresponding target bytes.

[0222] Optionally, the mapping matrix and the neural network model are obtained after joint training based on training data, and the type of the training data is the same as the type of the input data.

[0223] Optionally, the apparatus 80 is applied to a computer device, and the computer device includes a first processor and a second processor;

[0224] An acquisition module 801 is configured to acquire input data through a first processor;

[0225] A vector module 802 is configured to obtain, through a second processor, a feature vector corresponding to each target byte in a plurality of bytes;

[0226] A probability estimation module 803 is configured to obtain, through a second processor, a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte;

[0227] The processing module 804 is configured to obtain a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte through the first processor.

[0228] Optionally, the input data is stored in the first processor;

[0229] The vector module 802 is used to:

[0230] For each target byte, updating a first-in-first-out (FIFO) storage queue of the second processor according to a difference byte transmitted by the first processor, where the difference byte is a byte increased by the context byte corresponding to the target byte relative to the context byte of the previous target byte of the target byte;

[0231] Obtaining, by the second processor, a context byte corresponding to the target byte from the FIFO storage queue;

[0232] A feature vector corresponding to the target byte is obtained by the second processor according to the context byte corresponding to the target byte.

[0233] Optionally, the probability estimation result corresponding to each target byte includes an upper probability limit value and a lower probability limit value corresponding to the corresponding target byte;

[0234] The processing module 804 is used to:

[0235] Obtaining, by the first processor, an upper probability limit value and a lower probability limit value corresponding to each target byte from the second processor;

[0236] The first processor obtains a processing result corresponding to the input data based on the probability upper bound value and the probability lower bound value corresponding to each target byte.

[0237] Optionally, the processing module 804 is configured to:

[0238] Obtaining, by the first processor, second information from the second processor, where the second information is used to indicate a fitting result obtained by fitting a probability estimation result corresponding to a target byte among the multiple bytes;

[0239] A processing result corresponding to the input data is obtained based on the second information through the first processor.

[0240] Optionally, the context bytes corresponding to the target byte include the first P bytes of the target byte among the multiple bytes, where P is an integer greater than 1.

[0241] Optionally, the neural network model includes K fully connected layers, where K is a positive integer.

[0242] Figure 9 shows a possible logical structure diagram of a computer device 90 provided in an embodiment of the present application. Computer device 90 is used to implement the functions of the computer device involved in any of the above embodiments. Computer device 90 includes: memory 901, processor 902, communication interface 903, and bus 904. Memory 901, processor 902, and communication interface 903 are interconnected via bus 904.

[0243] The memory 901 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 may store programs. When the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are configured to perform one or more of steps 201-204 of the above-described data processing method embodiment.

[0244] The processor 902 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component or any combination thereof, for executing relevant programs to implement the functions required to be executed by the acquisition module, vector module, probability estimation module and processing module in the data processing device in the above embodiment, or to execute one or more steps in steps 201-204 of the method embodiment of the present application. The steps of the method disclosed in conjunction with the embodiment of the present application can be executed by a compiler and an executor, wherein the compiler and the executor can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 901 , and the processor 902 reads the information in the memory 901 and executes one or more steps of steps 201 - 204 of the above-mentioned data processing method embodiment in combination with its hardware.

[0245] The communication interface 903 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the computer device 90 and other devices or a communication network.

[0246] Bus 904 provides a pathway for transmitting information between the various components of computer device 90 (e.g., memory 901, processor 902, and communication interface 903). Bus 904 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, control buses, and the like. For ease of illustration, FIG9 shows a single thick line, but this does not imply that there is only one bus or only one type of bus.

[0247] In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor of the device executes the computer-executable instructions, the device executes the steps executed by the processor in Figure 9 above.

[0248] In another embodiment of the present application, a computer program product is also provided, which includes computer execution instructions stored in a computer-readable storage medium; when the processor of the device executes the computer execution instructions, the device executes the steps performed by the processor in Figure 9 above.

[0249] In another embodiment of the present application, a chip system is provided, comprising a processor configured to implement the steps performed by the processor in FIG. 9 . In one possible design, the chip system may further comprise a memory configured to store program instructions and data necessary for data writing. The chip system may consist of a chip alone or may include a chip and other discrete components.

[0250] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0251] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0252] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0253] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0254] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0255] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0256] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire input data, where the input data is data to be compressed or data to be decompressed, and the input data includes a plurality of bytes; Obtaining a feature vector corresponding to each target byte in the plurality of bytes, wherein the feature vector is obtained based on a clustering result of context bytes corresponding to the corresponding target byte; Based on the neural network model and the feature vector corresponding to each target byte, a probability estimation result corresponding to each target byte is obtained; According to the probability estimation result corresponding to each target byte, a processing result corresponding to the input data is obtained, where the processing result is a compression result corresponding to the input data, or a decompression result corresponding to the input data.

2. The method according to claim 1, characterized in that The input data is data to be compressed; Obtaining a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte includes: Obtaining a first encoding result corresponding to each target byte according to a probability estimation result corresponding to the target byte; According to the first encoding result corresponding to the target byte, a processing result corresponding to the input data is obtained.

3. The method according to claim 1, characterized in that The input data is data to be decompressed, and the input data includes the multiple bytes and the second encoding result; Obtaining a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte includes: Decoding the second encoding result in the input data according to the probability estimation result corresponding to the target byte to obtain a decoding result; A processing result corresponding to the input data is obtained according to the decoding result and the multiple bytes.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining of a feature vector corresponding to each target byte in the plurality of bytes includes: Obtain a clustering result of context bytes corresponding to the target byte, the clustering result including g groups of bytes, each group of bytes in the g groups of bytes including at least two bytes, and g is a positive integer; Mapping each byte in the context bytes corresponding to the target byte through a mapping matrix to obtain a mapping result corresponding to each byte in the context bytes; The multiple mapping results corresponding to each group of bytes are merged to obtain the feature vector corresponding to the corresponding target bytes.

5. The method according to claim 4, characterized in that The mapping matrix and the neural network model are obtained after joint training based on training data, and the type of the training data is the same as the type of the input data.

6. The method according to any one of claims 1 to 5, characterized in that The method is applied to a computer device, wherein the computer device includes a first processor and a second processor; The obtaining of input data includes: acquiring input data through the first processor; The obtaining of a feature vector corresponding to each target byte in the plurality of bytes includes: Obtaining, by the second processor, a feature vector corresponding to each target byte in the plurality of bytes; The obtaining of a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte includes: Obtaining, by the second processor, a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte; Obtaining a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte includes: The first processor obtains a processing result corresponding to the input data according to the probability estimation result corresponding to each target byte.

7. The method according to claim 6, characterized in that The input data is stored in the first processor; Obtaining, by the second processor, a feature vector corresponding to each target byte in the plurality of bytes includes: For each target byte, updating a first-in-first-out (FIFO) storage queue of the second processor according to a difference byte transmitted by the first processor, wherein the difference byte is a byte increased by a context byte corresponding to the target byte relative to a context byte of a previous target byte of the target byte; Obtaining, by the second processor, a context byte corresponding to the target byte from the FIFO storage queue; The second processor obtains a feature vector corresponding to the target byte according to the context byte corresponding to the target byte.

8. The method according to claim 6 or 7, characterized in that The probability estimation result corresponding to each target byte includes the probability upper bound value and the probability lower bound value corresponding to the corresponding target byte; Obtaining, by the first processor, a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte, includes: Obtaining, by the first processor, an upper probability bound value and a lower probability bound value corresponding to each target byte from the second processor; The first processor obtains a processing result corresponding to the input data based on the probability upper bound value and the probability lower bound value corresponding to each target byte.

9. The method according to claim 6 or 7, characterized in that Obtaining, by the first processor, a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte, includes: Obtaining, by the first processor, first information from the second processor, where the first information includes probability estimation results corresponding to H bytes with the highest probabilities among all target bytes, where H is a positive integer; A processing result corresponding to the input data is obtained based on the first information through the first processor.

10. The method according to claim 6 or 7, characterized in that Obtaining, by the first processor, a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte, includes: obtaining, by the first processor, second information from the second processor, where the second information is used to indicate a fitting result obtained by fitting a probability estimation result corresponding to a target byte among the multiple bytes; Obtaining, by the first processor, a processing result corresponding to the input data based on the second information.

11. The method according to any one of claims 1 to 10, characterized in that The context bytes corresponding to the target byte include the first P bytes of the target byte among the multiple bytes, where P is an integer greater than 1.

12. The method according to any one of claims 1 to 11, characterized in that The neural network model includes K fully connected layers, where K is a positive integer.

13. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire input data, wherein the input data is data to be compressed or data to be decompressed, and the input data includes a plurality of bytes; A vector module, configured to obtain a feature vector corresponding to each target byte in the plurality of bytes, wherein the feature vector is obtained based on a clustering result of context bytes corresponding to the corresponding target byte; A probability estimation module is used to obtain a probability estimation result corresponding to each target byte based on a neural network model and a feature vector corresponding to each target byte; A processing module is used to obtain a processing result corresponding to the input data based on a probability estimation result corresponding to each target byte, wherein the processing result is a compression result corresponding to the input data, or a decompression result corresponding to the input data.

14. The device according to claim 13, characterized in that The input data is data to be compressed; The processing module is used for: Obtaining a first encoding result corresponding to each target byte according to a probability estimation result corresponding to the target byte; According to the first encoding result corresponding to the target byte, a processing result corresponding to the input data is obtained.

15. The device according to claim 13, characterized in that The input data is data to be decompressed, and the input data includes the multiple bytes and the second encoding result; The processing module is used for: Decoding the second encoding result in the input data according to the probability estimation result corresponding to the target byte to obtain a decoding result; A processing result corresponding to the input data is obtained according to the decoding result and the multiple bytes.

16. The device according to any one of claims 13 to 15, characterized in that The vector module is used to: Obtain a clustering result of context bytes corresponding to the target byte, the clustering result including g groups of bytes, each group of bytes in the g groups of bytes including at least two bytes, and g is a positive integer; Mapping each byte in the context bytes corresponding to the target byte through a mapping matrix to obtain a mapping result corresponding to each byte in the context bytes; The multiple mapping results corresponding to each group of bytes are merged to obtain the feature vector corresponding to the corresponding target bytes.

17. The device according to claim 16, characterized in that The mapping matrix and the neural network model are obtained after joint training based on training data, and the type of the training data is the same as the type of the input data.

18. The device according to any one of claims 13 to 17, characterized in that The apparatus is applied to a computer device, which includes a first processor and a second processor; The acquisition module is configured to acquire input data through the first processor; The vector module is configured to obtain, through the second processor, a feature vector corresponding to each target byte in the plurality of bytes; The probability estimation module is configured to obtain, through the second processor, a probability estimation result corresponding to each target byte based on the neural network model and the feature vector corresponding to each target byte; The processing module is used to obtain a processing result corresponding to the input data according to a probability estimation result corresponding to each target byte through the first processor.

19. The device according to claim 18, characterized in that The input data is stored in the first processor; The vector module is used to: For each target byte, updating a first-in-first-out (FIFO) storage queue of the second processor according to a difference byte transmitted by the first processor, wherein the difference byte is a byte increased by a context byte corresponding to the target byte relative to a context byte of a previous target byte of the target byte; Obtaining, by the second processor, a context byte corresponding to the target byte from the FIFO storage queue; The second processor obtains a feature vector corresponding to the target byte according to the context byte corresponding to the target byte.

20. The device according to claim 18 or 19, characterized in that The probability estimation result corresponding to each target byte includes the probability upper bound value and the probability lower bound value corresponding to the corresponding target byte; The processing module is used for: Obtaining, by the first processor, an upper probability bound value and a lower probability bound value corresponding to each target byte from the second processor; The first processor obtains a processing result corresponding to the input data based on the probability upper bound value and the probability lower bound value corresponding to each target byte.

21. The device according to claim 18 or 19, characterized in that The processing module is used for: Obtaining, by the first processor, first information from the second processor, where the first information includes probability estimation results corresponding to H bytes with the highest probabilities among all target bytes, where H is a positive integer; A processing result corresponding to the input data is obtained based on the first information through the first processor.

22. The device according to claim 18 or 19, characterized in that The processing module is used for: obtaining, by the first processor, second information from the second processor, where the second information is used to indicate a fitting result obtained by fitting a probability estimation result corresponding to a target byte among the multiple bytes; Obtaining, by the first processor, a processing result corresponding to the input data based on the second information.

23. The device according to any one of claims 13 to 22, characterized in that The context bytes corresponding to the target byte include the first P bytes of the target byte among the multiple bytes, where P is an integer greater than 1.

24. The device according to any one of claims 13 to 23, characterized in that The neural network model includes K fully connected layers, where K is a positive integer.

25. A computer device, characterized in that: The computer device includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method according to any one of claims 1 to 12.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Citation Information

Patent Citations

  • FPGA configuration file arithmetic compression and decompression method based on neural network model

    CN111431540A

  • Statistical probability-based data compression method and system

    CN112667633A

  • Data compression method and computing equipment

    CN113055017A

  • Automatic input method and device based on translation, equipment and computer storage medium

    CN113761945A

  • Text compression method and device, text decompression method and device, computer equipment and storage medium

    CN114048711A