Data compression processing method and related device

Abstract

The embodiment of the invention provides a data compression processing method and a related device. The method comprises the following steps: receiving initial data and a connection bitmap of a plurality of channels; preprocessing the initial data of each channel to obtain first data of each channel; inputting initial data of a plurality of channels and the connection bitmap into an intermediate storage area; according to the connection bitmap in the middle storage area, splicing the first data of the multiple channels to obtain spliced data; and performing zero compression processing on the spliced data to obtain a compressed data packet. According to the method and the device, multiple paths of data are integrated into the single data stream through data splicing, and zero compression processing is performed on the single data stream, so that storage space compression is realized. Data compression is realized through two links of data compression, data splicing and zero compression, and special constraints on input data types are avoided, so that the method has wider applicability, namely, high expansibility of data compression processing is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing. More specifically, the embodiments of the present application relate to a data compression processing method and related devices. Background Art

[0002] A sparse matrix is ​​a matrix in which most of the elements are zero. This type of matrix is ​​often used in engineering applications, especially in the field of artificial intelligence. For example, large sparse matrices are often used in machine learning fields such as natural language processing. Since there are a large number of redundant 0s in a sparse matrix, if it is represented and processed as a dense matrix, the computing resources are consumed a lot, resulting in a high computing cost. Therefore, it is necessary to preprocess the sparse matrix to reduce its computing cost and improve computing efficiency.

[0003] In the related art, the dictionary-based compression scheme replaces data words with corresponding short codes in the dictionary. By compressing frequently occurring data words, this scheme can achieve a higher compression rate. However, the scheme needs to rely on a dedicated dictionary. Due to the complexity of dictionary synchronization, the scalability of the scheme is poor and the delay is long.

[0004] In summary, there is an urgent need to provide a new technical solution to solve at least one of the above-mentioned technical problems existing in the relevant technologies. Summary of the invention

[0005] In this context, the embodiments of the present application hope to provide a data compression processing method and related devices to solve the problem of low scalability in related technologies and to achieve high scalability.

[0006] In a first aspect of the embodiments of the present application, a data compression processing method is provided, the method applying a multi-channel pipeline architecture in a sparse matrix compression scenario, the multi-channel pipeline architecture at least comprising a data splicing module and / or a zero compression module, the data splicing module and / or the zero compression module using a combinational logic circuit; the method comprising:

[0007] Receive initial data and a connection bitmap of multiple channels; wherein the connection bitmap is used to characterize the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same;

[0008] Preprocessing the initial data of each channel to obtain the first data of each channel;

[0009] Inputting the first data of the plurality of channels and the connection bitmap into an intermediate storage area;

[0010] splicing the first data of multiple channels by a data splicing module according to the connection bitmap in the intermediate storage area to obtain spliced ​​data;

[0011] The spliced ​​data is subjected to zero compression processing by a zero compression module to obtain a compressed data packet.

[0012] In a second aspect of the embodiments of the present application, a data compression processing device is provided, wherein the device applies a multi-channel pipeline architecture in a sparse matrix compression scenario, wherein the multi-channel pipeline architecture includes at least a data splicing module and / or a zero compression module, wherein the data splicing module and / or the zero compression module adopts a combinational logic circuit; the device includes at least:

[0013] A data receiving module, used for receiving initial data and a connection bitmap of multiple channels; wherein the connection bitmap is used for representing the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same;

[0014] A data preprocessing module, used for preprocessing the initial data of each channel to obtain the first data of each channel;

[0015] A data splicing module, configured to input the initial data of the plurality of channels and the connection bitmap into an intermediate storage area; and splice the first data of the plurality of channels to obtain spliced ​​data according to the connection bitmap in the intermediate storage area;

[0016] The zero compression module is used to perform zero compression processing on the spliced ​​data to obtain a compressed data packet.

[0017] In a third aspect of the embodiments of the present application, a computing device is provided, the computing device comprising:

[0018] at least one processor, memory, and input-output unit;

[0019] The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the data compression processing method of the first aspect.

[0020] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which includes instructions. When the instructions are executed on a computer, the computer executes the data compression processing method of the first aspect.

[0021] In a fifth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data compression processing method of the first aspect is implemented.

[0022] In a sixth aspect of the embodiments of the present application, a chip is provided, which includes a processor coupled to a transceiver and is used to execute the data compression processing method of the first aspect.

[0023] In the seventh aspect of the implementation of the present application, a chip system is provided, which includes: a communication interface for inputting and / or outputting information; a processor for executing a computer executable program so that a device equipped with the chip system performs a data compression processing method as in the first aspect.

[0024] In the implementation mode of the present application, a data compression processing method and related devices are provided. In the implementation mode of the present application, by receiving the initial data and connection bitmaps of multiple channels; pre-processing the initial data of each channel to obtain the first data of each channel; inputting the initial data of the multiple channels and the connection bitmap into the intermediate storage area; according to the connection bitmap in the intermediate storage area, splicing the first data of the multiple channels to obtain spliced ​​data; performing zero compression processing on the spliced ​​data to obtain a compressed data packet. Multiple channels of data are integrated into a single data stream through data splicing, and the single data stream is subjected to zero compression processing to achieve storage space compression. Moreover, data compression is achieved in the two links of data splicing and zero compression, and there are no special constraints on the input data type, so it has a wider applicability, that is, it achieves high scalability of data compression processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0026] Figure 1 A flowchart of a data compression processing method provided in one embodiment of the present application;

[0027] Figure 2 A schematic diagram of the implementation process of step S104 provided in an optional embodiment of the present application;

[0028] Figure 3 A schematic diagram of the implementation process of step S105 provided in an optional embodiment of the present application;

[0029] Figure 4 A schematic diagram of the architecture of a zero compression circuit provided in an optional embodiment of the present application;

[0030] Figure 5 A schematic diagram of a working process of a zero compression circuit provided in a specific example of the present application;

[0031] Figure 6 A schematic diagram of a working process of an output circuit provided in a specific example of the present application;

[0032] Figure 7 A schematic diagram of the working process of a data compression processing device provided in a specific example of the present application;

[0033] Figure 8 This is a schematic diagram of a 16-channel data compression example provided in a specific example of the present application;

[0034] Fig. 9 A schematic diagram of the structure of a data compression processing device provided in one embodiment of the present application;

[0035] Fig.10 A schematic diagram of the structure of a medium in an embodiment of the present application is schematically shown;

[0036] Fig.11 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown schematically.

[0037] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0038] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] A sparse matrix is ​​a matrix in which most of the elements are zero. This type of matrix is ​​often used in engineering applications, especially in the field of artificial intelligence. Large sparse matrices are common in many cases, especially in machine learning applications, such as coefficients of linear equations, data encoding that maps categories to counts, and even in machine learning subfields such as natural language processing. Due to the large number of redundant zeros in sparse matrices, the computational cost is very high if they are represented and processed as dense matrices. However, by using circuits that specialize in processing sparse matrices, the performance can be greatly improved. Therefore, sparse matrices usually need to pass through preprocessing compression circuits before calculation. Preprocessing compression circuits are dedicated hardware circuits used to compress sparse matrices. The main purpose of compression is to remove a large number of zero elements in the matrix and only retain non-zero elements and their position information, thereby greatly reducing storage space and computational complexity. Therefore, in sparse matrix calculations, preprocessing compression circuits are a very important module that can effectively reduce storage and computational overhead and improve the efficiency of the overall system.

[0041] At present, in the related technology, since the dictionary-based compression scheme replaces data words with corresponding short codes in the dictionary, this scheme requires dictionary synchronization, and the dictionary synchronization is relatively complicated. On the one hand, different data formats require synchronization of different dictionaries, resulting in poor scalability of this compression scheme; on the other hand, dictionary synchronization requires a long data transmission time, resulting in a long data compression processing time, resulting in delay overhead.

[0042] In the related art, since the pattern-based compression scheme encodes according to the frequency of occurrence of predefined patterns in data words, this scheme has limited ability to compress multiple data types, especially floating-point and character data, and is difficult to adapt to new data patterns, resulting in poor scalability of this compression scheme.

[0043] In summary, the zero data compression scheme in the related art has poor scalability due to the difficulty of dictionary synchronization migration, the inflexible data types that can be processed, and the complex compression process and the delay of data compression overhead.

[0044] In summary, there is an urgent need to provide a new technical solution to solve at least one of the above-mentioned technical problems existing in the relevant technologies.

[0045] In order to overcome the above technical problems existing in the existing solutions, according to the implementation of the present application, a data compression processing method and related devices are proposed.

[0046] In the implementation mode of the present application, first, by receiving the initial data and connection bitmap of multiple channels, the data bit width of the initial data of each channel is the same; the initial data of each channel is preprocessed to obtain the first data of each channel; the initial data of the multiple channels and the connection bitmap are input into the intermediate storage area; according to the connection bitmap in the intermediate storage area, the first data of the multiple channels are spliced ​​to obtain spliced ​​data through the data splicing module; the spliced ​​data is zero-compressed to obtain a compressed data packet. Multiple data are integrated into a single data stream through data splicing, and the single data stream is zero-compressed to achieve storage space compression. And the data compression is achieved in the two links of data splicing and zero compression, which does not have special constraints on the input data type, so it has a wider applicability, that is, it realizes the high scalability of data compression processing. For example, if the input data to be processed belongs to floating point type and / or character type data, then the constraint on the input data can be set to that the input data needs to have the same bit width. The role set here, as described below, is intended to use data with the same bit width to act on the logic or operation.

[0047] Secondly, through the combined use of splicing and zero compression, it is possible to support parallel work of data processing in two stages, thereby significantly improving data throughput and processing efficiency. After receiving the input data, it is even possible to complete zero data compression in three clock cycles, thereby significantly reducing latency overhead.

[0048] The technical solutions provided in the embodiments of the present application can be implemented by servers and / or terminal devices. It should be noted that the server involved in the embodiments of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, big data and artificial intelligence platforms.

[0049] The terminal device involved in the embodiments of the present application may be a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, may be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges voice and / or data with a wireless access network. For example, Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and other devices.

[0050] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0051] The principles and spirit of the present application are explained in detail below with reference to several representative implementations of the present application.

[0052] Reference below Figure 1 , Figure 1 A data compression processing method provided in an embodiment of the present application is a flowchart of a data compression processing method provided in an embodiment of the present application, comprising:

[0053] Step S101, receiving initial data and connection bitmaps of multiple channels.

[0054] Specifically, the connection bitmap is used to represent the data connection relationship between multiple channels. The connection bitmap contains multiple bits, and the number of bits is equal to the number of channels. Each bit represents whether the channel corresponding to this bit is connected to the channels corresponding to other bits. For example, the first bit corresponds to channel 1, the second bit corresponds to channel 2, the third bit corresponds to channel 3, and so on. The nth bit corresponds to channel n. The value of each bit is 0 or 1, where 0 represents that the channel data corresponding to this bit has no connection with the channels corresponding to other bits, that is, the channel data corresponding to this bit is all 0; 1 represents that the channel data corresponding to this bit is connected to the channels corresponding to other bits, that is, the channel data corresponding to this bit is not all 0.

[0055] Exemplarily, the values ​​of the first, fourth and seventh bits in the connection bitmap are 1, and the values ​​of the remaining bits are 0, indicating that the data of channel one, channel four and channel seven are not all 0, and the data of channel one, channel four and channel seven are related and need to be spliced ​​together.

[0056] It should be noted that the initial data of multiple channels come from the sparse matrix that needs to be compressed, which contains a large number of redundant 0s and needs to be compressed to remove a large number of zero elements.

[0057] Step S102, preprocessing the initial data of each channel to obtain first data of each channel.

[0058] It can be understood that the bit width of the initial data of each channel is the same. For example, the data bit width may be m, that is, the bit width of the initial data of each channel is the same, which is m bits.

[0059] In a specific embodiment, the implementation process of step S102 includes: performing a shift and zero padding process on the initial data of each channel so that the data bit width of the first data is aligned to N times the data bit width of the initial data, thereby realizing multi-channel data splicing. It should be noted that N is the number of channels of the multiple channels. By performing a pre-processing of shift and zero padding on the initial data, the initial data of each channel is expanded to a multiple of the number of channels of the original bit width, so as to realize the subsequent multi-channel data splicing.

[0060] In an exemplary embodiment of the present disclosure, the initial data of each channel is shifted and filled with zeros, including: the initial data of each channel is shifted left, and zeros are filled at the front end of the data after the left shift. It can be understood that for each channel, the initial data of each channel is shifted left according to the target number of bits corresponding to the preset rule in the front-stage module, and the left-shifted data is filled with zeros, wherein the left-shifted data is filled with zeros at the front end of the data. Exemplarily, the initial data of the nth channel is shifted left by (n-1)×m bits, where m is the data bit width of the initial data of each channel, and 0 is filled at the front end of the data after the left shift, so that the data bit widths of all channels are aligned to n×m bits.

[0061] Step S103: input the first data of the plurality of channels and the connection bitmap into an intermediate storage area.

[0062] In some embodiments of the present disclosure, initial data and connection bitmaps of multiple channels are temporarily stored in a first-in first-out queue FIFO.

[0063] It should be noted that FIFO (First-In, First-Out) is a common data structure or algorithm strategy. FIFO can ensure that the data that enters the queue first is processed first, ensuring that the processing order of all data is fair. And the implementation of FIFO is very simple and intuitive, and does not require complex logic to track the status of data items. In hardware implementation, FIFO can usually be implemented through simple circuit design, which is easy to understand and maintain. By using FIFO, data flow can be effectively managed to avoid the situation where new data overwrites old data. In particular, for data that needs to be processed in chronological order in the embodiment of the present disclosure, FIFO can help reduce unnecessary delays and maintain the order and consistency of data. In the case of needing to increase capacity or change performance, it is relatively easy to expand and adjust using the FIFO structure.

[0064] Accordingly, in the above embodiment, the specific implementation process of step S103 includes: in response to receiving the compression instruction signal, taking out the connection bitmap from the FIFO; based on the connection bitmap, extracting the first data of multiple channels from the FIFO for splicing to obtain spliced ​​data. Further optionally, the compression instruction signal at least includes a memory address to be written.

[0065] In some embodiments of the present disclosure, the data compression processing method provided further includes the following step after step S104: storing the compressed data packet into the memory through the credit interface. It should be noted that storing the compressed data packet into the memory through the credit interface, that is, sending data only when the memory can accept data, can effectively reduce storage overhead and reduce the traffic for accessing the memory.

[0066] The credit interface refers to a credit-based interface. The purpose of the credit-based interface is to ensure that the receiving end has sufficient capacity (buffer) to receive any data that has been sent by the sending end, so the interface needs to be set according to the size of the downstream receiving end buffer and the amount of data that the downstream can accept. See below. Here, a matching hardware unit needs to be set. The main components of this hardware unit are counters and FIFOs. See below for a specific example of the output circuit combination logic.

[0067] In an exemplary embodiment of the present disclosure, after the compressed data packet is obtained, the compressed data packet is stored in the FIFO to ensure that the generated compressed data packet can be stored in the memory through the credit interface according to the processing order.

[0068] Step S104: splicing the first data of the plurality of channels by a data splicing module according to the connection bitmap in the intermediate storage area to obtain spliced ​​data.

[0069] Since the connection bitmap represents the association relationship between data from different channels, the splicing relationship between multiple channels can be determined based on the connection bitmap. The spliced ​​data of a single data stream can be obtained by splicing the first data of multiple channels based on the splicing relationship, so as to achieve subsequent data zero compression.

[0070] Specifically, an optional embodiment of step S104 is shown in Figure 2 As shown, it can be implemented as follows:

[0071] Step S201: determining at least two channels to be spliced ​​among a plurality of channels based on a connection bitmap.

[0072] Step S202: splice the first data of at least two channels to be spliced ​​to obtain spliced ​​data.

[0073] In some embodiments of the present disclosure, when step S201 is specifically implemented, it includes: determining at least two bits in the connection bitmap whose values ​​are target values; it should be noted that the target value is 1; and obtaining at least two channels to be spliced ​​according to the at least two bits. Determine at least two bits in the connection bitmap whose values ​​are 1, each bit corresponds to a data channel, and then at least two channels to be spliced ​​can be determined. For example, the at least two bits in the connection bitmap whose values ​​are 1 are the seventh bit, the ninth bit, the eleventh bit, and the sixteenth bit, so the at least two channels to be spliced ​​are channel seven, channel nine, channel eleven, and channel sixteen.

[0074] In some embodiments of the present disclosure, when step S202 is specifically implemented, it includes: performing a logical OR operation on the first data of at least two channels to be spliced ​​to obtain spliced ​​data. When specifically implemented, the first data of at least two channels to be spliced ​​are extracted, and a logical OR operation is performed on them to splice the first data of at least two channels to be spliced ​​together. It can be understood that the data of other channels except the at least two channels to be spliced ​​are all 0, and the result obtained by performing a logical OR operation on the result spliced ​​together with the first data of at least two channels to be spliced ​​remains unchanged. Therefore, by selecting at least two channels to be spliced ​​in the early stage for data splicing, it is possible to obtain accurate spliced ​​data by reducing unnecessary data calculation processes.

[0075] It should be noted that the data bit width of the spliced ​​data is the same as the data bit width of the first data before splicing. Performing a logical OR operation can realize the splicing of n channel data, and the data bit width after splicing is still n×m bits.

[0076] It can be understood that the above data splicing process only retains the channel data corresponding to the value of 1 in the connection bitmap, and the remaining channel data is not retained, which is equivalent to deleting all the 0 data corresponding to the channel with a value of 1 in the connection bitmap to achieve the elimination of some 0 data.

[0077] Step S105, performing zero compression processing on the spliced ​​data to obtain a compressed data packet.

[0078] Specifically, an optional embodiment of step S105 is shown in Figure 3 As shown, it can be implemented as follows:

[0079] Step S301, deleting zero data from the spliced ​​data to obtain valid non-zero data and a corrected connection bitmap.

[0080] Step S302, counting the number and position information of valid non-zero data.

[0081] Step S303: determine the header information based on the number and position information of the valid non-zero data and the corrected connection bitmap.

[0082] Step S304: obtaining a compressed data packet based on the header information and the valid non-zero data.

[0083] It should be noted that, for the spliced ​​data, the zero compression circuit will delete the all-0 data segment in the spliced ​​data, and the remaining data is valid non-zero data, that is, data with a value of 1. In this process, the connection bitmap will also be corrected. For example, the channel data corresponding to the bit with a value of 1 in the connection bitmap should not be all-0 data, but when deleting the 0 data, it may be recognized that the value of one or some bits is 1, but the corresponding channel data is all-0 data. At this time, the corresponding bit value in the connection bitmap needs to be corrected to 0.

[0084] In the above zero compression process, the zero compression circuit will count and record the number of valid non-zero data, location information and the updated splicing bitmap, which will be stored as header information. The data packet after zero compression only contains two parts: header information and valid non-zero data.

[0085] As an alternative example, in Figure 4 In the zero compression circuit, the main components include comparators, counters, data selectors and other combinational logic circuits, as well as necessary registers and control logic. The zero compression circuit will traverse the spliced ​​data and only store the non-zero parts into new temporary variables.

[0086] It may be noted that, in some embodiments, Figure 4 The counter part shown in the figure can also be deleted, that is, the zero compression circuit does not include Figure 4 The counter portion is shown.

[0087] Exemplarily, the main components of the zero compression combinational logic circuit include: a comparator for determining whether the current data is valid non-zero data, a counter for counting the number of valid non-zero data in the current data, a data selector for selecting the valid non-zero data, and storing the selected valid non-zero data in an array.

[0088] Further optionally, if Figure 5 The schematic diagram of zero compression combinational logic circuit is shown in Figure 1. Figure 5 In the process, the data in the zero compression circuit is processed by the comparator and two counters to filter out some information in the header information, such as the number of valid non-zero data, location information, etc., and the compressed data is also obtained synchronously. That is, the data packet transmitted to the FIFO waiting for output, see Figure 6 The purpose of this step is to store non-zero values ​​(i.e., valid non-zero data) at the front of the array and 0 at the back of the array, thereby filtering out non-zero values.

[0089] In the embodiment of the present application, illustratively, the main components of the output combinational logic circuit are the counter and the FIFO. Figure 6 The output combinational logic circuit diagram is given in Figure 6 In the process, the compressed data is stored in FIFO. When the credit value (Credit) corresponding to a certain data in the counter is greater than 0, the corresponding data stored in FIFO is used as output data.

[0090] In the implementation mode of the present application, through the above steps S101 to S105, multiple data channels are integrated into a single data stream through data splicing, and the single data stream is subjected to zero compression processing to achieve storage space compression. Moreover, data compression is achieved in the two links of data splicing and zero compression, and there are no special constraints on the input data type, so it has a wider applicability, that is, it achieves high scalability of data compression processing. Through the combined use of splicing and zero compression, it is possible to support parallel work of data processing in two stages, thereby significantly improving the data throughput and processing efficiency, and even after receiving the input data, it is possible to start outputting compressed data packets in only three clock cycles, thereby significantly reducing delay overhead. By storing compressed data packets in memory through a credit interface, storage overhead can be effectively reduced and the flow of accessing memory can be reduced.

[0091] In order to better illustrate the data compression processing method provided by the embodiment of the present disclosure, a specific example is now provided for further explanation. This specific example is applied to the data compression processing method provided by the embodiment of the present disclosure to provide a data compression processing device. The working process of the data compression processing device is as follows: Figure 7 shown.

[0092] like Figure 8 The 16-channel data processed by the data compression processing device and the connection bitmap are shown. It can be seen from the figure that among the 16-channel data, the data of channels 1, 2, 3, 8, 14, and 15 are valid and can be spliced ​​together for zero compression.

[0093] like Figure 7 As shown, the specific working process of the above device includes: compressing the data of 16 channels, and the initial data bit width of each channel is the same as m bits. In order to realize multi-channel data splicing, the data of each channel will be shifted and zero-filled in the previous module to align the data bit width of all channels to 16×m bits.

[0094] After receiving the pre-processed data and connection bitmap, they are temporarily stored in the first-in first-out queue FIFO, waiting for a specific compression command signal to trigger subsequent processing. Once the compression command signal is received, the connection bitmap is taken out from the FIFO, and the corresponding data is extracted according to the channel corresponding to the bit 1 in the connection bitmap (that is, Data1, Data2, Data3, Data8, Data14, Data15), and a logical or operation is performed on the extracted data to realize the splicing of 16 channel data. After splicing, the data width is still 16×m bits.

[0095] Next, zero compression processing is performed. For the spliced ​​data, the all-0 data segment will be deleted, and the remaining data is valid non-zero data. In this process, the counter in the zero compression circuit will count and record the number of valid non-zero data, and the register in the zero compression circuit will record the location information of the valid non-zero data and the corrected connection bitmap, and the number of valid non-zero data, the location information of the valid non-zero data, and the corrected connection bitmap are stored as header information.

[0096] Finally, the data packet after zero compression contains only two parts: header information and valid non-zero data. The data packet will be stored in the memory through the credit interface, that is, data is sent only when the memory can accept data, effectively reducing storage overhead.

[0097] It can be seen from the above working process that the above device realizes the splicing and zero compression of multi-channel data, greatly reduces the space required for data storage, and improves the processing efficiency. The introduction of FIFO queue ensures the real-time and reliability of data processing. The data throughput is increased by pipeline design, that is, the data splicing module and the zero compression module are both implemented by pure combinational logic circuits, and their respective output results are cached by the subsequent register. The pipeline architecture supports the parallel operation of modules at all levels, thereby significantly improving the data throughput and processing efficiency. Compatibility with different data formats is achieved through splicing and zero compression, that is, this device achieves the goal of data compression through two links of data splicing and zero compression, and does not have special constraints on the input data type, so it has a wider range of applications. The data splicing module integrates multiple input data into a single data stream, while the zero compression module eliminates invalid data, thereby achieving the purpose of compressing the storage space. And by using a credit-based interface, the flow of accessing memory is reduced.

[0098] After introducing the method of the exemplary embodiment of the present application, next, refer to Fig. 9 A data compression processing device according to an exemplary embodiment of the present application is described. The device applies a multi-channel pipeline architecture in a sparse matrix compression scenario. The multi-channel pipeline architecture includes at least a data splicing module and / or a zero compression module. The data splicing module and / or the zero compression module use a combinational logic circuit. The device includes the following modules:

[0099] The data receiving module 701 is used to receive the initial data and connection bitmap of multiple channels; wherein the connection bitmap is used to characterize the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same;

[0100] A data preprocessing module 702 is used to preprocess the initial data of each channel to obtain the first data of each channel;

[0101] The data splicing module 703 is used to input the initial data of the multiple channels and the connection bitmap into the intermediate storage area; and splice the first data of the multiple channels to obtain spliced ​​data according to the connection bitmap in the intermediate storage area;

[0102] The zero compression module 704 is used to perform zero compression processing on the spliced ​​data to obtain a compressed data packet.

[0103] As an optional implementation, the data preprocessing module 702 is specifically used to:

[0104] The initial data of each channel is shifted and zero-filled to align the data bit width of the first data to N times the data bit width of the initial data; wherein N is the number of channels of the multiple channels.

[0105] As an optional implementation, the data preprocessing module 702 is specifically used for: for each channel, in the previous module, the initial data of each channel is left-shifted by the target number of bits corresponding to the preset rule, and the left-shifted data is padded with zeros, wherein the left-shifted data is padded with zeros at the front end of the data.

[0106] As an optional implementation, the data preprocessing module 702 shifts the initial data of each channel left by the target number of bits corresponding to a preset rule, and is configured to: shift the initial data of the nth channel left by (n-1)×m bits; wherein m is the data bit width of the initial data of each channel.

[0107] As an optional implementation, the data splicing module 703, according to the connection bitmap in the intermediate storage area, splices the first data of multiple channels through the data splicing module to obtain spliced ​​data, specifically for:

[0108] Based on the connection bitmap, determining at least two channels to be spliced ​​among the multiple channels;

[0109] The first data of at least two channels to be spliced ​​are spliced ​​to obtain spliced ​​data.

[0110] As an optional implementation, the data splicing module 703 is configured as follows:

[0111] Determine at least two bits in the connection bitmap whose values ​​are a target value; wherein the target value is one;

[0112] At least two channels to be spliced ​​are obtained according to the at least two bits.

[0113] As an optional implementation, the data splicing module 703 is configured as follows:

[0114] A logical OR operation is performed on the first data of at least two channels to be spliced ​​to obtain spliced ​​data; the data bit width of the spliced ​​data is the same as the data bit width of the first data before splicing.

[0115] As an optional implementation, if the number of the multiple channels is n and the data bit width of the first data of each channel is m, the data bit width of the spliced ​​data and the data bit width of the first data before splicing are both n×m bits.

[0116] As an optional implementation manner, the data splicing module 703 inputs the first data of the multiple channels and the connection bitmap into the intermediate storage area, specifically for:

[0117] The first data of the multiple channels and the connection bitmap are temporarily stored in a first-in first-out queue FIFO, and the FIFO supports multi-channel parallel extraction.

[0118] Correspondingly, the data splicing module 703, according to the connection bitmap in the intermediate storage area, splices the first data of multiple channels through the data splicing module to obtain spliced ​​data, which is specifically used for:

[0119] In response to receiving the compression instruction signal, taking out the connection bitmap from the FIFO;

[0120] Based on the connection bitmap, the first data of the plurality of channels are extracted from the FIFO and spliced ​​to obtain spliced ​​data. Further optionally, the compression instruction signal at least includes a memory address to be written.

[0121] As an optional implementation, the zero compression module 704 is specifically configured to:

[0122] Traversing the spliced ​​data through a zero compression module and deleting zero data in the spliced ​​data to obtain valid non-zero data and a corrected connection bitmap;

[0123] Count the number and location information of valid non-zero data;

[0124] Determine the header information based on the number and position information of the valid non-zero data and the corrected connection bitmap;

[0125] Get the compressed data packet based on the header information and valid non-zero data.

[0126] As an optional implementation, the combinational logic circuit of the zero compression module includes at least a comparator, a counter, and a data selector. Further, the comparator is used to determine whether the current data is the valid non-zero data, the counter is used to count the number of the valid non-zero data in the current data, and the data selector is used to select the valid non-zero data and store the selected valid non-zero data in an array.

[0127] As an optional implementation, the provided data compression processing device further includes: a data access module, used to: store the compressed data packet into the memory through the credit interface.

[0128] In the embodiment of the present application, a data compression processing device is used to integrate multiple data streams into a single data stream through data splicing, and the single data stream is subjected to zero compression processing to achieve storage space compression. Moreover, the two links of data splicing and zero compression achieve data compression without any special constraints on the input data type, so it has a wider applicability, that is, it achieves high scalability of data compression processing.

[0129] After introducing the devices, methods and apparatus of the exemplary embodiments of the present application, next, reference is made to Fig.10The computer-readable storage medium of the exemplary embodiment of the present application is described. The computer-readable storage medium can be set in the data compression processing device. Please refer to Fig.10 , the computer-readable storage medium shown is a CD 110, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, it will implement the steps recorded in the above method implementation, for example, receiving the initial data and connection bitmap of multiple channels; wherein the connection bitmap is used to characterize the data connection relationship between multiple channels, and the data bit width of the initial data of each channel is the same; pre-processing the initial data of each channel to obtain the first data of each channel; inputting the initial data of the multiple channels and the connection bitmap into the intermediate storage area; according to the connection bitmap in the intermediate storage area, splicing the first data of multiple channels through the data splicing module to obtain spliced ​​data; and performing zero compression processing on the spliced ​​data through the zero compression module to obtain a compressed data packet. The specific implementation method of each step will not be repeated here.

[0130] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0131] After introducing the devices, methods, media and apparatus of the exemplary embodiments of the present application, reference is now made to Fig.11 For the computing device for data compression processing according to the exemplary embodiment of the present application, the computing device may be arranged in a data compression processing device.

[0132] Fig.11 A block diagram of an exemplary computing device 120 suitable for implementing embodiments of the present application is shown. The computing device 120 may be a computer system or a server. Fig.11 The computing device 120 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Fig.11 As shown, the components of computing device 120 may include, but are not limited to: one or more processors or processing units 1201, a system memory 1202, and a bus 1203 connecting different system components (including system memory 1202 and processing unit 1201).

[0134] The computing device 120 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 120, including volatile and nonvolatile media, removable and non-removable media.

[0135] System memory 1202 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 12021 and / or cache memory 12022. Computing device 120 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 12023 may be used to read and write non-removable, non-volatile magnetic media ( Fig.11 is not shown in the Fig.11 As shown in FIG. 1 , a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to bus 1203 via one or more data medium interfaces. System memory 1202 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of each embodiment of the present application.

[0136] A program / utility 12025 having a set (at least one) of program modules 12024 may be stored, for example, in system memory 1202, and such program modules 12024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 12024 generally perform the functions and / or methods of the embodiments described herein.

[0137] The computing device 120 may also communicate with one or more external devices 1204 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 1205. In addition, the computing device 120 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1206. Fig.11 As shown, the network adapter 1206 communicates with other modules (such as the processing unit 1201, etc.) of the computing device 120 via the bus 1203. It should be understood that although Fig.11 Not shown, other hardware and / or software modules may be used in conjunction with computing device 120 .

[0138] The processing unit 1201 executes various functional applications and data processing by running the program stored in the system memory 1202, for example, receiving the initial data and connection bitmap of multiple channels; wherein the connection bitmap is used to characterize the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same; preprocessing the initial data of each channel to obtain the first data of each channel; inputting the initial data of the multiple channels and the connection bitmap into the intermediate storage area; according to the connection bitmap in the intermediate storage area, splicing the first data of the multiple channels through the data splicing module to obtain spliced ​​data; performing zero compression processing on the spliced ​​data through the zero compression module to obtain a compressed data packet. The specific implementation of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the data compression processing device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiment of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. On the contrary, the features and functions of one unit / module described above may be further divided into multiple units / modules to be embodied.

[0139] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above data compression processing method when executed by a processor.

[0140] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0146] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0147] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

Claims

1. A data compression processing method, It is characterized in that The method applies a multi-channel pipeline architecture in a sparse matrix compression scenario, wherein the multi-channel pipeline architecture at least includes a data splicing module and / or a zero compression module, wherein the data splicing module and / or the zero compression module adopts a combinational logic circuit; the method includes: Receive initial data and a connection bitmap of multiple channels; wherein the connection bitmap is used to characterize the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same; Preprocessing the initial data of each channel to obtain the first data of each channel; Inputting the first data of the plurality of channels and the connection bitmap into an intermediate storage area; splicing the first data of multiple channels by a data splicing module according to the connection bitmap in the intermediate storage area to obtain spliced ​​data; The spliced ​​data is subjected to zero compression processing by a zero compression module to obtain a compressed data packet.

2. The data compression processing method according to claim 1, It is characterized in that Preprocess the initial data of each channel to obtain the first data of each channel, including: Performing a shift and zero-padding process on the initial data of each channel so that the data bit width of the first data is aligned to N times the data bit width of the initial data; Wherein, N is the number of channels of the multiple channels.

3. The data compression processing method according to claim 2, It is characterized in that Perform shift and zero padding processing on the initial data of each channel, including: For each channel, the initial data of each channel is left-shifted according to the target number of bits corresponding to the preset rule in the front-end module, and the left-shifted data is padded with zeros, wherein the left-shifted data is padded with zeros at the front end of the data.

4. The data compression processing method according to claim 3, It is characterized in that Shift the initial data of each channel to the left by the target number of bits corresponding to the preset rules, including: The initial data of the nth channel is shifted left by (n-1)×m bits; wherein m is the data bit width of the initial data of each channel.

5. The data compression processing method according to claim 1, It is characterized in that According to the connection bitmap in the intermediate storage area, the first data of multiple channels are spliced ​​by a data splicing module to obtain spliced ​​data, including: Based on the connection bitmap, determining at least two channels to be spliced ​​among the multiple channels; The first data of the at least two channels to be spliced ​​are spliced ​​to obtain the spliced ​​data.

6. The data compression processing method according to claim 5, It is characterized in that Determining at least two channels to be spliced ​​among the multiple channels based on the connection bitmap includes: Determine at least two bits in the connection bitmap whose values ​​are target values; wherein the target value is one; The at least two channels to be spliced ​​are obtained according to the at least two bits.

7. The data compression processing method according to claim 6, It is characterized in that The step of splicing the first data of the at least two channels to be spliced ​​to obtain the spliced ​​data includes: A logical OR operation is performed on the first data of the at least two channels to be spliced ​​to obtain the spliced ​​data; the data bit width of the spliced ​​data is the same as the data bit width of the first data before splicing.

8. The data compression processing method according to claim 7, It is characterized in that If the number of the multiple channels is n, and the data bit width of the first data of each channel is m, then the data bit width of the spliced ​​data and the data bit width of the first data before splicing are both n×m bits.

9. The data compression processing method according to claim 1, It is characterized in that Inputting the first data of the plurality of channels and the connection bitmap into an intermediate storage area comprises: The first data of the multiple channels and the connection bitmap are temporarily stored in a first-in first-out queue FIFO, and the FIFO supports multi-channel parallel extraction.

10. A data compression processing device, It is characterized in that The device applies a multi-channel pipeline architecture in a sparse matrix compression scenario, wherein the multi-channel pipeline architecture at least includes a data splicing module and / or a zero compression module, and the data splicing module and / or the zero compression module adopt a combinational logic circuit; The device at least comprises: A data receiving module, used for receiving initial data and a connection bitmap of multiple channels; wherein the connection bitmap is used for representing the data connection relationship between the multiple channels, and the data bit width of the initial data of each channel is the same; A data preprocessing module, used for preprocessing the initial data of each channel to obtain the first data of each channel; A data splicing module, configured to input the initial data of the plurality of channels and the connection bitmap into an intermediate storage area; and splice the first data of the plurality of channels to obtain spliced ​​data according to the connection bitmap in the intermediate storage area; The zero compression module is used to perform zero compression processing on the spliced ​​data to obtain a compressed data packet.

Images