Data compression method and device
By performing probability statistics and interval allocation on convolutional neural network data, arithmetic coding is determined, which solves the problem of poor scalability of arithmetic coding and achieves efficient data compression and storage bandwidth utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-27
AI Technical Summary
Existing arithmetic coding techniques have poor scalability when compressing convolutional neural network data. They cannot effectively utilize the representation range of double-precision floating-point numbers, making them unusable directly and limiting the data compression effect.
By statistically analyzing the probability of character occurrences in the data to be compressed, interval allocation and encoding interval division are performed. Arithmetic codes are determined using the left and right boundary values of the encoding intervals. Segmented compression of data segments is achieved by removing the least significant binary digit bit bit by bit to meet preset conditions.
It achieves lossless compression while saving storage bandwidth, has good scalability, improves the efficiency and accuracy of data compression, and avoids the problem of encoding exceeding the range of double precision representation.
Smart Images
Figure CN121745176A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically to a data compression method and apparatus. Background Technology
[0002] Currently, with the rapid development of deep learning, convolutional neural networks require a large amount of storage bandwidth, necessitating compression techniques to improve bandwidth utilization.
[0003] The algorithm uses lossless compression and belongs to the variable code length (VCR) family of algorithms. Frequently occurring symbols have a wide frequency range, resulting in a large final encoding interval. This makes it easier to select a shorter encoding length within a larger range, saving space and improving bandwidth utilization. This lossless compression achieves a coding efficiency of 20%-50%.
[0004] However, during compression using the aforementioned algorithm, the interval shrinks exponentially. The minimum interval that double-precision floating-point numbers can accurately represent is finite. This means the amount of data with the same value cannot be too large; otherwise, the iterative region space easily exceeds the typical range of double-precision representation. For example, the sequence "AABABCABAB," with only 3 possible values (10 values), has an arithmetic code of 00101011001011, corresponding to the decimal number 0.16864013671875. If this corresponds to the 256 values of an int8 integer in a neural network, the traditional arithmetic encoding process exceeds the range of double-precision representation, making direct arithmetic encoding impossible. This significant limitation on data volume results in poor scalability of traditional arithmetic encoding. Summary of the Invention
[0005] Based on this, the present invention provides a data compression method and apparatus that achieves data compression with good scalability.
[0006] According to one aspect of the present invention, a data compression method is proposed, comprising: S1: statistically analyzing the occurrence probability of characters in the data to be compressed to obtain a probability table; S2: dividing the data to be compressed into multiple data segments according to a preset byte length; S3: selecting the current data segment to be processed from the multiple data segments; S4: allocating intervals to the current data segment to be processed according to the probability table to obtain an encoding interval; S5: determining the arithmetic code of the current data segment to be processed according to the left and right boundary values of the encoding interval, and using the arithmetic code of the current data segment to be processed as the compression result; S6: updating the current data segment to be processed according to the next data segment; S7: repeating steps S4-S6 until multiple data segments are traversed. According to some embodiments, step S4 includes: S41: taking a preset interval as the current target interval and extracting the first character of the current data segment to be processed as the current input; S42: according to the probability table, assigning the current target interval to the characters in the current data segment to be processed according to the corresponding probabilities of the characters in the current data segment to be processed, and obtaining the interval allocation result; S43: extracting the interval corresponding to the current input in the interval allocation result, and updating the current target interval using the extracted result; S44: updating the current input according to the next character of the current input; S45: repeating steps S42-S44 until the characters of the current data segment to be processed are traversed, and outputting the current target interval of the last iteration as the encoding interval.
[0007] According to some embodiments, step S4 further includes: outputting an error message if the current input does not belong to the probability table.
[0008] According to some embodiments, step S5 includes: converting the left and right boundary values of the encoding interval into binary format, and expanding the conversion result to a preset number of bits to obtain the target minimum value and the target maximum value; removing the least significant binary number of the target minimum value and / or the target maximum value bit by bit until a preset condition is met; using the target maximum value of the previous iteration that meets the preset condition as the arithmetic code of the current data segment to be processed, and using the arithmetic code of the current data segment to be processed as the compression result.
[0009] According to some embodiments, the least significant binary digits of the target minimum value and / or target maximum value are removed bit by bit until a preset condition is met, including: simultaneously removing the least significant binary digits of the target minimum value and target maximum value bit by bit until a preset condition is met, wherein the preset condition includes that the target maximum value after removing the least significant digits is less than the target minimum value after removing the least significant digits.
[0010] According to some embodiments, the least significant binary digit of the target minimum value and / or target maximum value is removed bit by bit until a preset condition is met, including: removing the least significant binary digit of the target maximum value bit by bit until a preset condition is met, wherein the preset condition includes the target maximum value after removing the least significant digit being less than the target minimum value.
[0011] According to some embodiments, step S1 includes: counting the number of times a character appears in the data to be compressed, as the weight of the character; calculating the probability of occurrence of the corresponding character based on the weight of the character in the data to be compressed; and forming a probability table based on the characters in the data to be compressed and their probability of occurrence.
[0012] According to one aspect of the present invention, a data compression apparatus includes: a character probability module for statistically analyzing the occurrence probability of characters in data to be compressed to obtain a probability table; a data segmentation module for segmenting the data to be compressed into multiple data segments according to a preset byte length; a current data module for selecting a current data segment to be processed from the multiple data segments; an interval allocation module for allocating intervals to the current data segment to be processed according to the probability table to obtain an encoding interval; a boundary encoding module for determining the arithmetic code of the current data segment to be processed according to the left and right boundary values of the encoding interval, and using the arithmetic code of the current data segment to be processed as the compression result; and an update iteration module for updating the current data segment to be processed according to the next data segment.
[0013] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described above.
[0014] According to one aspect of the invention, a computer-readable medium is provided on which a computer program or instructions are stored, which, when executed by a processor, implement the method described above.
[0015] Through the above embodiments provided by the present invention, the data segments after segmentation are allocated intervals by the probability table corresponding to the data to be compressed to obtain the encoding interval. The left and right boundary values of the obtained encoding interval are used to determine the arithmetic code of the corresponding data segment, thereby realizing the segmented compression of a data segment. All data segments are traversed to obtain the arithmetic code of all data segments. The precision of each code will not exceed the double precision representation range, thereby saving storage bandwidth while achieving lossless compression and having good scalability. Attached Figure Description
[0016] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention.
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without exceeding the scope of protection claimed by the present invention.
[0018] Figure 1 A flowchart of a data compression method provided in an embodiment of the present invention; Figure 2This is a flowchart provided by an embodiment of the present invention, which describes how to allocate intervals of the current data segment to be processed according to a probability table to obtain the encoding interval. Figure 3 The data compression method provided in this embodiment of the invention determines the arithmetic code of the current data segment to be processed based on the left and right boundary values of the encoding interval, and uses the arithmetic code of the current data segment to be processed as a flowchart of the compression result; Figure 4 A block diagram of a data compression apparatus provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.
[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0023] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the concept of the invention. As used herein, the term "and / or" includes any and all combinations of any one or more of the associated listed items.
[0024] For specific implementation details, please refer to the following examples.
[0025] Figure 1 A flowchart illustrating a data compression method provided in an embodiment of the present invention. Figure 1 As shown, the method includes steps S1-S7.
[0026] In step S1, the probability of occurrence of characters in the data to be compressed is calculated to obtain a probability table.
[0027] The data to be compressed can be any data that needs to be compressed, such as the weight data of a neural network to be compressed, or a string to be compressed. This invention does not impose any restrictions on this.
[0028] According to the example embodiment, the weight data of layer 1 of the neural network is extracted, totaling 4096 points. Each point is represented by an int8 (8-bit signed integer) as the data to be compressed.
[0029] The occurrence of each character (symbol) is statistically analyzed, and its probability of occurrence is calculated. Based on all the characters and their occurrence probabilities, a complete probability table is formed, denoted as tab(symbol, probability).
[0030] In step S2, the data to be compressed is divided into multiple data segments according to the preset byte length.
[0031] The data to be compressed is divided into segments according to a preset byte length, resulting in multiple data segments.
[0032] According to the example embodiment, the preset byte length is 4 bytes.
[0033] In step S3, the current data segment to be processed is selected from multiple data segments.
[0034] During the selection process, data segments can be selected according to the cutting order or other preset order; this invention does not limit this.
[0035] This invention does not limit the number of data segments selected; there can be one or more.
[0036] The selected data segment is denoted as the current data segment to be processed. The current data segment to be processed is used as the input for compression to perform encoding and compression.
[0037] In step S4, the current data segment to be processed is assigned intervals according to the probability table to obtain the encoding interval.
[0038] Based on the probability table, the current data segment to be processed is divided into intervals. Specifically, for each character in the current data segment, according to its probability in the probability table, the character is assigned to a fixed interval, with higher-frequency characters assigned larger intervals. This process determines the interval to which a character in the current data segment belongs. The intervals are then re-divided iteratively, gradually shrinking until the interval assigned to the last character becomes the encoding interval for the current data segment. This mapping uniquely represents the current data segment.
[0039] In step S5, the arithmetic code of the current data segment to be processed is determined based on the left and right boundary values of the encoding interval, and the arithmetic code of the current data segment to be processed is used as the compression result.
[0040] The encoded value is output based on the encoding interval. Specifically, the left and right boundary values of the encoding interval are processed and used as the arithmetic code of the current data segment to be processed. This code is the compression result of the current data segment to be processed.
[0041] In step S6, the current data segment to be processed is updated according to the next data segment to be processed.
[0042] Select the next data segment to be processed as the current data segment to be processed.
[0043] It is important to emphasize that "next" is defined based on the cutting order or other pre-defined order, and does not necessarily represent selection in sequence.
[0044] In step S7, steps S4-S6 are repeated until multiple data segments are traversed.
[0045] Each data segment formed after cutting is encoded using steps S4-S6 to obtain the compression result of each data segment.
[0046] Furthermore, the method also includes: Step S9: According to the positional order of multiple data segments in the data to be compressed, output the compression result corresponding to the data segments, that is, obtain the compression result of the data to be compressed.
[0047] This invention uses a probability table corresponding to the data to be compressed to allocate intervals to the segmented data segments, thereby obtaining encoding intervals. The left and right boundary values of the obtained encoding intervals are used to determine the arithmetic codes of the corresponding data segments, thus realizing segmented compression of a data segment. By traversing all data segments, the arithmetic codes of all data segments are obtained. The precision of each code will not exceed the double precision representation range, thereby saving storage bandwidth while achieving lossless compression and having good scalability.
[0048] According to some embodiments, refer to Figure 2 In step S4, the current data segment to be processed is allocated according to the probability table to obtain the encoding interval, which can be achieved through steps S41-S45.
[0049] In step S41, the preset interval is used as the current target interval, and the first character of the current data segment to be processed is extracted as the current input.
[0050] A range is set as the basis for interval division, denoted as the preset range. According to the example embodiment, the preset range is [0, 1).
[0051] The preset interval is used as the current target interval for the first iteration. The first character of the current data segment to be processed is extracted and used as the current input for subsequent interval division.
[0052] In step S42, according to the probability table, the current target interval is assigned to the characters in the current data segment according to the corresponding probabilities of the characters in the current data segment to be processed, and the interval assignment result is obtained.
[0053] Based on the probability table, the current target interval is divided into intervals, and each character is assigned one interval range (the interval size is the probability value). The preset interval is filled, and the character and its corresponding interval are used as the interval allocation result.
[0054] In step S43, the interval corresponding to the current input in the interval allocation result is extracted, and the current target interval is updated using the extracted result.
[0055] From the interval allocation results, extract the interval containing the character corresponding to the current input, and use it as the new current target interval.
[0056] In step S44, the current input is updated based on the next character of the current input.
[0057] The next character of the current input is used as the current input to iterate through new interval divisions based on the new current target interval.
[0058] In step S45, steps S42-S44 are repeated until the characters of the current data segment to be processed are traversed, and the current target interval of the last iteration is output as the encoding interval.
[0059] In each iteration, a new current target interval is assigned based on the probability table, and the interval corresponding to the current input is selected as the new current target interval. Steps S42-S44 are repeated to iterate and search for the interval corresponding to the character of the current data segment to be processed, until the last character of the current data segment to be processed has been searched. Then, the interval range is output and recorded as the encoding interval.
[0060] To provide a more detailed explanation of steps S41-S45, the following embodiments are given.
[0061] In this embodiment, the preset interval is [0, 1), the current data segment to be processed is CAACAB, and the characters in the probability table are sorted by probability as follows: A: 50%; C: 33.3%; B: 16.7%.
[0062] In step S41, [0, 1) is taken as the current target interval, and the first character C is extracted as the current input.
[0063] In step S42, the interval allocation result is as follows, filling the 0-1 space: A: [0, 0.5), C: [0.5, 0.833), B: [0.833, 1).
[0064] In step S43, the current input is C, which falls within the interval [0.5, 0.833). In other words, [0.5, 0.833) is the new current target interval.
[0065] In step S44, the second character A is taken as the new current input.
[0066] In step S45, steps S42-S44 are repeated.
[0067] The second iteration is as follows: according to the proportion of the probability table, the interval [0.5, 0.833) is re-divided, and the interval allocation results are: A: [0.5, 0.665), C: [0.665, 0.77588), B: [0.77588, 0.833).
[0068] If the current input is A, then the value falls within the interval [0.5, 0.665). In other words, [0.5, 0.665) is the new current target interval.
[0069] For each input character in the current data segment to be processed, repeat the above steps to divide it into new intervals and the final value interval, until all characters in the current data segment to be processed are traversed. Output the current target interval of the last iteration, which is the encoding interval.
[0070] In this embodiment of the invention, the segmented data segments are allocated into intervals to obtain an encoding interval that can uniquely represent the current data segment to be processed. Simultaneously, since the storage bandwidth occupied by the segmented data segments is a set value, the number of characters included is limited, resulting in high compression efficiency, and the precision will not exceed the range of double-precision representation.
[0071] According to some embodiments, in step S4, the current data segment to be processed is allocated into intervals according to the probability table to obtain the encoding interval, and the step S46 is also included.
[0072] In step S46, if the current input does not belong to the probability table, an error message is output.
[0073] Since the probability table is built based on all the characters in the data to be compressed, if the currently input character does not exist in the probability table, it means that the input character is incorrect, and an error message is output to remind the user that the encoding is incorrect.
[0074] This invention does not limit the specific content of the error message; it may include "error," or it may provide specific incorrect current input and other suggestions.
[0075] This invention provides a response when the current input cannot be found in the probability table, further refining the steps of interval allocation.
[0076] According to some embodiments, refer to Figure 3 In step S5, the arithmetic code of the current data segment to be processed is determined according to the left and right boundary values of the encoding interval, and the arithmetic code of the current data segment to be processed is used as the compression result. This can be achieved through steps S310-S330.
[0077] In step S310, the left and right boundary values of the encoding interval are converted into binary format, and the conversion result is extended to a preset number of bits to obtain the target minimum value and the target maximum value.
[0078] The maximum value (right boundary) and minimum value (left boundary) of the encoding interval are converted into binary values, high (right boundary) and low (left boundary).
[0079] The maximum value (right boundary) and minimum value (left boundary) of the encoding interval are converted into binary values and then expanded to a preset number of bits.
[0080] According to the example embodiment, the preset bit length is 64 bits.
[0081] The result after High expansion is denoted as the target maximum value, and the result after Low expansion is denoted as the target minimum value.
[0082] In step S320, the least significant binary digit of the target minimum value and / or target maximum value is removed bit by bit until the preset condition is met.
[0083] Remove the least significant 1 bit of the target minimum and / or target maximum values, then compare the target minimum and target maximum values. If the preset conditions are not met, continue to step S320; if they are met, jump to step S330.
[0084] In step S330, the target maximum value of the previous iteration that satisfies the preset conditions is used as the arithmetic code of the current data segment to be processed, and the arithmetic code of the current data segment to be processed is used as the compression result.
[0085] The maximum value of the target value in the previous iteration that satisfies the preset conditions is selected as the shortest encoded value of the interval, denoted as the arithmetic code. This arithmetic code is the compression result of the current data segment to be processed.
[0086] This invention improves compression efficiency by performing binary conversion and bit extension on the left and right boundaries of the encoding interval, then comparing the data corresponding to the left and right boundaries, dynamically removing the least significant bit to obtain the corresponding value of the right boundary as an arithmetic code. This minimizes the number of bits in the arithmetic code while ensuring accuracy.
[0087] According to some embodiments, in step S320, the least significant binary digit of the target minimum value and / or target maximum value is removed bit by bit until a preset condition is met, which can be specifically implemented through step S321.
[0088] In step S321, the least significant binary bits of the target minimum and target maximum values are removed bit by bit until a preset condition is met. The preset condition includes that the target maximum value after removing the least significant bit is less than the target minimum value after removing the least significant bit.
[0089] Specifically, the least significant binary bits of the target minimum and target maximum values are removed bit by bit until a preset condition is met in order to find the shortest code for the region.
[0090] The preset conditions include that the target maximum value after removing the least significant bit is less than the target minimum value after removing the least significant bit.
[0091] According to the example embodiment, the encoding range is (low, high); Minimum target value low_v ='b110011011; The target maximum value is high_v = 'b110011101'; Let the current iteration output value ref_v = high_v; Preset conditions: Ref_v <low_v。
[0092] Simultaneously, remove the least significant binary digits of both the target minimum and target maximum values bit by bit, that is: First iteration: simultaneously delete the rightmost bit of both low_v and ref_v, resulting in: low_v_1='b11001101; ref_v_1='b11001110; At this point, Ref_v_1 > low_v_1, which does not meet the preset conditions, so deletion continues.
[0093] Second iteration: simultaneously delete the rightmost bit of both low_v_1 and ref_v_1, resulting in: low_v_2='b1100110; ref_v_2='b1100111; At this point, Ref_v_2 > low_v_2, which does not meet the preset conditions, so deletion continues.
[0094] Third iteration: simultaneously delete the rightmost bit of both low_v_2 and Ref_v_2, resulting in: low_v_3='b110011; ref_v_3='b110011; Ref_v_3 = low_v_3, which satisfies the preset condition, so the final output value ref_v = Ref_v_2, and the final encoding is 'b1100111'.
[0095] The embodiments of the present invention simultaneously reduce the number of bits of the binary numbers to be compared bit by bit, which helps to control computational complexity; at the same time, during the comparison process, the lower bits that have been compared are released in a timely manner, saving computational memory.
[0096] According to some embodiments, in step S320, the least significant binary digit of the target minimum value and / or target maximum value is removed bit by bit until a preset condition is met, which can be specifically implemented through step S322.
[0097] In step S322, the least significant bit of the target maximum value is removed bit by bit until a preset condition is met. The preset condition includes that the target maximum value after removing the least significant bit is less than the target minimum value.
[0098] Specifically, the least significant bit of the target maximum value is removed bit by bit until a preset condition is met to find the shortest encoding for the region. The preset condition includes that the target maximum value after removing the least significant bit is less than the target minimum value.
[0099] According to the exemplary embodiment, the coding interval is (low, high); The target minimum value low_v = ’b110011011; The target maximum value high_v = ’b110011101; Let the current iteration output value ref_v = high_v; Preset condition: Ref_v < low_v.
[0100] At the same time, the least significant bits of the target minimum value and the target maximum value are removed bit by bit, that is: First iteration: Delete the rightmost bit[0] of the output value ref_v, and fill with 0 for alignment (which can be regarded as changing 1 to 0), to get: low_v = ’b110011011; ref_v_1 = ’b110011100; At this time, Ref_v_1 > low_v, which does not meet the preset condition, so continue to delete.
[0101] Second iteration: Delete the rightmost bit[1] of ref_v_1, and fill with 0 for alignment (changing 1 to 0), to get: low_v = ’b110011011; ref_v_2 = ’b110011100; At this time, Ref_v_2 > low_v, which does not meet the preset condition, so continue to delete.
[0102] Third iteration: Delete the rightmost bit[2] of Ref_v_2, and fill with 0 for alignment (changing 1 to 0), to get: low_v = ’b110011011; ref_v_3 = ’b110011000; At this time, Ref_v_3 < low_v, which meets the preset condition, so the final output value ref_v = Ref_v_2.
[0103] It should be emphasized that in order to save memory, at this time, all the 0s on the right side of Ref_v_2 need to be deleted. The final encoding is ’b1100111.
[0104] This invention embodiment only removes the least significant bit from the target maximum value, saving computational memory and reducing computational overhead, finding the shortest encoding bit range with minimal computational overhead.
[0105] According to some embodiments, in step S1, the probability of occurrence of characters in the data to be compressed is calculated to obtain a probability table, which can be specifically implemented through steps S11-S13.
[0106] In step S11, the number of occurrences of each character in the data to be compressed is counted and used as the weight of the character.
[0107] The occurrence count of characters in the compressed data is used to determine the weight of each character based on its occurrence count.
[0108] In step S12, the probability of occurrence of the corresponding character is calculated based on the weight of the character in the data to be compressed.
[0109] The weight of a character is calculated as a proportion of the sum of the weights of all characters, and this proportion is used as the probability of the corresponding character appearing.
[0110] That is, .
[0111] In step S13, a probability table is formed based on the characters in the data to be compressed and their occurrence probabilities.
[0112] The probability table records all the characters in the data to be compressed, as well as the probability of each character appearing.
[0113] In this embodiment of the invention, a probability table is obtained based on the frequency of occurrence of characters in the data to be compressed, which serves as the basis for arithmetic coding.
[0114] According to the example embodiment, the data compression method provided by the present invention can be used to compress the weight data of a neural network, thereby improving the bandwidth utilization of the neural network by 20%-50% and reducing the storage space requirements.
[0115] The following describes an embodiment of the apparatus of the present invention, which can be used to perform the embodiment of the method of the present invention. For details not disclosed in the embodiment of the apparatus of the present invention, please refer to the embodiment of the method of the present invention.
[0116] Figure 4 A block diagram of a data compression apparatus according to an exemplary embodiment is shown.
[0117] Figure 4 The apparatus shown can perform the data compression method described above according to embodiments of the present invention.
[0118] like Figure 4 As shown, the data compression device may include: See Figure 4 Referring to the preceding description, the character probability module 410 is used to count the probability of characters appearing in the data to be compressed, and obtain a probability table.
[0119] The data segmentation module 420 is used to segment the data to be compressed according to the preset byte length to obtain multiple data segments.
[0120] The current data module 430 is used to select the current data segment to be processed from multiple data segments.
[0121] The interval allocation module 440 is used to allocate intervals to the current data segment to be processed according to the probability table to obtain the encoding interval.
[0122] The boundary encoding module 450 is used to determine the arithmetic code of the current data segment to be processed based on the left and right boundary values of the encoding interval, and use the arithmetic code of the current data segment to be processed as the compression result.
[0123] The update iteration module 460 is used to update the current data segment to be processed based on the next data segment to be processed.
[0124] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.
[0125] This invention discloses an electronic device, including: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor executes the above-described instruction generation method.
[0126] For example, refer to Figure 5 , Figure 5 The illustrated electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of the present invention.
[0127] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this disclosure. Processor 501 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0128] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0129] The memory 503 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0130] The memory 503 is used to store application code that executes the present invention, and its execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the content shown in the foregoing method embodiments.
[0131] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0132] This invention discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.
[0133] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0134] The above are only some embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A data compression method, characterized in that, include: S1: Calculate the probability of occurrence of characters in the data to be compressed to obtain a probability table; S2: The data to be compressed is divided into multiple data segments according to the preset byte length; S3: Select the current data segment to be processed from the plurality of data segments; S4: According to the probability table, the current data segment to be processed is allocated into intervals to obtain the encoding interval; S5: Determine the arithmetic code of the current data segment to be processed based on the left and right boundary values of the encoding interval, and use the arithmetic code of the current data segment to be processed as the compression result; S6: Update the current data segment to be processed according to the next data segment; S7: Repeat steps S4-S6 until all data segments have been traversed.
2. The method according to claim 1, characterized in that, Step S4 includes: S41: Take the preset interval as the current target interval, and extract the first character of the current data segment to be processed as the current input; S42: According to the probability table, the current target interval is assigned to the characters in the current data segment according to their corresponding probabilities, to obtain the interval assignment result; S43: Extract the interval corresponding to the current input in the interval allocation result, and update the current target interval using the extracted result; S44: Update the current input based on the next character of the current input; S45: Repeat steps S42-S44 until the characters of the current data segment to be processed are traversed, and output the current target interval of the last iteration as the encoding interval.
3. The method according to claim 2, characterized in that, Step S4 also includes: If the current input does not belong to the probability table, an error message will be output.
4. The method according to claim 1, characterized in that, Step S5 includes: The left and right boundary values of the encoding interval are converted into binary format, and the conversion result is extended to a preset number of bits to obtain the target minimum value and the target maximum value; Remove the least significant binary digit of the target minimum value and / or target maximum value bit by bit until the preset condition is met; The target maximum value of the previous iteration that satisfies the preset conditions is used as the arithmetic code of the current data segment to be processed, and the arithmetic code of the current data segment to be processed is used as the compression result.
5. The method according to claim 4, characterized in that, Remove the least significant bit of the target minimum and / or target maximum value bit by bit until a preset condition is met, including: Simultaneously, the least significant binary digits of the target minimum and the target maximum are removed bit by bit until the preset condition is met. The preset condition includes that the target maximum after removing the least significant digit is less than the target minimum after removing the least significant digit.
6. The method according to claim 4, characterized in that, Remove the least significant bit of the target minimum and / or target maximum value bit by bit until a preset condition is met, including: Remove the least significant bit of the target maximum value bit by bit until the preset condition is met, wherein the preset condition includes the target maximum value after removing the least significant bit being less than the target minimum value.
7. The method according to claim 1, characterized in that, Step S1 includes: The frequency of occurrence of each character in the data to be compressed is counted and used as the weight of that character. The probability of occurrence of a corresponding character is calculated based on the weight of the character in the data to be compressed. The probability table is constructed based on the characters in the data to be compressed and their probability of occurrence.
8. A data compression device, characterized in that, include: The character probability module is used to calculate the probability of occurrence of characters in the data to be compressed, and to obtain a probability table. The data segmentation module is used to segment the data to be compressed into multiple data segments according to a preset byte length; The current data module is used to select the current data segment to be processed from the plurality of data segments; The interval allocation module is used to allocate intervals to the current data segment to be processed according to the probability table to obtain the encoding interval; The boundary encoding module is used to determine the arithmetic code of the current data segment to be processed based on the left and right boundary values of the encoding interval, and to use the arithmetic code of the current data segment to be processed as the compression result. The update iteration module is used to update the current data segment to be processed based on the next data segment.
9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-7.