A hybrid data dynamic adjustment mini-buffer encoding method and device

Abstract

The application relates to the field of data stream coding algorithms, and particularly provides a mixed data dynamic adjustment mini-buffer coding method and device, which is based on a data stream, a Simplestream compression algorithm is reformed on simple8b, selector 0 and selector 1 of the simple8b are redefined, the minimum unit of the Simplestream compression algorithm is 8 bytes, and the 8 bytes are divided into a control block, a data block and a byte block; the control block is defined by the selector 0; the data block is defined by redefining the selector 1 on the basis of the original simple8b; the byte block is not coded and adopts original binary code. Compared with the prior art, the application can efficiently compress data, consume very small resources, reduce bandwidth demand and improve transmission efficiency.

Description

Technical Field

[0001] This invention relates to the field of data stream coding algorithms, and specifically provides a method and apparatus for dynamically adjusting mini-buffer coding of mixed data. Background Technology

[0002] Data encoding is widely used in data transmission. In big data environments, with the continuous growth of data transmission volume, there is an urgent need to introduce an efficient and simple encoding algorithm to reduce bandwidth requirements and improve transmission efficiency. Summary of the Invention

[0003] This invention addresses the shortcomings of the prior art by providing a highly practical method for dynamically adjusting mini-buffer encoding of mixed data.

[0004] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable hybrid data dynamic adjustment mini buffer encoding device.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A hybrid data dynamic adjustment mini-buffer encoding method is proposed. Based on data stream, the Simplestream compression algorithm modifies simple8b by redefining selector 0 and selector 1 of simple8b.

[0007] The Simplestream compression algorithm has a minimum unit of 8 bytes, which is divided into control blocks, data blocks, and byte blocks.

[0008] The control block is defined by selector 0; the data block is based on the original simple8b, with selector 0 removed and selector 1 redefined; the byte block is not encoded and uses native binary code.

[0009] Furthermore, selector 1 is redefined in the data block, Bits 0-3 selector 1;

[0010] Simplestream redefines the meaning of selector 1 and supports run-length encoding;

[0011] Bit32-63 is a 32-bit integer, representing a value;

[0012] Bit 4-31 is a 28-bit integer representing the number of times the value is repeated.

[0013] Furthermore, selector 0 is a new control block, Bits 0-3 selector 0;

[0014] Simplestream redefines the original selector 0 of simple8b as the header of a control block and gives new meaning to the remaining 60 bits;

[0015] Bits 4-7 are control flags, which consist of four groups: session flag, integer flag, floating-point flag, and byte stream flag. Each group has three values, representing start, end, and in progress, respectively.

[0016] Bits 8-63 are further defined according to different control flags.

[0017] Furthermore, the control flags Bits 4-7 occupy 4 bits, and there are 16 flags in total, numbered 0-15. The flag definitions are as follows:

[0018] 0. reserved;

[0019] 1. Session begin: The session begins.

[0020] 2. Session end: The session ends.

[0021] 3. Session continued;

[0022] 4. integerparagraphbegin: Integer transmission begins;

[0023] 5. Integer paragraph end, integer transmission ends;

[0024] 6. The integer paragraph continues; integer transfer is in progress.

[0025] 7. `float paragraphbegin`: Floating-point number transmission begins;

[0026] 8. float paragraph end: Floating-point number transmission ends;

[0027] 9. Float paragraph continued; floating-point number transfer in progress.

[0028] 10. bytesparagraphbegin: byte stream transmission begins;

[0029] 11. bytes paragraph end, the byte stream transmission ends;

[0030] 12. Bytes paragraph continued; Byte stream transmission in progress.

[0031] 13-15, reserved.

[0032] Furthermore, control flags 1-3 of the session flag group;

[0033] Bits 8-31, reserved; Bits 32-63, original data size.

[0034] Furthermore, control flags 4-6 of the integer flag group;

[0035] Bits 8-10, data type indicator, started from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types;

[0036] Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used.

[0037] Bit 11, delta;

[0038] Bit 12, delta over delta;

[0039] Bit 13, zigzag;

[0040] Bits 14-19 are reserved.

[0041] Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted by using the greatest common divisor as the divisor;

[0042] Bits 32-35: The number of consecutive zeros starting from the units digit;

[0043] Bits 36-63, the number of integers, are only meaningful in the integer transfer start control block.

[0044] Furthermore, control flags 7-9 of the floating-point flag group;

[0045] Bits 8-10, data type indicator, started from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types, only meaningful when floating-point numbers are treated as integers;

[0046] Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used.

[0047] Bit 11, delta;

[0048] Bit 12, delta over delta;

[0049] Bit 13, zigzag;

[0050] Bit 14, XOR;

[0051] Bits 15-19, reserved;

[0052] Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted by using the greatest common divisor as the divisor;

[0053] Bits 32-35, the number of decimal places;

[0054] Bits 36-63, the number of decimals, are only meaningful in the integer transfer start control block.

[0055] Furthermore, control flags 10-12 of the byte stream flag group;

[0056] Bits 8-31 are reserved.

[0057] Bits 32-63, the number of bytes to be transmitted.

[0058] Furthermore, the size of the mini-buffer is calculated as count(data) x size(data type).

[0059] count is a constant N, with a minimum of 60 and a maximum of 1200;

[0060] The initial N numbers are stored in a mini-buffer. After the N numbers are evaluated, the encoding algorithm is determined, which is called the starting algorithm. The mini-buffer is a circular buffer that is used cyclically.

[0061] The data stream enters a mini-buffer, and encoding is triggered every N numbers.

[0062] A hybrid data dynamic adjustment mini buffer encoding device includes: at least one memory and at least one processor;

[0063] The at least one memory is used to store a machine-readable program;

[0064] The at least one processor is configured to invoke the machine-readable program to execute a hybrid data dynamic adjustment mini-buffer encoding method.

[0065] Compared with the prior art, the hybrid data dynamic adjustment mini-buffer coding method and apparatus of the present invention have the following outstanding advantages:

[0066] This invention compresses data efficiently while consuming minimal resources, thereby reducing bandwidth requirements and improving transmission efficiency. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Appendix Figure 1 This is a schematic diagram of simple8b in a hybrid data dynamic adjustment mini-buffer coding method.

[0069] Appendix Figure 2 This is a schematic diagram of the selector 0 control block in a hybrid data dynamic adjustment mini-buffer encoding method.

[0070] Appendix Figure 3 This is a schematic diagram of control flags Bits 4-7 in a hybrid data dynamic adjustment mini-buffer encoding method. Detailed Implementation

[0071] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The following is a preferred embodiment:

[0073] like Figure 1 As shown in this embodiment, a hybrid data dynamic adjustment mini-buffer encoding method is based on data stream. The Simplestream compression algorithm modifies simple8b by redefining selector 0 and selector1 of simple8b.

[0074] The Simplestream compression algorithm has a minimum unit of 8 bytes, which is divided into control blocks, data blocks, and byte blocks.

[0075] The control block is defined by selector 0; the data block is based on the original simple8b, with selector 0 removed and selector 1 redefined; the byte block is not encoded and uses native binary code.

[0076] Selector 1 is redefined in the data block, Bits 0-3 selector 1;

[0077] Simplestream redefines the meaning of selector 1 and supports run-length encoding;

[0078] Bit32-63 is a 32-bit integer, representing a value;

[0079] Bit 4-31 is a 28-bit integer representing the number of times the value is repeated.

[0080] like Figure 2 As shown, selector 0 is the new control block, bits 0-3 selector 0;

[0081] Simplestream redefines the original selector 0 of simple8b as the header of a control block and gives new meaning to the remaining 60 bits;

[0082] Bits 4-7 are control flags, which consist of four groups: session flag, integer flag, floating-point flag, and byte stream flag. Each group has three values, representing start, end, and in progress, respectively.

[0083] Bits 8-63 are further defined according to different control flags.

[0084] like Figure 3 As shown, control flags bits 4-7 occupy 4 bits each. There are 16 flags in total, numbered 0-15, and their definitions are as follows:

[0085] 0. reserved;

[0086] 1. Session begin: The session begins.

[0087] 2. Session end: The session ends.

[0088] 3. Session continued;

[0089] 4. integerparagraphbegin: Integer transmission begins;

[0090] 5. Integer paragraph end, integer transmission ends;

[0091] 6. The integer paragraph continues; integer transfer is in progress.

[0092] 7. `float paragraphbegin`: Floating-point number transmission begins;

[0093] 8. float paragraph end: Floating-point number transmission ends;

[0094] 9. Float paragraph continued; floating-point number transfer in progress.

[0095] 10. bytesparagraphbegin: byte stream transmission begins;

[0096] 11. bytes paragraph end, the byte stream transmission ends;

[0097] 12. Bytes paragraph continued; Byte stream transmission in progress.

[0098] 13-15, reserved.

[0099] Control flags 1-3 of the session flag group;

[0100] Bits 8-31, reserved; Bits 32-63, original data size.

[0101] Control flags 4-6 of the integer flag group;

[0102] Bits 8-10, data type indicator, started from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types;

[0103] Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used.

[0104] Bit 11, delta;

[0105] Bit 12, delta over delta;

[0106] Bit 13, zigzag;

[0107] Bits 14-19 are reserved.

[0108] Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted by using the greatest common divisor as the divisor;

[0109] Bits 32-35: The number of consecutive zeros starting from the units digit;

[0110] Bits 36-63, the number of integers, are only meaningful in the integer transfer start control block.

[0111] Control flags 7-9 of the floating-point flag group;

[0112] Bits 8-10, data type indicator, started from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types, only meaningful when floating-point numbers are treated as integers;

[0113] Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used.

[0114] Bit 11, delta;

[0115] Bit 12, delta over delta;

[0116] Bit 13, zigzag;

[0117] Bit 14, XOR;

[0118] Bits 15-19, reserved;

[0119] Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted by using the greatest common divisor as the divisor;

[0120] Bits 32-35, the number of decimal places;

[0121] Bits 36-63, the number of decimals, are only meaningful in the integer transfer start control block.

[0122] Control flags 10-12 of the byte stream flag group;

[0123] Bits 8-31 are reserved.

[0124] Bits 32-63, the number of bytes to be transmitted.

[0125] The size of a mini buffer = count(data) x size(data type);

[0126] The second size refers to the data type size. All transmitted data has a data type; common data types include integers, floating-point numbers, and short integers. The size of a data type is fixed. For example, the size of a commonly used integer is 4 bytes, meaning 4 bytes. The size of a short integer is 2 bytes, a typical floating-point number is 4 bytes, a double-precision floating-point number is 8 bytes, and so on.

[0127] count is a constant N, with a minimum of 60 and a maximum of 1200;

[0128] The initial N numbers are stored in a mini-buffer. After the N numbers are evaluated, the encoding algorithm is determined, which is called the starting algorithm. The mini-buffer is a circular buffer that is used cyclically.

[0129] The data stream enters a mini-buffer, and encoding is triggered every N numbers.

[0130] If the starting algorithm fails to work after reading more numbers, a control block in progress is introduced;

[0131] If the data is an integer, control flag 6 is introduced, indicating "Integer transmission in progress".

[0132] If the data is a floating-point number, control flag 9 is introduced, indicating "Floating-point data transfer in progress".

[0133] Based on the above method, a hybrid data dynamic adjustment mini buffer encoding device in this embodiment includes: at least one memory and at least one processor;

[0134] The at least one memory is used to store a machine-readable program;

[0135] The at least one processor is configured to invoke the machine-readable program to execute a hybrid data dynamic adjustment mini-buffer encoding method.

[0136] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or substitutions made by a person skilled in the art that conform to the claims of the present invention regarding a hybrid data dynamic adjustment mini buffer encoding method and apparatus should fall within the patent protection scope of the present invention.

[0137] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for dynamically adjusting mini-buffer encoding of mixed data, characterized in that, In data transmission, based on the data stream, the Simplestream compression algorithm modifies simple8b by redefining selector 0 and selector1 of simple8b. The Simplestream compression algorithm has a minimum unit of 8 bytes, which is divided into control blocks, data blocks, and byte blocks. The control block is defined by selector 0; the data block is based on the original simple8b, with selector 0 removed. Redefine selector 1; the byte block is not encoded, and uses the native binary code; Selector 1 is redefined in the data block, Bits 0-3 selector 1; Simplestream redefines the meaning of selector 1 and supports run-length encoding; Bit32-63 is a 32-bit integer, representing a value; Bit 4-31 is a 28-bit integer representing the number of times the value is repeated. Selector 0 is the new control block, Bits 0-3 selector 0; Simplestream redefines simple8b's original selector 0 as the header of a control block and assigns new meaning to the remaining 60 bits; Bits 4-7 are control flags, which consist of four groups: session flag, integer flag, floating-point flag, and byte stream flag. Each group has three values, representing start, end, and in progress, respectively. Bits 8-63 are further defined according to different control flags; The data stream enters a mini-buffer, and encoding is triggered every N numbers. If the starting algorithm fails to work after reading more numbers, a control block in progress is introduced; If the data is an integer, control flag 6 is introduced, indicating "Integer transmission in progress". If the data is a floating-point number, control flag 9 is introduced, indicating "Floating-point number transfer in progress".

2. The hybrid data dynamic adjustment mini-buffer coding method according to claim 1, characterized in that, The control flags Bits 4-7 occupy 4 bits each. There are 16 flags in total, numbered 0-15, and their definitions are as follows:

0. reserved; 1. Session begin: The session begins.

2. Session end: The session ends.

3. Session continued; 4. Integerparagraphbegin: Integer transmission begins; 5. Integer paragraph end, integer transmission ends; 6. Integer paragraph continued; Integer transmission in progress.

7. float paragraphbegin, floating-point number transmission begins; 8. float paragraph end: Floating-point number transmission ends; 9. Floating paragraph continued; Floating-point number transfer in progress.

10. bytesparagraphbegin: Byte stream transmission begins; 11. Bytes paragraph end: The byte stream transmission ends.

12. Bytes paragraph continued; Byte stream transmission in progress. 13-15, reserved.

3. The hybrid data dynamic adjustment mini-buffer coding method according to claim 2, characterized in that, Control flags 1-3 of the session flag group; Bits 8-31, reserved; Bits 32-63, original data size.

4. The hybrid data dynamic adjustment mini-buffer coding method according to claim 3, characterized in that, Control flags 4-6 of the integer flag group; Bits 8-10, data type indicator, starting from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types; Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used. Bit 11, delta; Bit 12, delta over delta; Bit 13, zigzag; Bits 14-19 are reserved. Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted using the greatest common divisor as the divisor; Bits 32-35: The number of consecutive zeros starting from the units digit; Bits 36-63, the number of integers, are only meaningful in the integer transfer start control block.

5. The hybrid data dynamic adjustment mini-buffer coding method according to claim 4, characterized in that, Control flags 7-9 of the floating-point flag group; Bits 8-10, data type indicator, started from 0: int64, uint64, int32, uint32, int16, uint16, int8, uint8, representing 8 integer data types, only meaningful when floating-point numbers are treated as integers; Bits 11-19, bit map of encoding method, indicate the compression algorithm used. It is presented in bitmap format, with one bit corresponding to one algorithm. 1 indicates that an algorithm is used, and 0 indicates that no algorithm is used. Bit 11, delta; Bit 12, delta over delta; Bit 13, zigzag; Bit 14, XOR; Bits 15-19, reserved; Bits 20-31, Greatest Common Divisor, indicate that the encoded set of numbers has a greatest common divisor, and the subsequent encoding has been adjusted using the greatest common divisor as the divisor; Bits 32-35, the number of decimal places; Bits 36-63, the number of decimals, are only meaningful in the integer transfer start control block.

6. The hybrid data dynamic adjustment mini-buffer coding method according to claim 5, characterized in that, Control flags 10-12 of the byte stream flag group; Bits 8-31, reserved. Bits 32-63, the number of bytes to be transmitted.

7. The hybrid data dynamic adjustment mini-buffer coding method according to claim 6, characterized in that, The size of the mini-buffer = count x size; count is a constant N, with a minimum of 60 and a maximum of 1200; The initial N numbers are stored in a mini-buffer. After the N numbers are evaluated, the encoding algorithm is determined, which is called the starting algorithm. The mini-buffer is a circular buffer that is used cyclically. The data stream enters a mini-buffer, and encoding is triggered every N numbers.

8. A hybrid data dynamic adjustment mini buffer encoding device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 7.

Images