Data coding method and device, storage medium and program product

Abstract

The invention provides a data coding method and device, a storage medium and a program product. The data coding method comprises the following steps: acquiring a to-be-coded data column from a data table, wherein the to-be-coded data column comprises multiple rows of data; writing a first row of data in the to-be-coded data column into a cache, and taking the first row of data as current storage data; according to the sequence from the second row to the Nth row in the to-be-coded data column, the XOR value of the ith row of data and the current storage data is calculated, # imgabs0 #, and N is the total number of rows; if the exclusive-or value is not zero, judging whether the exclusive-or value is within a specified range or not; if the exclusive-or value is within a specified range, performing exclusive-or coding on the ith row of data by using the exclusive-or value to obtain a coded value of the ith row of data; if the exclusive-or value is not within the specified range, performing Huffman coding on the ith row of data by using the exclusive-or value to obtain a coded value of the ith row of data; and writing the coded value of the ith row of data into the cache.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data encoding method and device, a storage medium, and a program product. Background Art

[0002] In vehicle systems, multiple signals often occur at the same time with the same timestamp. For example, the braking operation of a vehicle triggers multiple signals, or the vehicle periodically collects status information from various systems. These signals are usually time-synchronized. Therefore, by storing information with the same timestamp together, storage space can be effectively saved.

[0003] On this basis, the data can be compressed and encoded to further save the space occupied by storing information. Summary of the invention

[0004] The inventors noticed that in the related art, XOR coding can effectively compress long zero sequences by using the XOR value of two adjacent data for coding. However, during the coding process, if a certain data deviates greatly from the adjacent data, the XOR value of the two data will be large, thus failing to provide a higher compression ratio.

[0005] Accordingly, the present disclosure provides a data encoding method, which selects different encoding methods according to the size of the XOR value of two data, thereby providing a higher compression ratio.

[0006] In a first aspect of the present disclosure, a data encoding method is provided, which is executed by a data encoding device, comprising: obtaining a data column to be encoded from a data table, wherein the data column to be encoded includes multiple rows of data; writing the first row of data in the data column to be encoded into a cache, and using the first row of data as current storage data; calculating an exclusive OR value of the i-th row of data and the current storage data in order from the second row to the Nth row in the data column to be encoded, , N is the total number of rows; if the XOR value is not zero, determine whether the XOR value is within a specified range; if the XOR value is within the specified range, perform XOR encoding on the i-th row of data using the XOR value to obtain the encoding value of the i-th row of data; if the XOR value is not within the specified range, perform Huffman encoding on the i-th row of data using the XOR value to obtain the encoding value of the i-th row of data; write the encoding value of the i-th row of data into the cache.

[0007] In some embodiments, the determining whether the XOR value is within a specified range includes: determining the number of leading zeros and the number of suffix zeros of the XOR value; determining whether the number of suffix zeros is less than or equal to the suffix zero threshold; if the number of suffix zeros is less than or equal to the suffix zero threshold, determining whether the number of leading zeros is equal to the currently stored leading zero value; if the number of leading zeros is not equal to the currently stored leading zero value, determining whether the number of leading zeros is less than the leading zero threshold; if the number of leading zeros is less than the leading zero threshold, determining that the XOR value is not within the specified range; if the number of leading zeros is greater than or equal to the leading zero threshold, determining that the XOR value is within the specified range.

[0008] In some embodiments, the XOR encoding of the i-th row of data using the XOR value includes: generating a flag bit of the i-th row of data based on the number of leading zeros and the number of trailing zeros; querying a leading zero mapping value corresponding to the number of leading zeros based on a predetermined leading zero mapping table; and generating an encoding value of the i-th row of data based on the flag bit of the i-th row of data, the leading zero mapping value and the XOR value.

[0009] In some embodiments, generating a flag bit for the i-th row of data includes: generating a flag bit for the i-th row of data to indicate that the XOR value is not zero, a flag bit to indicate encoding, and a flag bit to indicate that the number of leading zeros is greater than or equal to the leading zero threshold.

[0010] In some embodiments, generating the encoded value of the i-th row of data includes: generating the XOR encoded value of the i-th row of data based on the flag bit indicating that the XOR value is not zero, the flag bit indicating encoding, the flag bit indicating that the number of leading zeros is greater than or equal to the leading zero threshold, the leading zero mapping value, and the XOR value.

[0011] In some embodiments, the i-th row of data is used as currently stored data; and the number of leading zeros is used as the currently stored leading zero value.

[0012] In some embodiments, performing Huffman encoding on the i-th row of data using the XOR value includes: generating a flag bit of the i-th row of data based on the number of leading zeros and the number of trailing zeros; querying a Huffman coding mapping value corresponding to the frequency of occurrence of the XOR value in a predetermined Huffman coding table; and generating a coding value of the i-th row of data based on the flag bit of the i-th row of data and the Huffman coding mapping value.

[0013] In some embodiments, generating a flag for the i-th row of data includes: generating a flag for the i-th row of data to indicate that the XOR value is not zero, a flag to indicate that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag to indicate that the number of leading zeros is less than the leading zero threshold.

[0014] In some embodiments, generating the Huffman coding value of the i-th row of data includes: generating the Huffman coding value of the i-th row of data according to the flag bit indicating that the XOR value is not zero, the flag bit indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, the flag bit indicating that the number of leading zeros is less than the leading zero threshold, and the Huffman coding mapping value.

[0015] In some embodiments, if the number of suffix zeros is greater than the suffix zero threshold, or if the number of leading zeros is equal to the currently stored leading zero value, a flag bit of the i-th row of data is generated; and based on the flag bit and the XOR value, an encoded value of the i-th row of data is generated.

[0016] In some embodiments, generating a flag bit for the i-th row of data includes: generating a flag bit for indicating that the XOR value is not zero, and a flag bit for indicating that no encoding is performed, for the i-th row of data.

[0017] In some embodiments, generating the encoding value of the i-th row of data includes: generating the encoding value of the i-th row of data based on the flag bit indicating that the XOR value is not zero, the flag bit indicating no encoding, and the XOR value.

[0018] In some embodiments, the i-th row of data is used as currently stored data; and the number of leading zeros is used as the currently stored leading zero value.

[0019] In some embodiments, when the XOR value is zero, a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is to be performed are generated for the i-th row of data; based on the flag bit for indicating that the XOR value is zero and the flag bit for indicating that no encoding is required, the encoding of the i-th row of data is generated.

[0020] In some embodiments, the i-th row of data is used as currently stored data.

[0021] In some embodiments, after obtaining the data column to be encoded from the data table, it is detected whether each row in the data column to be encoded includes data; if the mth row does not include data, the data of the adjacent rows are filled into the mth row, so as to update the data column to be encoded. .

[0022] In some embodiments, the data table can be divided into a plurality of data rows or a plurality of data columns, wherein the plurality of data included in each data row has the same timestamp, and the plurality of data included in each data column has the same format.

[0023] In a second aspect of the present disclosure, a data encoding device is provided, comprising: a memory; a processor, coupled to the memory, the processor being configured to execute a method as described in any of the above embodiments based on instructions stored in the memory.

[0024] In a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0025] In a fourth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0026] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 The present invention is a flowchart of a data encoding method according to an embodiment of the present invention.

[0029] Figure 2 The figure is a flowchart of an XOR encoding method according to an embodiment of the present disclosure.

[0030] Figure 3 The figure is a flowchart of a Huffman coding method according to an embodiment of the present disclosure.

[0031] Figure 4 The present invention is a schematic diagram of the structure of a data encoding device according to an embodiment of the present invention.

[0032] Figure 5 The figure is a flowchart of a data encoding method according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0037] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0038] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] Figure 1 The following is a flow chart of a data encoding method according to an embodiment of the present disclosure. In some embodiments, the following data encoding method is performed by a data encoding device, including steps 11-17.

[0040] In step 11, a data column to be encoded is obtained from a data table, wherein the data column to be encoded includes multiple rows of data.

[0041] In some embodiments, the data table can be divided into multiple data rows or multiple data columns, wherein the multiple data included in each data row have the same timestamp, and the multiple data included in each data column have the same format.

[0042] That is, the data in the data table adopts a matrix storage structure, for example, the storage structure can be called vTimeGrid. In this storage structure, multiple data with the same timestamp are stored in the same row, which can effectively save storage space. In addition, through data columns, a large amount of time series data can be efficiently stored and queried.

[0043] In some embodiments, after obtaining the data column to be encoded from the data table, each row in the data column to be encoded is detected to see whether it includes data. If the mth row does not include data, the data of the adjacent row is filled into the mth row to update the data column to be encoded. , N is the total number of rows in the data column to be encoded.

[0044] For example, in the data column to be encoded, if the data in the third row is missing, the data in the second row is filled into the third row to ensure the continuity of the data.

[0045] For example, the first column of the data table is a timestamp column, which can be compressed using secondary difference coding.

[0046] For example, if the data in a column of a data table is string data, dictionary encoding can be used for compression.

[0047] For example, the data in a column in a data table is numerical data and can be processed according to the scheme described in the present disclosure.

[0048] In step 12, the first row of data in the to-be-encoded data column is written into the cache, and the first row of data is used as the current storage data.

[0049] For example, if the first row of data is 75, the corresponding binary representation is 1001011, and then 1001011 is directly written into the cache.

[0050] Calculate the XOR value of the i-th row data and the currently stored data in the order from the 2nd row to the Nth row in the data column to be encoded. , N is the total number of rows.

[0051] In step 14, if the XOR value is not zero, it is determined whether the XOR value is within a specified range.

[0052] In some embodiments, the step of determining whether the XOR value is within a specified range includes the following.

[0053] 1) Determine the number of leading and trailing zeros in the XOR value.

[0054] 2) Determine whether the number of suffix zeros is less than or equal to the suffix zero threshold.

[0055] 3) If the number of trailing zeros is less than or equal to the trailing zero threshold, determine whether the number of leading zeros is equal to the currently stored leading zero value.

[0056] 4) If the number of leading zeros is not equal to the currently stored leading zero value, determine whether the number of leading zeros is less than the leading zero threshold.

[0057] 5) If the number of leading zeros is less than the leading zero threshold, it is determined that the XOR value is not within the specified range.

[0058] 6) If the number of leading zeros is greater than or equal to the leading zero threshold, determine that the XOR value is within the specified range.

[0059] For example, if the XOR value is 15, the binary representation of 15 is 1111. In this case, the number of leading zeros in the XOR value is 24. A larger number of leading zeros in the XOR value indicates that the XOR value is smaller. If the leading zero threshold is set to 20, since 24>20, it can be determined that the XOR value is within the specified range.

[0060] For another example, if the XOR value is 931, the binary representation of 931 is 1110100011. In this case, the number of leading zeros in the XOR value is 18. A smaller number of leading zeros in the XOR value indicates that the XOR value is larger. If the leading zero threshold is set to 20, since 18<20, it can be determined that the XOR value is not within the specified range.

[0061] In step 15, if the XOR value is within the specified range, the XOR value is used to perform XOR encoding on the i-th row of data to obtain the encoding value of the i-th row of data.

[0062] Figure 2 The flowchart of the XOR encoding method of one embodiment of the present disclosure is shown in FIG. 1. In some embodiments, the following XOR encoding method is performed by a data encoding device, including steps 21-23.

[0063] In step 21, a flag bit of the i-th row of data is generated according to the number of leading zeros and the number of trailing zeros of the i-th row of data.

[0064] In some embodiments, a flag bit for indicating that the XOR value is not zero, a flag bit for indicating that encoding is performed, and a flag bit for indicating that the number of leading zeros is greater than or equal to a leading zero threshold are generated for the i-th row of data.

[0065] In step 22, a leading zero mapping value corresponding to the number of leading zeros is searched according to a predetermined leading zero mapping table.

[0066] For example, if the number of leading zeros is 24, according to the predetermined leading zero mapping table, the leading zero mapping value corresponding to the number of leading zeros 24 is 7, and the corresponding binary representation is 111.

[0067] In step 23, the encoding value of the i-th row of data is generated according to the flag bit, the leading zero mapping value and the XOR value of the i-th row of data.

[0068] In some embodiments, an XOR encoding value of the i-th row of data is generated based on a flag bit indicating that the XOR value is not zero, a flag bit indicating that encoding is to be performed, a flag bit indicating that the number of leading zeros is greater than or equal to a leading zero threshold, a leading zero mapping value, and the XOR value.

[0069] For example, the flag bit used to indicate that the XOR value is not zero is "1", the flag bit used to indicate encoding is "1", the flag bit used to indicate that the number of leading zeros is greater than or equal to the leading zero threshold is "0", and the leading zero mapping value is 7. The corresponding binary representation is "111", and the XOR encoding value of the i-th row of data is generated as shown in expression (1).

[0070] 1+1+0+111+valid part of XOR value (1) In addition, the i-th row of data is used as the currently stored data, and the number of leading zeros in the i-th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0071] return Figure 1 In step 16, if the XOR value is not within the specified range, the XOR value is used to perform Huffman encoding on the i-th row of data to obtain the encoding value of the i-th row of data.

[0072] Figure 3 The flowchart of the Huffman coding method according to one embodiment of the present disclosure is shown in FIG. 1. In some embodiments, the following Huffman coding method is performed by a data coding device, including steps 31-33.

[0073] In step 31, a flag bit of the i-th row of data is generated according to the number of leading zeros and the number of trailing zeros.

[0074] In some embodiments, a flag is generated for the i-th row of data to indicate that the XOR value is not zero, a flag is generated to indicate that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag is generated to indicate that the number of leading zeros is less than the leading zero threshold.

[0075] In step 32, a Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched in a predetermined Huffman coding table.

[0076] For example, if the XOR value is 931, the Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is 1 when it is queried in the predetermined Huffman coding table.

[0077] In step 33, the encoding value of the i-th row of data is generated according to the flag bit of the i-th row of data and the Huffman coding mapping value.

[0078] In some embodiments, a Huffman coding value of the i-th row of data is generated based on a flag bit indicating that the XOR value is not zero, a flag bit indicating that the number of suffix zeros is less than or equal to a suffix zero threshold and the number of leading zeros is not equal to a currently stored leading zero value, a flag bit indicating that the number of leading zeros is less than a leading zero threshold, and a Huffman coding mapping value.

[0079] For example, the flag bit used to indicate that the XOR value is not zero is "1", the flag bit used to indicate that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value is "1", the flag bit used to indicate that the number of leading zeros is less than the leading zero threshold is "1", and the Huffman coding mapping value is "1", then the Huffman coding value of the i-th row of data is as shown in expression (2).

[0080] 1+1+1+1 (2) It should be noted that, in this embodiment, since the XOR value is not within the specified range, it indicates that the corresponding i-th row of data is abnormal. In this case, in order to avoid the i-th row of data affecting the encoding operation of subsequent data, the i-th row of data will not be used as the current stored data, and the number of leading zeros in the i-th row of data will not be used as the currently stored leading zero value.

[0081] In step 17, the encoded value of the i-th row of data is written into the cache.

[0082] In some embodiments, if the number of suffix zeros is greater than the suffix zero threshold, the following operations are performed.

[0083] 1) Generate the flag bit of the i-th row of data.

[0084] For example, a flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

[0085] 2) Generate the encoding value of the i-th row of data based on the flag bit and the XOR value.

[0086] For example, the encoding value of the i-th row of data is generated according to a flag bit indicating that the XOR value is not zero, a flag bit indicating that no encoding is performed, and the XOR value.

[0087] For example, the flag bit used to indicate that the XOR value is not zero is "1" and the flag bit used to indicate that no encoding is performed is "0", then the encoding value of the i-th row of data is as shown in expression (3).

[0088] 1+0+valid part of XOR value (3) In addition, the i-th row of data is used as the currently stored data, and the number of leading zeros in the i-th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0089] In some embodiments, if the number of leading zeros is equal to the currently stored leading zero value, the following operations are performed.

[0090] 1) Generate the flag bit of the i-th row of data.

[0091] For example, a flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

[0092] 2) Generate the encoding value of the i-th row of data based on the flag bit and the XOR value.

[0093] For example, the encoding value of the i-th row of data is generated according to a flag bit indicating that the XOR value is not zero, a flag bit indicating that no encoding is performed, and the XOR value.

[0094] For example, the flag bit used to indicate that the XOR value is not zero is "1" and the flag bit used to indicate that no encoding is performed is "0", then the encoding value of the i-th row of data is as shown in expression (4).

[0095] 1+0+valid part of XOR value (4) In addition, the i-th row of data is used as the currently stored data, and the number of leading zeros in the i-th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0096] In some embodiments, when the XOR value is zero, the following operations are performed.

[0097] 1) Generate a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is performed for the i-th row of data.

[0098] 2) Generate the code of the i-th row of data according to the flag bit used to indicate that the XOR value is zero and the flag bit used to indicate that no coding is required.

[0099] For example, the flag bit used to indicate that the XOR value is zero is "0", and the flag bit used to indicate that no encoding is performed is "0", then the encoding value of the i-th row of data is as shown in Expression (5).

[0100] 0+0 (5) In addition, the i-th row of data is used as the current storage data for use in subsequent data processing.

[0101] In the data encoding method provided in the above embodiment of the present disclosure, different encoding methods are selected according to the size of the XOR value of two data, so as to provide a higher compression ratio.

[0102] Figure 4 The present invention is a schematic diagram of the structure of a data encoding device according to an embodiment of the present invention.

[0103] like Figure 4As shown, the data encoding apparatus 40 can be embodied in the form of a general-purpose computing device. The data encoding apparatus 40 includes a memory 41, a processor 42, and a bus 43 connecting different system components.

[0104] The memory 41 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions of a corresponding embodiment of at least one data encoding method being executed. The non-volatile storage medium includes, but is not limited to, a disk storage, an optical storage, a flash memory, etc.

[0105] The processor 42 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, and other discrete hardware components. Accordingly, each module such as an acquisition module, a calculation module, and an adjustment module may be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or may be implemented by a dedicated circuit that executes corresponding steps.

[0106] For example, the processor 42 is configured to execute instructions stored in the memory to implement the following Figure 1 The method of any one of the embodiments.

[0107] The bus 43 may use any of a variety of bus structures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0108] The interfaces 44, 45, 46 of the data encoding device 40, the memory 41 and the processor 42 can be connected via a bus 43. The input / output interface 44 can provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 45 provides a connection interface for various networked devices. The storage interface 46 provides a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0109] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0111] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0112] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0113] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the following is implemented: Figures 1 to 3 The method of any one of the embodiments.

[0114] The present disclosure also provides a computer program product, including computer instructions, wherein when the computer instructions are executed by a processor, the following is achieved: Figures 1 to 3 The method of any one of the embodiments.

[0115] The present disclosure is described below through specific examples.

[0116] For example, in the data column to be encoded extracted from the data table, the first row of data is 75, the second row of data is 75, the third row of data is 1000, the fourth row of data is 68, the fifth row of data is 32, and the sixth row of data is 106. The following is a sequence from the first row of data to the sixth row of data, and each layer of data is processed in turn. The processing results are shown in Table 1.

[0117] 1. Layer 1 data: 75.

[0118] For layer 1 data, write it directly to the cache.

[0119] For example, the binary representation of 75 is 1001011, and the binary number 1001011 is written into the cache. At this time, the first layer data 75 is used as the current storage data.

[0120] Next, follow the Figure 5 As shown in the flowchart, each data from the 2nd layer data to the 6th layer data is encoded in turn.

[0121] 2. Layer 2 data: 75.

[0122] Since the second layer data and the current storage data are both 75, the XOR value of 75 and 75 is 0. In this case, Figure 2 As shown, the flag bits generated for the layer 2 data include: a flag bit 0 for indicating that the XOR value is zero; and a flag bit 0 for indicating that no encoding is performed.

[0123] Next, according to the flag bit 0 indicating that the XOR value is zero and the flag bit 0 indicating that no encoding is required, the encoding value of the second row of data is generated according to the above expression (5), and the encoding value 00 is written into the cache.

[0124] In addition, the second row of data is used as the current stored data, that is, the current stored data is updated.

[0125] 3. Layer 3 data: 1000.

[0126] The third layer data is 1000, and the current stored data is 75. The XOR value of 75 and 1000 is 931, and the binary representation of 931 is 1110100011.

[0127] According to the XOR value, the number of leading zeros is 18 and the number of trailing zeros is 0.

[0128] Here, the suffix zero threshold is assumed to be 1. In this case, the flag bits generated for the third layer data include: flag bit 1 used to indicate that the XOR value is not zero; flag bit 1 used to indicate that the number of suffix zeros is less than or equal to the suffix zero threshold, and the number of leading zeros is not equal to the currently stored leading zero value; flag bit 1 used to indicate that the number of leading zeros is less than the leading zero threshold.

[0129] Next, the Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched in the predetermined Huffman coding table. Thus, the Huffman coding value of the third row of data is generated according to the above flag bit and the Huffman coding mapping value.

[0130] For example, if the Huffman coding mapping value is 1, then according to the above flag bit Huffman coding mapping value, using the above expression (2), the Huffman coding value of the third row of data generated is 1111, and the coding value 1111 is written into the cache.

[0131] It should be noted that, in this embodiment, since the XOR value 931 is not within the specified range, it indicates that the corresponding 3rd line of data is abnormal. In this case, in order to avoid the influence of the 3rd line of data on the encoding operation of subsequent data, the 3rd line of data will not be used as the current stored data, and the number of leading zeros in the 3rd line of data will not be used as the currently stored leading zero value.

[0132] 4. Layer 4 data: 68.

[0133] The 4th layer data is 68, and the current stored data is 75. The XOR value of 75 and 68 is 15, and the binary representation of 15 is 1111.

[0134] According to the XOR value, the number of leading zeros is 24, and the number of trailing zeros is 0.

[0135] Assume that the leading zero threshold is 21. In this case, the flag bits generated for the layer 4 data include: flag bit 1 for indicating that the XOR value is not zero; flag bit 1 for indicating that encoding is performed; flag bit 0 for indicating that the number of leading zeros is greater than or equal to the leading zero threshold.

[0136] In addition, according to the predetermined leading zero mapping table, the leading zero mapping value corresponding to the number of leading zeros 24 is 7, and the corresponding binary representation is 111. Therefore, according to the above flag bit, the leading zero mapping value and the XOR value, the XOR encoding value of the fourth row of data generated by the above expression (1) is 1101111111.

[0137] It should be noted that when generating the encoding value of the fourth row of data, the XOR value 1111 may be expanded as needed, for example, the XOR value 1111 may be expanded to 40 bits.

[0138] Next, the 4th row of data is used as the currently stored data, and the number of leading zeros in the 4th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0139] 5. Layer 5 data: 32.

[0140] The fifth layer data is 32, and the current stored data is 68, so the XOR value of 68 and 32 is 100, and the binary representation of 100 is 1100100.

[0141] According to the XOR value, the number of leading zeros is 21 and the number of trailing zeros is 2.

[0142] In this case, the flag bits generated for the layer 5 data include: a flag bit 1 for indicating that the XOR value is not zero; and a flag bit 0 for indicating that no encoding is performed.

[0143] Therefore, according to the above flag bit and XOR value, using the above expression (3), the encoding value of the fifth row of data generated is 101100100.

[0144] It should be noted that when generating the encoding value of the fifth row of data, the XOR value 1100100 may be expanded as needed, for example, the XOR value 1100100 may be expanded to 40 bits.

[0145] Next, the 5th row of data is used as the currently stored data, and the number of leading zeros in the 5th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0146] 6. Layer 6 data: 106.

[0147] The 6th layer data is 106, and the current stored data is 32. The XOR value of 32 and 106 is 74, and the binary representation of 74 is 1001010.

[0148] According to the XOR value, the number of leading zeros is 21, and the number of trailing zeros is 1. It should be noted that the number of leading zeros 21 of the XOR value 1001010 is equal to the currently stored leading zero value 21. In this case, the flag bits generated for the 6th layer data include: a flag bit 1 for indicating that the XOR value is not zero; and a flag bit 0 for indicating that no encoding is performed.

[0149] Therefore, according to the above flag bit and XOR value, using the above expression (4), the encoding value of the 6th row of data generated is 101001010.

[0150] It should be noted that when generating the encoding value of the sixth row of data, the XOR value 1001010 may be expanded as needed, for example, the XOR value 1001010 may be expanded to 40 bits.

[0151] Next, the 6th row of data is used as the currently stored data, and the number of leading zeros in the 6th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0152] Table 1.

[0153] By implementing the above-mentioned embodiments of the present disclosure, the following beneficial effects can be obtained.

[0154] • If the difference between adjacent data points is small (for example, the sensor data changes slightly), using XOR encoding can save a lot of space. Assuming the difference between data points is less than a certain threshold, the compression ratio may reach 50%-90%.

[0155] • For data with uneven frequency distribution and large changes (such as time series data of BMS (Battery Management System)), Huffman coding can usually significantly improve the compression efficiency. The compression ratio can sometimes reach 40%-70%.

[0156] • In scenarios where data has both small differences (suitable for XOR coding) and large differences (suitable for Huffman coding), dynamically selecting the appropriate compression method can provide the best compression effect. In practical applications, this hybrid method may increase the overall compression rate of data by 20%-60%.

[0157] In some embodiments, the functional unit described above may be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof, for performing the functions described in the present disclosure.

[0158] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0159] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A data encoding method, performed by a data encoding device, comprising: Acquire a data column to be encoded from a data table, wherein the data column to be encoded includes multiple rows of data; Writing the first row of data in the to-be-encoded data column into a cache, and using the first row of data as currently stored data; Calculate the XOR value of the i-th row data and the currently stored data in the order from the 2nd row to the Nth row in the column of data to be encoded, , N is the total number of rows; If the XOR value is not zero, determining whether the XOR value is within a specified range; If the XOR value is within a specified range, XOR-encode the i-th row of data using the XOR value to obtain an encoding value of the i-th row of data; If the XOR value is not within the specified range, performing Huffman encoding on the i-th row of data using the XOR value to obtain an encoding value of the i-th row of data; Write the encoded value of the i-th row of data into the cache.

2. The method according to claim 1, wherein: The determining whether the XOR value is within a specified range includes: Determining the number of leading zeros and the number of trailing zeros of the XOR value; Determine whether the number of suffix zeros is less than or equal to a suffix zero threshold; If the number of suffix zeros is less than or equal to the suffix zero threshold, determining whether the number of leading zeros is equal to the currently stored leading zero value; If the number of leading zeros is not equal to the currently stored leading zero value, determining whether the number of leading zeros is less than a leading zero threshold; If the number of leading zeros is less than the leading zero threshold, determining that the XOR value is not within a specified range; If the number of leading zeros is greater than or equal to the leading zero threshold, it is determined that the XOR value is within a specified range.

3. The method according to claim 2, wherein: The XOR encoding of the i-th row of data by using the XOR value comprises: Generate a flag bit of the i-th row of data according to the number of leading zeros and the number of trailing zeros; According to a predetermined leading zero mapping table, querying a leading zero mapping value corresponding to the number of leading zeros; Generate an encoding value of the i-th row of data according to the flag bit of the i-th row of data, the leading zero mapping value and the XOR value.

4. The method according to claim 3, wherein: The generating of the flag bit of the i-th row of data comprises: A flag bit for indicating that the XOR value is not zero, a flag bit for indicating that encoding is performed, and a flag bit for indicating that the number of leading zeros is greater than or equal to the leading zero threshold are generated for the i-th row of data.

5. The method according to claim 4, wherein: The generating of the encoding value of the i-th row of data comprises: The XOR encoding value of the i-th row of data is generated according to the flag bit indicating that the XOR value is not zero, the flag bit indicating encoding, the flag bit indicating that the number of leading zeros is greater than or equal to the leading zero threshold, the leading zero mapping value, and the XOR value.

6. The method according to claim 3, further comprising: Taking the i-th row of data as current storage data; The number of leading zeros is used as the currently stored leading zero value.

7. The method according to claim 2, wherein: The performing Huffman encoding on the i-th row of data by using the XOR value comprises: Generate a flag bit of the i-th row of data according to the number of leading zeros and the number of trailing zeros; In a predetermined Huffman coding table, a Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched; Generate a coding value of the i-th row of data according to the flag bit of the i-th row of data and the Huffman coding mapping value.

8. The method according to claim 7, wherein: The generating of the flag bit of the i-th row of data comprises: Generate for the i-th row of data a flag bit for indicating that the XOR value is not zero, a flag bit for indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag bit for indicating that the number of leading zeros is less than the leading zero threshold.

9. The method according to claim 8, wherein: The generating of the Huffman coding value of the i-th row of data comprises: The Huffman coding value of the i-th row of data is generated according to the flag bit indicating that the XOR value is not zero, the flag bit indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, the flag bit indicating that the number of leading zeros is less than the leading zero threshold, and the Huffman coding mapping value.

10. The method according to claim 2, further comprising: If the number of suffix zeros is greater than the suffix zero threshold, or if the number of leading zeros is equal to the currently stored leading zero value, generate a flag bit for the i-th row of data; Generate the encoding value of the i-th row of data according to the flag bit and the XOR value.

11. The method according to claim 10, wherein: The generating of the flag bit of the i-th row of data comprises: A flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

12. The method according to claim 11, wherein: The generating of the encoding value of the i-th row of data comprises: The encoding value of the i-th row of data is generated according to the flag bit indicating that the XOR value is not zero, the flag bit indicating that no encoding is performed, and the XOR value.

13. The method according to claim 10, further comprising: Taking the i-th row of data as current storage data; The number of leading zeros is used as the currently stored leading zero value.

14. The method according to claim 1, further comprising: When the XOR value is zero, generating, for the i-th row of data, a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is performed; The encoding of the i-th row of data is generated according to the flag bit indicating that the XOR value is zero and the flag bit indicating that no encoding is required.

15. The method according to claim 14, further comprising: The i-th row of data is used as the current storage data.

16. The method according to claim 1, further comprising: After obtaining the data column to be encoded from the data table, detecting whether each row in the data column to be encoded includes data; If the mth row does not include data, the data of the adjacent row is filled into the mth row, so as to update the data column to be encoded. .

17. The method according to any one of claims 1 to 16, wherein: The data table can be divided into a plurality of data rows or a plurality of data columns, wherein the plurality of data included in each data row has the same time stamp, and the plurality of data included in each data column has the same format.

18. A data encoding device, comprising: Memory; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 17 based on instructions stored in the memory.

19. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 17 is implemented.

20. A computer program product, comprising computer instructions, wherein the computer instructions implement the method according to any one of claims 1 to 17 when executed by a processor.

Images