Data encoding method and device, storage medium and program product

Abstract

The present disclosure provides a data encoding method and device, a storage medium, and a program product. The data encoding method includes: obtaining a data column to be encoded from a data table, wherein the data column to be encoded includes multiple rows of data; writing the first row of data in the data column to be encoded into a cache, and using the first row of data as the current stored data; calculating the exclusive OR value of the i-th row of data and the current stored data in order from the second row to the N-th row in the data column to be encoded, #imgabs0#, where N is the total number of rows; if the exclusive OR value is not zero, determining whether the exclusive OR value is within a specified range; if the exclusive OR value is within the specified range, performing exclusive OR encoding on the i-th row of data using the exclusive OR value to obtain the encoding value of the i-th row of data; if the exclusive OR value is not within the specified range, performing Huffman encoding on the i-th row of data using the exclusive OR value to obtain the encoding value of the i-th row of data; and writing the encoding value of the i-th row of data into the cache.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data encoding method and device, a storage medium, and a program product. Background Art

[0002] In automotive systems, multiple signals often occur simultaneously at the same timestamp. For example, a vehicle's braking operation can trigger multiple signals, or the vehicle can periodically collect status information from various systems. These signals are often time-synchronized. Therefore, storing information with the same timestamp together can effectively save storage space.

[0003] On this basis, the data can be compressed and encoded to further save the space occupied by storing information. Summary of the Invention

[0004] The inventors noted that in related technologies, exclusive-OR (XOR) encoding effectively compresses long zero sequences by using the XOR value of two adjacent data. However, during the encoding process, if a data point deviates significantly from its adjacent data, the XOR value of the two data points will be large, thus failing to provide a higher compression ratio.

[0005] Accordingly, the present disclosure provides a data encoding method, which selects different encoding methods according to the size of the XOR value of two data, thereby providing a higher compression ratio.

[0006] In a first aspect of the present disclosure, a data encoding method is provided, which is performed by a data encoding device, comprising: obtaining a data column to be encoded from a data table, wherein the data column to be encoded includes multiple rows of data; writing the first row of data in the data column to be encoded into a cache, and using the first row of data as currently stored data; and calculating an exclusive OR value of the i-th row of data and the currently stored data in order from the second row to the Nth row in the data column to be encoded, , N is the total number of rows; if the XOR value is not zero, determine whether the XOR value is within a specified range; if the XOR value is within the specified range, perform XOR encoding on the i-th row of data using the XOR value to obtain the encoding value of the i-th row of data; if the XOR value is not within the specified range, perform Huffman encoding on the i-th row of data using the XOR value to obtain the encoding value of the i-th row of data; and write the encoding value of the i-th row of data into the cache.

[0007] In some embodiments, determining whether the XOR value is within a specified range includes: determining the number of leading zeros and the number of trailing zeros of the XOR value; determining whether the number of trailing zeros is less than or equal to the trailing zero threshold; if the number of trailing zeros is less than or equal to the trailing zero threshold, determining whether the number of leading zeros is equal to the currently stored leading zero value; if the number of leading zeros is not equal to the currently stored leading zero value, determining whether the number of leading zeros is less than the leading zero threshold; if the number of leading zeros is less than the leading zero threshold, determining that the XOR value is not within the specified range; if the number of leading zeros is greater than or equal to the leading zero threshold, determining that the XOR value is within the specified range.

[0008] In some embodiments, the XOR encoding of the i-th row of data using the XOR value includes: generating a flag bit of the i-th row of data based on the number of leading zeros and the number of trailing zeros; querying the leading zero mapping value corresponding to the number of leading zeros according to a predetermined leading zero mapping table; and generating an encoding value of the i-th row of data based on the flag bit of the i-th row of data, the leading zero mapping value and the XOR value.

[0009] In some embodiments, generating a flag bit for the i-th row of data includes: generating a flag bit for the i-th row of data to indicate that the XOR value is not zero, a flag bit to indicate encoding, and a flag bit to indicate that the number of leading zeros is greater than or equal to the leading zero threshold.

[0010] In some embodiments, generating the encoding value of the i-th row of data includes: generating the XOR encoding value of the i-th row of data based on the flag bit indicating that the XOR value is not zero, the flag bit indicating encoding, the flag bit indicating that the number of leading zeros is greater than or equal to the leading zero threshold, the leading zero mapping value, and the XOR value.

[0011] In some embodiments, the i-th row of data is used as the currently stored data; and the number of leading zeros is used as the currently stored leading zero value.

[0012] In some embodiments, performing Huffman encoding on the i-th row of data using the XOR value includes: generating a flag bit of the i-th row of data based on the number of leading zeros and the number of trailing zeros; querying a Huffman coding mapping value corresponding to the frequency of occurrence of the XOR value in a predetermined Huffman coding table; and generating a coding value of the i-th row of data based on the flag bit of the i-th row of data and the Huffman coding mapping value.

[0013] In some embodiments, generating a flag for the i-th row of data includes: generating a flag for the i-th row of data to indicate that the XOR value is not zero, a flag to indicate that the number of trailing zeros is less than or equal to the trailing zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag to indicate that the number of leading zeros is less than the leading zero threshold.

[0014] In some embodiments, generating the Huffman coding value of the i-th row of data includes: generating the Huffman coding value of the i-th row of data based on the flag indicating that the XOR value is not zero, the flag indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, the flag indicating that the number of leading zeros is less than the leading zero threshold, and the Huffman coding mapping value.

[0015] In some embodiments, if the number of suffix zeros is greater than the suffix zero threshold, or if the number of leading zeros is equal to the currently stored leading zero value, a flag bit of the i-th row of data is generated; and based on the flag bit and the XOR value, an encoding value of the i-th row of data is generated.

[0016] In some embodiments, generating a flag bit for the i-th row of data includes: generating a flag bit for indicating that the XOR value is not zero, and a flag bit for indicating that no encoding is performed, for the i-th row of data.

[0017] In some embodiments, generating the encoding value of the i-th row of data includes: generating the encoding value of the i-th row of data based on the flag indicating that the XOR value is not zero, the flag indicating no encoding, and the XOR value.

[0018] In some embodiments, the i-th row of data is used as the currently stored data; and the number of leading zeros is used as the currently stored leading zero value.

[0019] In some embodiments, when the XOR value is zero, a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is to be performed are generated for the i-th row of data; based on the flag bit for indicating that the XOR value is zero and the flag bit for indicating that no encoding is required, the encoding of the i-th row of data is generated.

[0020] In some embodiments, the i-th row of data is used as the currently stored data.

[0021] In some embodiments, after obtaining the data column to be encoded from the data table, each row in the data column to be encoded is detected to see whether it contains data; if the mth row does not contain data, the data of the adjacent rows are filled into the mth row to update the data column to be encoded. .

[0022] In some embodiments, the data table can be divided into multiple data rows or multiple data columns, wherein the multiple data included in each data row have the same timestamp, and the multiple data included in each data column have the same format.

[0023] In a second aspect of the present disclosure, a data encoding device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the method described in any of the above embodiments based on instructions stored in the memory.

[0024] In a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0025] In a fourth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method as described in any one of the above embodiments is implemented.

[0026] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 The figure is a flowchart of a data encoding method according to an embodiment of the present disclosure.

[0029] Figure 2 The figure is a flowchart of an XOR encoding method according to an embodiment of the present disclosure.

[0030] Figure 3 The figure is a flowchart of a Huffman coding method according to an embodiment of the present disclosure.

[0031] Figure 4 This is a structural diagram of a data encoding device according to an embodiment of the present disclosure.

[0032] Figure 5 Schematic diagram of a data encoding method according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0034] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0037] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0038] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] Figure 1 1 is a flow chart of a data encoding method according to an embodiment of the present disclosure. In some embodiments, the following data encoding method is performed by a data encoding device, including steps 11-17.

[0040] In step 11, a data column to be encoded is obtained from a data table, wherein the data column to be encoded includes multiple rows of data.

[0041] In some embodiments, the data table can be divided into multiple data rows or multiple data columns, wherein the multiple data included in each data row have the same timestamp, and the multiple data included in each data column have the same format.

[0042] In other words, the data in the data table uses a matrix storage structure, for example, called vTimeGrid. In this storage structure, multiple data with the same timestamp are stored in the same row, effectively saving storage space. Furthermore, data columns enable efficient storage and querying of large amounts of time series data.

[0043] In some embodiments, after obtaining the data column to be encoded from the data table, each row in the data column to be encoded is checked to see if it contains data. If the mth row does not contain data, the data of the adjacent rows are filled into the mth row to update the data column to be encoded. , N is the total number of rows in the data column to be encoded.

[0044] For example, in the data column to be encoded, if the data in the third row is missing, the data in the second row is filled into the third row to ensure the continuity of the data.

[0045] For example, the first column of the data table is a timestamp column, which can be compressed using secondary difference coding.

[0046] For example, if the data in a column of a data table is string data, dictionary encoding can be used for compression.

[0047] For example, the data in a column of a data table is numerical data and can be processed according to the scheme described in this disclosure.

[0048] In step 12, the first row of data in the data column to be encoded is written into the cache, and the first row of data is used as the current storage data.

[0049] For example, if the first row of data is 75, the corresponding binary representation is 1001011, and then 1001011 is directly written into the cache.

[0050] Calculate the XOR value of the i-th row data and the currently stored data in the order from the 2nd row to the Nth row in the data column to be encoded. , N is the total number of rows.

[0051] In step 14, if the XOR value is not zero, it is determined whether the XOR value is within a specified range.

[0052] In some embodiments, the step of determining whether the XOR value is within a specified range includes the following.

[0053] 1) Determine the number of leading and trailing zeros in the XOR value.

[0054] 2) Determine whether the number of suffix zeros is less than or equal to the suffix zero threshold.

[0055] 3) If the number of trailing zeros is less than or equal to the trailing zero threshold, determine whether the number of leading zeros is equal to the currently stored leading zero value.

[0056] 4) If the number of leading zeros is not equal to the currently stored leading zero value, determine whether the number of leading zeros is less than the leading zero threshold.

[0057] 5) If the number of leading zeros is less than the leading zero threshold, it is determined that the XOR value is not within the specified range.

[0058] 6) If the number of leading zeros is greater than or equal to the leading zero threshold, determine that the XOR value is within the specified range.

[0059] For example, if the XOR value is 15, the binary representation of 15 is 1111. In this case, the number of leading zeros in the XOR value is 24. A larger number of leading zeros in the XOR value indicates a smaller XOR value. If the leading zero threshold is set to 20, since 24 > 20, the XOR value can be determined to be within the specified range.

[0060] For example, if the XOR value is 931, the binary representation of 931 is 1110100011. In this case, the number of leading zeros in the XOR value is 18. A smaller number of leading zeros in the XOR value indicates a larger XOR value. If the leading zero threshold is set to 20, since 18 < 20, it can be determined that the XOR value is not within the specified range.

[0061] In step 15, if the XOR value is within the specified range, the XOR value is used to perform XOR encoding on the i-th row of data to obtain the encoding value of the i-th row of data.

[0062] Figure 2 2 is a flow chart of an XOR encoding method according to an embodiment of the present disclosure. In some embodiments, the following XOR encoding method is performed by a data encoding device, including steps 21-23.

[0063] In step 21 , a flag bit of the i-th row of data is generated according to the number of leading zeros and the number of trailing zeros of the i-th row of data.

[0064] In some embodiments, a flag indicating that the XOR value is not zero, a flag indicating that encoding is performed, and a flag indicating that the number of leading zeros is greater than or equal to a leading zero threshold are generated for the i-th row of data.

[0065] In step 22, a leading zero mapping value corresponding to the number of leading zeros is searched according to a predetermined leading zero mapping table.

[0066] For example, if the number of leading zeros is 24, according to the predetermined leading zero mapping table, the leading zero mapping value corresponding to the number of leading zeros 24 is 7, and the corresponding binary representation is 111.

[0067] In step 23, the encoding value of the i-th row of data is generated according to the flag bit, the leading zero mapping value and the XOR value of the i-th row of data.

[0068] In some embodiments, an XOR encoding value of the i-th row of data is generated based on a flag bit indicating that the XOR value is not zero, a flag bit indicating that encoding is to be performed, a flag bit indicating that the number of leading zeros is greater than or equal to a leading zero threshold, a leading zero mapping value, and an XOR value.

[0069] For example, the flag bit used to indicate that the XOR value is not zero is "1", the flag bit used to indicate that encoding is performed is "1", the flag bit used to indicate that the number of leading zeros is greater than or equal to the leading zero threshold is "0", and the leading zero mapping value is 7. The corresponding binary representation is "111", and the XOR encoding value of the i-th row data is generated as shown in Expression (1).

[0070] 1+1+0+111+valid part of XOR value (1)

[0071] In addition, the data in the i-th row is used as the currently stored data, and the number of leading zeros in the data in the i-th row is used as the currently stored leading zero value for use in subsequent data processing.

[0072] return Figure 1 In step 16, if the XOR value is not within the specified range, Huffman encoding is performed on the i-th row data using the XOR value to obtain the encoding value of the i-th row data.

[0073] Figure 3 FIG. 1 is a flow chart of a Huffman encoding method according to an embodiment of the present disclosure. In some embodiments, the following Huffman encoding method is performed by a data encoding device, including steps 31-33.

[0074] In step 31, a flag bit of the i-th row of data is generated according to the number of leading zeros and the number of trailing zeros.

[0075] In some embodiments, a flag is generated for the i-th row of data to indicate that the XOR value is not zero, a flag is generated to indicate that the number of trailing zeros is less than or equal to the trailing zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag is generated to indicate that the number of leading zeros is less than the leading zero threshold.

[0076] In step 32, a Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched in a predetermined Huffman coding table.

[0077] For example, if the XOR value is 931, the Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is found to be 1 in the predetermined Huffman coding table.

[0078] In step 33, the encoding value of the i-th row of data is generated according to the flag bit of the i-th row of data and the Huffman encoding mapping value.

[0079] In some embodiments, a Huffman coding value of the i-th row of data is generated based on a flag indicating that the XOR value is not zero, a flag indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, a flag indicating that the number of leading zeros is less than the leading zero threshold, and a Huffman coding mapping value.

[0080] For example, the flag bit used to indicate that the XOR value is not zero is "1", the flag bit used to indicate that the number of trailing zeros is less than or equal to the trailing zero threshold and the number of leading zeros is not equal to the currently stored leading zero value is "1", the flag bit used to indicate that the number of leading zeros is less than the leading zero threshold is "1", and the Huffman coding mapping value is "1", then the Huffman coding value of the i-th row data is as shown in Expression (2).

[0081] 1+1+1+1 (2)

[0082] It should be noted that in this embodiment, since the XOR value is not within the specified range, it indicates that the corresponding i-th row of data is abnormal. In this case, to prevent the i-th row of data from affecting the encoding operation of subsequent data, the i-th row of data is not used as the current stored data, and the number of leading zeros in the i-th row of data is not used as the currently stored leading zero value.

[0083] In step 17, the encoded value of the i-th row of data is written into the cache.

[0084] In some embodiments, if the number of suffix zeros is greater than the suffix zero threshold, the following operations are performed.

[0085] 1) Generate the flag bit of the i-th row of data.

[0086] For example, a flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

[0087] 2) Generate the encoding value of the i-th row data based on the flag bit and the XOR value.

[0088] For example, the encoding value of the i-th row of data is generated based on the flag bit indicating that the XOR value is not zero, the flag bit indicating that no encoding is performed, and the XOR value.

[0089] For example, if the flag indicating that the XOR value is not zero is "1" and the flag indicating that no encoding is performed is "0", the encoding value of the i-th row of data is as shown in Expression (3).

[0090] 1+0+valid part of XOR value (3)

[0091] In addition, the data in the i-th row is used as the currently stored data, and the number of leading zeros in the data in the i-th row is used as the currently stored leading zero value for use in subsequent data processing.

[0092] In some embodiments, if the number of leading zeros is equal to the currently stored leading zero value, the following operations are performed.

[0093] 1) Generate the flag bit of the i-th row of data.

[0094] For example, a flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

[0095] 2) Generate the encoding value of the i-th row data based on the flag bit and the XOR value.

[0096] For example, the encoding value of the i-th row of data is generated based on the flag bit indicating that the XOR value is not zero, the flag bit indicating that no encoding is performed, and the XOR value.

[0097] For example, if the flag indicating that the XOR value is not zero is "1" and the flag indicating that no encoding is performed is "0", the encoding value of the i-th row of data is as shown in Expression (4).

[0098] 1+0+valid part of XOR value (4)

[0099] In addition, the data in the i-th row is used as the currently stored data, and the number of leading zeros in the data in the i-th row is used as the currently stored leading zero value for use in subsequent data processing.

[0100] In some embodiments, when the XOR value is zero, the following operations are performed.

[0101] 1) Generate a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is performed for the i-th row of data.

[0102] 2) Generate the code for the i-th row of data based on the flag bit indicating that the XOR value is zero and the flag bit indicating that no coding is required.

[0103] For example, the flag bit used to indicate that the XOR value is zero is "0", and the flag bit used to indicate that no encoding is performed is "0", then the encoding value of the i-th row data is as shown in Expression (5).

[0104] 0+0 (5)

[0105] In addition, the i-th row of data is used as the current stored data for use in subsequent data processing.

[0106] In the data encoding method provided in the above embodiment of the present disclosure, different encoding modes are selected according to the size of the XOR value of two data, thereby providing a higher compression ratio.

[0107] Figure 4 This is a structural diagram of a data encoding device according to an embodiment of the present disclosure.

[0108] like Figure 4 As shown, the data encoding apparatus 40 can be embodied in the form of a general-purpose computing device and includes a memory 41, a processor 42, and a bus 43 connecting different system components.

[0109] The memory 41 may include, for example, system memory, non-volatile storage media, and the like. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions corresponding to at least one embodiment of the data encoding method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, and flash memory.

[0110] Processor 42 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, or discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the acquisition module, the calculation module, and the adjustment module, can be implemented by a central processing unit (CPU) executing instructions stored in memory to execute the corresponding steps, or by dedicated circuits to execute the corresponding steps.

[0111] For example, the processor 42 is configured to execute instructions stored in the memory to implement the following Figure 1 The method according to any one of the embodiments.

[0112] The bus 43 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0113] The interfaces 44, 45, and 46 of the data encoding device 40, as well as the memory 41 and the processor 42, can be connected via a bus 43. The input / output interface 44 provides a connection interface for input / output devices such as a display, mouse, and keyboard. The network interface 45 provides a connection interface for various networked devices. The storage interface 46 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0114] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks, can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0116] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to operate in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0117] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0118] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the following Figures 1 to 3 The method according to any one of the embodiments.

[0119] The present disclosure also provides a computer program product, including computer instructions, wherein when the computer instructions are executed by a processor, the following is achieved: Figures 1 to 3 The method according to any one of the embodiments.

[0120] The present disclosure is described below through specific examples.

[0121] For example, in the data column to be encoded extracted from the data table, the first row of data is 75, the second row of data is 75, the third row of data is 1000, the fourth row of data is 68, the fifth row of data is 32, and the sixth row of data is 106. The following is the order from the first row of data to the sixth row of data, and the processing results are shown in Table 1.

[0122] 1. Layer 1 data: 75.

[0123] For layer 1 data, write it directly to the cache.

[0124] For example, the binary representation of 75 is 1001011, and the binary number 1001011 is written into the cache. At this time, the first layer data 75 is used as the current storage data.

[0125] Next, follow the Figure 5As shown in the flowchart, each data from the 2nd layer data to the 6th layer data is encoded in sequence.

[0126] 2. Layer 2 data: 75.

[0127] Since the second layer data and the current storage data are both 75, the XOR value of 75 and 75 is 0. In this case, Figure 2 As shown, the flag bits generated for the layer 2 data include: a flag bit 0 for indicating that the XOR value is zero; and a flag bit 0 for indicating that no encoding is performed.

[0128] Next, based on the flag bit 0 indicating that the XOR value is zero and the flag bit 0 indicating that no encoding is required, the encoding value of the second row of data is generated according to the above expression (5), and the encoding value 00 is written into the cache.

[0129] In addition, the data in the second row is used as the current stored data, that is, the current stored data is updated.

[0130] 3. Layer 3 data: 1000.

[0131] The third layer data is 1000, and the current stored data is 75. The XOR value of 75 and 1000 is 931, and the binary representation of 931 is 1110100011.

[0132] According to the XOR value, the number of leading zeros is 18 and the number of trailing zeros is 0.

[0133] Here, the suffix zero threshold is assumed to be 1. In this case, the flag bits generated for the third layer data include: flag bit 1 for indicating that the XOR value is not zero; flag bit 1 for indicating that the number of suffix zeros is less than or equal to the suffix zero threshold, and the number of leading zeros is not equal to the currently stored leading zero value; flag bit 1 for indicating that the number of leading zeros is less than the leading zero threshold.

[0134] Next, the Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched in the predetermined Huffman coding table. Thus, the Huffman coding value of the third row of data is generated based on the above flag bit and the Huffman coding mapping value.

[0135] For example, if the Huffman coding mapping value is 1, then according to the above flag bit Huffman coding mapping value, using the above expression (2), the Huffman coding value of the third row of data generated is 1111, and the coding value 1111 is written into the cache.

[0136] It should be noted that in this embodiment, since the XOR value 931 is not within the specified range, it indicates that the corresponding third row of data is abnormal. In this case, to prevent the third row of data from affecting the encoding operation of subsequent data, the third row of data is not used as the current stored data, and the number of leading zeros in the third row of data is not used as the currently stored leading zero value.

[0137] 4. Layer 4 data: 68.

[0138] The 4th layer data is 68, and the currently stored data is 75. The XOR value of 75 and 68 is 15, and the binary representation of 15 is 1111.

[0139] According to the XOR value, the number of leading zeros is 24 and the number of trailing zeros is 0.

[0140] Assume that the leading zero threshold is 21. In this case, the flag bits generated for the layer 4 data include: a flag bit 1 for indicating that the XOR value is not zero; a flag bit 1 for indicating that encoding is performed; and a flag bit 0 for indicating that the number of leading zeros is greater than or equal to the leading zero threshold.

[0141] In addition, according to the predetermined leading zero mapping table, the leading zero mapping value corresponding to the number of leading zeros 24 is 7, and the corresponding binary representation is 111. Therefore, based on the above flag bit, the leading zero mapping value and the XOR value, the XOR encoding value of the fourth row of data generated by the above expression (1) is 1101111111.

[0142] It should be noted that when generating the encoding value of the fourth row of data, the XOR value 1111 may be expanded as needed, for example, the XOR value 1111 may be expanded to 40 bits.

[0143] Next, the 4th row of data is used as the currently stored data, and the number of leading zeros in the 4th row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0144] 5. Layer 5 data: 32.

[0145] The fifth layer data is 32, and the current stored data is 68. The XOR value of 68 and 32 is 100, and the binary representation of 100 is 1100100.

[0146] According to the XOR value, the number of leading zeros is 21 and the number of trailing zeros is 2.

[0147] In this case, the flag bits generated for the layer 5 data include: a flag bit 1 for indicating that the XOR value is not zero; and a flag bit 0 for indicating that no encoding is performed.

[0148] Therefore, according to the above flag bit and the XOR value, using the above expression (3), the encoding value of the fifth row of data generated is 101100100.

[0149] It should be noted that when generating the encoding value of the fifth row of data, the XOR value 1100100 may be expanded as needed, for example, the XOR value 1100100 may be expanded to 40 bits.

[0150] Next, the fifth row of data is used as the currently stored data, and the number of leading zeros in the fifth row of data is used as the currently stored leading zero value for use in subsequent data processing.

[0151] 6. 6th layer data: 106.

[0152] The 6th layer data is 106, and the current stored data is 32. The XOR value of 32 and 106 is 74, and the binary representation of 74 is 1001010.

[0153] According to this XOR value, the number of leading zeros is 21, and the number of trailing zeros is 1. It should be noted that the number of leading zeros 21 in the XOR value 1001010 is equal to the currently stored leading zero value 21. In this case, the flag bits generated for the layer 6 data include: a flag bit 1 to indicate that the XOR value is not zero; and a flag bit 0 to indicate that no encoding is performed.

[0154] Therefore, according to the above flag bit and the XOR value, using the above expression (4), the encoding value of the 6th row of data generated is 101001010.

[0155] It should be noted that when generating the encoding value of the sixth row of data, the XOR value 1001010 may be expanded as needed, for example, the XOR value 1001010 may be expanded to 40 bits.

[0156] Next, the data in the sixth row is used as the currently stored data, and the number of leading zeros in the data in the sixth row is used as the currently stored leading zero value for use in subsequent data processing.

[0157] Table 1.

[0158] By implementing the above-mentioned embodiments of the present disclosure, the following beneficial effects can be obtained.

[0159] • If the differences between adjacent data points are small (e.g., sensor data with small variations), using XOR encoding can save significant space. Assuming the differences between data points are below a certain threshold, compression ratios of 50%-90% are possible.

[0160] • For data with uneven frequency distribution and large variations (such as time series data from a BMS (Battery Management System)), Huffman coding can often significantly improve compression efficiency, with compression ratios sometimes reaching 40%-70%.

[0161] • In scenarios where data contains both small differences (suitable for XOR encoding) and large differences (suitable for Huffman encoding), dynamically selecting the appropriate compression method can provide the best compression results. In practical applications, this hybrid approach may increase the overall data compression ratio by 20%-60%.

[0162] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof, for performing the functions described in the present disclosure.

[0163] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0164] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A data encoding method, performed by a data encoding device, comprising: Obtain a data column to be encoded from a data table, wherein the data column to be encoded includes multiple rows of data; Writing the first row of data in the to-be-encoded data column into a cache, and using the first row of data as currently stored data; Calculate the XOR value of the i-th row data and the currently stored data in the order from the 2nd row to the Nth row in the data column to be encoded, , N is the total number of rows; If the XOR value is not zero, determining whether the XOR value is within a specified range; If the XOR value is within a specified range, performing XOR encoding on the i-th row of data using the XOR value to obtain an encoding value of the i-th row of data; If the XOR value is not within the specified range, performing Huffman encoding on the i-th row of data using the XOR value to obtain an encoding value of the i-th row of data; Writing the encoded value of the i-th row of data into the cache; When the XOR value is zero, generating a flag bit for indicating that the XOR value is zero and a flag bit for indicating that no encoding is performed for the i-th row of data; Generate a code for the i-th row of data according to the flag bit for indicating that the XOR value is zero and the flag bit for indicating that no coding is required; Wherein, determining whether the XOR value is within a specified range includes: Determining the number of leading zeros and the number of trailing zeros of the XOR value; Determining whether the number of suffix zeros is less than or equal to a suffix zero threshold; If the number of trailing zeros is less than or equal to the trailing zero threshold, determining whether the number of leading zeros is equal to a currently stored leading zero value; If the number of leading zeros is not equal to the currently stored leading zero value, determining whether the number of leading zeros is less than a leading zero threshold; If the number of leading zeros is less than the leading zero threshold, determining that the XOR value is not within a specified range; If the number of leading zeros is greater than or equal to the leading zero threshold, it is determined that the XOR value is within a specified range.

2. The method according to claim 1, wherein The XOR encoding of the i-th row of data using the XOR value includes: Generate a flag bit of the i-th row of data according to the number of leading zeros and the number of trailing zeros; querying a leading zero mapping value corresponding to the number of leading zeros according to a predetermined leading zero mapping table; Generate an encoding value of the i-th row of data according to the flag bit of the i-th row of data, the leading zero mapping value and the XOR value.

3. The method according to claim 2, wherein: The generating of the flag bit of the i-th row of data comprises: A flag bit for indicating that the XOR value is not zero, a flag bit for indicating that encoding is performed, and a flag bit for indicating that the number of leading zeros is greater than or equal to the leading zero threshold are generated for the i-th row of data.

4. The method according to claim 3, wherein: Generating the encoding value of the i-th row of data includes: The XOR encoding value of the i-th row of data is generated according to the flag indicating that the XOR value is not zero, the flag indicating encoding, the flag indicating that the number of leading zeros is greater than or equal to the leading zero threshold, the leading zero mapping value, and the XOR value.

5. The method according to claim 2, further comprising: Taking the i-th row of data as the current stored data; The number of leading zeros is used as the currently stored leading zero value.

6. The method according to claim 1, wherein The performing Huffman encoding on the i-th row of data by using the XOR value comprises: Generate a flag bit of the i-th row of data according to the number of leading zeros and the number of trailing zeros; In a predetermined Huffman coding table, a Huffman coding mapping value corresponding to the occurrence frequency of the XOR value is searched; Generate a coding value of the i-th row of data according to the flag bit of the i-th row of data and the Huffman coding mapping value.

7. The method according to claim 6, wherein: The generating of the flag bit of the i-th row of data comprises: Generate a flag for indicating that the XOR value is not zero, a flag for indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, and a flag for indicating that the number of leading zeros is less than the leading zero threshold for the i-th row of data.

8. The method according to claim 7, wherein: Generating the Huffman coding value of the i-th row of data includes: The Huffman coding value of the i-th row of data is generated according to the flag indicating that the XOR value is not zero, the flag indicating that the number of suffix zeros is less than or equal to the suffix zero threshold and the number of leading zeros is not equal to the currently stored leading zero value, the flag indicating that the number of leading zeros is less than the leading zero threshold, and the Huffman coding mapping value.

9. The method according to claim 1, further comprising: If the number of trailing zeros is greater than the trailing zero threshold, or if the number of leading zeros is equal to the currently stored leading zero value, generating a flag bit for the i-th row of data; Generate the encoding value of the i-th row of data according to the flag bit and the XOR value.

10. The method according to claim 9, wherein: The generating of the flag bit of the i-th row of data comprises: A flag bit for indicating that the XOR value is not zero and a flag bit for indicating that no encoding is performed are generated for the i-th row of data.

11. The method according to claim 10, wherein: Generating the encoding value of the i-th row of data includes: The encoding value of the i-th row of data is generated according to the flag bit for indicating that the XOR value is not zero, the flag bit for indicating that no encoding is performed, and the XOR value.

12. The method according to claim 9, further comprising: Taking the i-th row of data as the current stored data; The number of leading zeros is used as the currently stored leading zero value.

13. The method according to claim 1, further comprising: The i-th row of data is used as the current storage data.

14. The method according to claim 1, further comprising: After obtaining the data column to be encoded from the data table, detecting whether each row in the data column to be encoded includes data; If the mth row does not contain data, the data of the adjacent row is filled into the mth row so as to update the data column to be encoded. .

15. The method according to any one of claims 1 to 14, wherein The data table can be divided into a plurality of data rows or a plurality of data columns, wherein the plurality of data included in each data row has the same time stamp, and the plurality of data included in each data column has the same format.

16. A data encoding device comprising: Memory; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 15 based on instructions stored in the memory.

17. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

18. A computer program product comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images