Vehicle-mounted log data compression storage method and device, vehicle, and log management system

Abstract

The present application relates to the technical field of log data storage, and discloses a vehicle-mounted log data compression storage method, device, vehicle and log management system. The method divides the log to be stored into blocks and reads byte by byte, queries a preset static Huffman coding table with byte values as indexes, obtains corresponding code words and lengths, then writes the code words into a bit buffer according to the lengths, splices into a compressed bit stream, and finally encapsulates the compressed bit stream into independent compression blocks containing block headers (recording effective bit lengths or padding bit numbers) and data bodies, and writes the independent compression blocks into a non-volatile storage medium. The present application effectively solves the problems of storage shortage and eMMC life consumption caused by large vehicle-mounted log quantity, and overcomes the defects of high computational complexity, poor real-time performance and difficult power-off recovery of existing algorithms in a low-resource environment.

Description

Technical Field

[0001] This invention relates to the field of log data storage technology, specifically to a method, apparatus, vehicle, and log management system for compressing and storing vehicle log data. Background Technology

[0002] With the development of intelligent vehicles, the amount of vehicle log data has surged, but existing storage solutions face the contradiction between storage pressure and resource constraints: traditional compression algorithms have high computational complexity, affecting real-time performance; variable-length encoded bitstream encapsulation lacks boundary management, which can easily lead to decoding errors; data is easily damaged and difficult to recover when there is an abnormal power outage; code table management has poor compatibility and high maintenance costs. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of this application is to provide a method, device, vehicle and log management system for compressing and storing vehicle log data. It aims to solve the problems of insufficient storage space due to the large amount of vehicle log data, rapid wear and tear of storage media such as eMMC, and high computational complexity, poor real-time performance and difficulty in power failure recovery of existing compression algorithms in low-resource vehicle environments.

[0004] In a first aspect, embodiments of this application provide a method for compressing and storing vehicle log data based on single-byte static Huffman coding, including:

[0005] Obtain a preset static Huffman coding table, wherein the static Huffman coding table includes Huffman codewords and codeword lengths corresponding to each of multiple single-byte symbols;

[0006] Divide the logs to be stored into log blocks to be compressed;

[0007] The log block to be compressed is read byte by byte, and the current byte value is used as an index to query the static Huffman coding table to obtain the corresponding Huffman codeword and codeword length.

[0008] The acquired Huffman codewords are written into the bit buffer according to the codeword length, so that multiple variable-length codewords are continuously concatenated to form a compressed bit stream.

[0009] The compressed bitstream is encapsulated into independent compressed log blocks and written to a non-volatile storage medium; the compressed log block includes a block header and a data body, and the block header records at least the effective bit length or the number of trailing padding bits.

[0010] In the above technical solution, by moving the complex Huffman tree construction process offline, only simple table lookup operations are required during vehicle operation, greatly reducing CPU computational overhead and memory usage, and meeting the real-time requirements of vehicle-mounted embedded systems. Log block processing divides the continuous log stream into independent compressed log blocks, achieving block-level storage management and fault isolation, preventing the impact of a single data block corruption on the overall log. By recording the effective bit length or the number of trailing padding bits in the block header, the boundaries of variable-length encoded bitstreams are precisely managed, ensuring accurate identification of the endpoint of valid data during decompression, avoiding decoding errors or data loss due to padding bits, and significantly improving the integrity and reliability of log data.

[0011] In one embodiment, obtaining the preset static Huffman coding table includes an offline table building process: collecting historical log samples from the target controller and converting them into byte streams, and counting the occurrence frequency of 256 single-byte symbols from 0x00 to 0xFF; if a single-byte symbol does not appear in the historical log samples, a preset minimum frequency is set for that single-byte symbol; based on the statistical results, a Huffman tree is constructed using standard Huffman coding or limited-length standard Huffman coding to generate the static Huffman coding table; when the codeword length generated by the initial Huffman tree exceeds the preset maximum code length limit, the code table is regenerated through code length rebalancing or standard code length adjustment.

[0012] In the above technical solution, offline collection of historical log samples and statistical analysis of byte frequencies ensure that the encoding table accurately reflects the actual log characteristics of the target controller, achieving targeted optimal compression. By setting a preset minimum frequency for unencoded byte symbols, the integrity of the encoding table is guaranteed, preventing compression failure due to unencoded symbols encountered during runtime. The use of standardized Huffman coding or limited-length standardized Huffman coding gives the codewords a standardized structure, simplifying decoding logic and improving decoding efficiency. Through code length rebalancing or standardized code length adjustment mechanisms, the maximum codeword length is effectively controlled, preventing buffer overflows or performance degradation due to excessively long codewords, ensuring stable and reliable operation of the encoding scheme in resource-constrained automotive environments.

[0013] In one embodiment, the step of writing the acquired Huffman codewords into a bit buffer according to the codeword length, so that multiple variable-length codewords are continuously concatenated to form a compressed bit stream, specifically includes: when the bit buffer is full of 8 bits, outputting a compressed byte as part of the data body; when the log block to be compressed ends and the last output byte is not full of 8 bits, padding bits are used to complete the block, and the number of end padding bits is recorded in the block header, the value of the end padding bits being in the range of 0 to 7; or, recording the total number of actual effective bits of the compressed bit stream in the block header as the effective bit length.

[0014] In the above technical solution, a bit buffer mechanism is used to achieve efficient conversion from variable-length codewords to byte-aligned data, making full use of storage space and avoiding space waste caused by byte alignment. By explicitly recording the number of trailing padding bits or the effective bit length in the block header, precise bit stream boundary information is provided to the decompression end, ensuring that the padding bits can be accurately truncated during decoding to restore the completely correct original data. This fine-grained bit-level management mechanism effectively solves the common boundary ambiguity problem of variable-length encoding, significantly improving the accuracy and reliability of compressed data decompression, while being compatible with different decompression implementations (based on padding bits or effective length), enhancing the flexibility of the solution.

[0015] In one embodiment, before encapsulating the compressed bitstream into an independent compressed log block, a compression benefit judgment step is further included: determining whether the length of the compressed data body plus the block header overhead is not less than the original bypass storage length, or whether the compression saving ratio is lower than a preset threshold; if so, the compression result is abandoned, the log block to be compressed is directly encapsulated in the original data manner, and marked as original bypass storage in the encoding type field of the block header; if not, the step of encapsulating the compressed bitstream into an independent compressed log block is performed, and marked as static Huffman compression in the encoding type field.

[0016] In the above technical solution, a compression benefit judgment mechanism is introduced to intelligently evaluate the actual effect of each compression, avoiding ineffective compression of high-entropy data or already compressed data, and preventing wasted storage space due to increased data volume after compression. By supporting the original bypass storage mode, the optimal storage strategy can be selected under any data characteristics, maximizing storage efficiency. By explicitly marking the encoding type in the block header, the decompression end can automatically identify and adopt the corresponding processing logic without additional configuration or manual intervention, improving the system's adaptability and ease of use, while ensuring optimal utilization of storage space.

[0017] In one embodiment, the header of the compressed log block further includes the following fields:

[0018] Starting marker, used for scanning and positioning;

[0019] The code table version number is used to match the corresponding decoding table when exporting and decompressing the log;

[0020] Original length, used to verify the length of the original log after decompression;

[0021] Storage length refers to the actual length of the stored data body;

[0022] Serial number and / or timestamp, used for sorting and fault time location;

[0023] The block header checksum and the data body checksum are used for integrity verification of the block header and the data body, respectively.

[0024] A commit flag is used to indicate whether the compressed log block is in an incomplete or complete state.

[0025] In the above technical solution, structured block header information provides rich metadata support for log management: the start identifier enables rapid scanning and location, improving log retrieval efficiency; the code table version number ensures correct decoding table matching during decompression, resolving compatibility issues between multiple software versions or different vehicle models; the original length and storage length fields support data integrity verification and storage space management; the serial number and timestamp facilitate log sorting and precise fault location, improving fault diagnosis efficiency; dual checksums (block header and data body) provide end-to-end data integrity protection, promptly detecting data corruption; and the commit flag provides status information for abnormal power outage recovery, ensuring system reliability. The overall design significantly improves the manageability, traceability, and robustness of log data.

[0026] In one embodiment, the encapsulation of the compressed bitstream into independent compressed log blocks and writing them to a non-volatile storage medium employs a two-phase commit and abnormal power-loss recovery mechanism. Specifically, this includes: first, writing a block header with the commit flag set to an incomplete state; then writing the data body and the data body checksum; and finally updating the commit flag to a complete state. If the commit flag is involved in the verification of the block header checksum, the commit flag and the block header checksum are updated synchronously. After system restart, the system scans the start identifier in the non-volatile storage medium to verify the block header checksum, the data body checksum, and the completion status of the commit flag. If a compressed log block fails verification or the commit flag remains in an incomplete state, it is marked as a corrupted block and skipped, and subsequent compressed log blocks are processed.

[0027] The above technical solution effectively addresses data corruption caused by abnormal power outages in the vehicle environment through a two-phase commit mechanism. Even if a power outage occurs during the write process, the system can quickly identify incomplete log blocks after restarting using commit flags and checksums, marking them as corrupted and skipping them to prevent corrupted log blocks from affecting the reading and parsing of subsequent normal logs. This mechanism ensures the atomicity and consistency of log storage, guaranteeing the recoverability of critical log data under extreme conditions. Simultaneously, block-level fault isolation prevents the entire log file from becoming unusable due to the corruption of a single log block, significantly improving the reliability and data security of the vehicle log system and meeting functional safety requirements.

[0028] Secondly, embodiments of this application provide an in-vehicle log data compression and storage device, which includes:

[0029] The encoding table acquisition module is used to acquire a preset static Huffman encoding table, which includes the Huffman codewords and codeword lengths corresponding to each of the multiple single-byte symbols.

[0030] The caching and segmentation module is used to divide the logs to be stored into log blocks to be compressed;

[0031] The lookup encoding module is used to read the log block to be compressed byte by byte, look up the static Huffman encoding table with the byte value as the index, and write the obtained Huffman codewords bit by bit into the bit buffer and concatenate them to form a compressed bit stream.

[0032] The encapsulation and storage module is used to encapsulate the compressed bit stream into an independent compressed log block containing a block header and a data body, and to record the effective bit length or the number of trailing padding bits in the block header and write it to a non-volatile storage medium.

[0033] In the above technical solution, the compressed storage function is divided into four independent modules through modular design: encoding table acquisition, caching and segmentation, table lookup encoding, and encapsulation and storage. This achieves functional decoupling and facilitates development, testing, and maintenance. Each module has a clear responsibility: the encoding table acquisition module is responsible for loading the pre-set encoding table; the caching and segmentation module implements log stream segmentation management; the table lookup encoding module performs efficient table lookup compression; and the encapsulation and storage module completes structured encapsulation and reliable storage. This design improves code reusability and scalability, facilitates rapid deployment and adaptation on different vehicle controllers, and supports modular verification in functional safety certification, reducing the complexity of engineering implementation.

[0034] Thirdly, embodiments of this application provide a vehicle, which includes: an on-board controller, the on-board controller including a processor and a non-volatile storage medium; a computer program is stored on the non-volatile storage medium, and when the computer program is executed by the processor, it implements the above-described on-board log data compression and storage method.

[0035] In the above technical solution, by integrating the compression storage method into the vehicle controller, the vehicle is equipped with efficient log compression and reliable storage capabilities, effectively reducing the storage space occupied by log data, reducing the number of times the storage medium is erased and rewritten, extending the service life of non-volatile storage media such as eMMC (Embedded MultiMediaCard) and UFS (Universal Flash Storage), and reducing the vehicle's total life cycle maintenance costs. At the same time, the lightweight compression scheme does not consume too many computing resources, ensuring that the real-time control tasks of the vehicle controller are not affected, improving the stability and reliability of the entire vehicle system, and providing a solid data foundation for fault diagnosis, remote monitoring, and data analysis of intelligent vehicles.

[0036] Fourthly, this application provides a vehicle log management system, which includes: the vehicle described above, and a cloud server or after-sales diagnostic device communicatively connected to the vehicle; the cloud server or after-sales diagnostic device is equipped with a decompression module, which is used to obtain the compressed log block issued by the vehicle and execute the vehicle log data decompression and restoration method to restore the original operating log of the vehicle.

[0037] In the above technical solution, by constructing a complete vehicle log management system, end-to-end collaboration between vehicle-side compressed storage and cloud / diagnostic decompression and restoration is achieved, supporting remote log collection, fault diagnosis, and data analysis. The cloud server or after-sales diagnostic equipment can accurately restore the vehicle logs through the decompression module. Combined with metadata such as the code table version number and timestamp in the block header, unified management and efficient analysis of logs from multiple vehicle models and versions are achieved. This system architecture supports log management throughout the entire vehicle lifecycle, providing data support for vehicle health monitoring, predictive maintenance, and algorithm iteration optimization, thereby improving vehicle operating efficiency and user service quality.

[0038] In one embodiment, the decompression and restoration method includes: reading the compressed log block, parsing the block header and performing integrity verification; determining the storage type based on the encoding type field in the block header: if it is original bypass storage, directly outputting the data body; if it is static Huffman compression, loading the corresponding decoding table according to the code table version number in the block header; reading the compressed bit stream in the data body according to the effective bit length or the number of trailing padding bits recorded in the block header, and restoring the original log byte by byte through a decoding tree or table lookup method; after decoding, verifying the output length using the original length field in the block header.

[0039] In the above technical solution, a complete decompression and restoration process ensures that compressed logs can be accurately and reliably restored to the original data: First, block header integrity verification promptly identifies and filters corrupted log blocks; the encoding type field automatically identifies the storage type, supporting unified processing of compressed and raw data; the code table version number accurately matches the decoding table, resolving multi-version compatibility issues; the effective bit length or the number of trailing padding bits precisely controls the decoding boundary, avoiding decoding errors; finally, original length verification ensures the integrity of the output data. The entire decompression process is rigorous and reliable, supports automated processing, requires no manual intervention, significantly improves the efficiency and accuracy of log analysis, and provides a high-quality data foundation for fault diagnosis and data mining.

[0040] Compared with the prior art, the present invention has the following advantages: By adopting single-byte static Huffman coding technology, the present invention moves the complex dynamic tree building process to the offline stage, and the vehicle terminal only needs to perform table lookup coding, which greatly reduces runtime resource consumption; at the same time, by designing a compressed log block structure that includes an effective bit length indicator, a two-phase commit mechanism and version management, highly reliable log storage and abnormal power failure recovery are achieved, effectively extending the service life of the vehicle storage medium. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application will be described below.

[0042] Figure 1 This is a flowchart illustrating the vehicle log data compression and storage method disclosed in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of the vehicle log data compression and storage device disclosed in the embodiments of this application.

[0044] Explanation of reference numerals in the attached figures:

[0045] 100 - Encoding table acquisition module, 200 - Cache and block segmentation module, 300 - Table lookup encoding module, 400 - Encapsulation and storage module. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In response to the surge in vehicle log data volume during the development of intelligent vehicles, and the contradiction between storage pressure and resource constraints faced by existing storage solutions (such as the high computational complexity of traditional compression algorithms affecting real-time performance, the lack of boundary management in variable-length encoding leading to decoding errors, data being easily damaged and difficult to recover during abnormal power outages, and poor code table management compatibility), this application provides a vehicle log data compression and storage method, device, vehicle, and log management system based on single-byte static Huffman coding.

[0048] See Figure 1 As shown, this embodiment discloses a method for compressing and storing vehicle log data based on single-byte static Huffman coding, mainly applied to the log storage module of vehicle controllers (such as ECUs, domain controllers, etc.). The method specifically includes the following steps:

[0049] Step S10: Obtain a preset static Huffman coding table, which includes the Huffman codewords and codeword lengths corresponding to multiple single-byte symbols (i.e., 256 byte values ​​from 0x00 to 0xFF). By moving the complex Huffman tree construction process offline, only a simple table lookup operation is required during vehicle operation, greatly reducing CPU (Central Processing Unit) computational overhead and memory consumption, fully meeting the stringent requirements of vehicle embedded systems for high real-time performance and low resource consumption.

[0050] Step S20: Divide the log to be stored into log blocks to be compressed: The continuously generated log data stream is divided into multiple independent log blocks to be compressed according to a preset block size or time window. By processing the log into blocks, block-level storage management and fault isolation are achieved, effectively preventing the overall log file availability from being affected by the corruption or write anomalies of a single data block.

[0051] Specifically, during vehicle controller operation, the log module writes logs to be stored into the RAM (Random Access Memory) cache. When the cache reaches a preset length, a preset time window is reached, a critical event occurs, or the system is about to lose power, a log block to be compressed is formed. The log block to be compressed can have a fixed maximum original length of 512 bytes, 1KB, 2KB, or 4KB, or it can be segmented based on event boundaries.

[0052] Step S30: Read the log block to be compressed byte by byte and perform table lookup encoding: Read the log block to be compressed byte by byte, and use the current byte value as an index to query the static Huffman coding table to obtain the corresponding Huffman codeword and codeword length.

[0053] Step S40: Concatenate to form a compressed bit stream: Write the acquired Huffman codewords into the bit buffer according to the codeword length, so that multiple variable-length codewords are concatenated to form a compressed bit stream.

[0054] Step S50: Encapsulate and write to non-volatile storage medium: Encapsulate the compressed bitstream into independent compressed log blocks and write them to non-volatile storage medium, such as eMMC or UFS. The compressed log block includes a block header and a data body. The block header records at least the effective bit length or the number of trailing padding bits. By recording the effective bit length or the number of trailing padding bits in the block header, the boundaries of the variable-length encoded bitstream are precisely managed, ensuring accurate identification of the endpoint of valid data during decompression. This avoids decoding errors or data loss due to padding bits, significantly improving the integrity and reliability of the log data.

[0055] In an optional embodiment, the static Huffman coding table obtained in step S10 above is generated through an offline table creation process, specifically including:

[0056] Sample collection and statistics: Collect historical log samples of the target controller and convert them into byte streams, and count the frequency of occurrence of 256 single-byte symbols from 0x00 to 0xFF.

[0057] Minimum Frequency Guarantee: If a single-byte symbol does not appear in the historical log samples, a preset minimum frequency is set for that single-byte symbol, such as a frequency of 1. This ensures the integrity of the encoding table and avoids compression failure or system crashes caused by encountering unencoded rare symbols at runtime.

[0058] Constructing the Huffman Tree: Based on the statistical results, a Huffman tree is constructed using canonical Huffman coding or limited-length canonical Huffman coding to generate the static Huffman coding table. The standardized codeword structure can significantly simplify the decoding logic and improve decoding efficiency.

[0059] Code length rebalancing: When the codeword length generated by the initial Huffman tree exceeds the preset maximum code length limit, such as limiting the maximum code length to 15 or 16 bits, the code table is regenerated through code length rebalancing (such as the Package-Merge algorithm) or a standard code length adjustment method.

[0060] This offline table creation mechanism ensures that the encoding table can accurately reflect the actual log characteristics of the target controller, achieving targeted optimal compression; at the same time, it effectively controls the maximum codeword length to prevent buffer overflow or decoding performance degradation due to excessively long codewords, ensuring that the encoding scheme operates stably and reliably in resource-constrained vehicle environments.

[0061] In an optional embodiment, the specific process of splicing to form a compressed bitstream in step S40 above includes: when the bit buffer is full (8 bits), outputting a compressed byte as part of the data body. When the log block to be compressed ends and the last output byte is less than 8 bits, padding bits (such as zeros) are used to complete the block.

[0062] The number of trailing padding bits is recorded in the block header, and the number of trailing padding bits ranges from 0 to 7. Alternatively, as an alternative, the total number of actual valid bits in the compressed bit stream is directly recorded in the block header as the valid bit length.

[0063] Specifically, for each input byte B in the log block to be compressed, the byte value B is used as an index to query the pre-constructed Huffman codeword table CodeTable

[256] and codeword length table LenTable

[256] to obtain the Huffman codeword Code[B] and its codeword length Len[B] corresponding to the byte. Then, Code[B] is written into the bit buffer according to the number of bits specified by Len[B], so that multiple variable-length codewords are continuously concatenated in the bit buffer to gradually form a compressed bit stream. In this process, whenever the number of bits accumulated in the bit buffer reaches 8 bits, a complete compressed byte is output as part of the data body; when the log block encoding ends and the number of bits remaining in the bit buffer is less than 8 bits, padding bits (such as 0) are used at the end to make it a complete byte, and at least one of the valid bit length ValidBitLength and the number of padding bits at the end PadBits is recorded in the block header; preferably, the number of padding bits at the end PadBits is recorded, and its value range is 0 to 7.

[0064] This embodiment achieves efficient conversion from variable-length codewords to byte-aligned data through a bit buffer mechanism, making full use of storage space. The refined bit-level management mechanism effectively solves the common boundary ambiguity problem in variable-length encoding, while being compatible with different decompression implementations based on padding bits or effective length, enhancing the flexibility of the solution.

[0065] Furthermore, this solution eliminates the need for dynamically constructing Huffman trees on the vehicle-mounted side, as well as maintaining complex data structures such as LZ dictionaries, sliding windows, template parsers, or token matching tables. The compression process involves only basic operations such as single-byte table lookups, bitwise shifts, bitwise OR buffer writes, and CRC calculations, resulting in extremely low computational and memory overhead. This facilitates efficient deployment in various vehicle-mounted software environments, including AUTOSAR Classic, RTOS, Linux, and bare metal. Simultaneously, since the input processing unit is fixed at a single byte, its interface is highly consistent with conventional byte stream writing. This not only allows for seamless integration into existing logging modules but also significantly improves its portability across different controller hardware platforms.

[0066] In an optional embodiment, a compression benefit determination step is included before encapsulation in step S50:

[0067] Determine whether the length of the compressed data body plus the block header overhead is not less than the original bypass storage length, or whether the compression saving ratio is lower than a preset threshold. For example, the preset threshold is 10%.

[0068] If compression yields no benefit, the compression result is abandoned, and the log block to be compressed is directly encapsulated as raw data, and marked as "raw bypass storage" in the Encode Type field of the block header.

[0069] If not, i.e. compression is beneficial, then the compression and encapsulation step is performed, and the encoding type is marked as "static Huffman compression" in the Encode Type field.

[0070] This embodiment intelligently evaluates the actual effect of each compression, avoiding ineffective compression of high-entropy data (such as encrypted or compressed data) and preventing data volume from increasing instead of decreasing. By supporting the original bypass storage mode, it ensures that the optimal storage strategy can be selected under any data characteristics, maximizing storage efficiency.

[0071] In an optional embodiment, the header of the compressed log block further includes the following key fields:

[0072] Start-up identifier (Magic Number): Used for quickly scanning and locating log blocks in the storage medium.

[0073] Table Version: Used to match the corresponding decoding table during log export and decompression, resolving compatibility issues between multiple software versions or different vehicle models.

[0074] Raw Length: Used to verify the length of the raw log after decompression, ensuring data integrity.

[0075] Stored Length: Represents the actual length of the stored data body, used for storage space management.

[0076] Sequence number and / or timestamp: used for log sorting and precise location of fault time, improving fault diagnosis efficiency.

[0077] Header CRC and Data CRC: These are used to verify the integrity of the block header and data body, respectively, providing end-to-end data protection.

[0078] Commit Flags: Used to indicate whether the compressed log block is in an "incomplete" or "complete" state.

[0079] In this embodiment, the structured block header design provides rich metadata support for log management, significantly improving the manageability, traceability and robustness of log data.

[0080] In an optional embodiment, when writing to the non-volatile storage medium in step S50, a two-phase commit and abnormal power-loss recovery mechanism is employed:

[0081] Phase 1: First, write the block header with the commit flag set to "incomplete".

[0082] Second stage: Then write the data body and the data body verification code.

[0083] Submit Update: Finally, update the submit flag to the "complete" status; if the submit flag is used in the block header checksum verification, then update the submit flag and the block header checksum simultaneously.

[0084] Power failure recovery: After the system restarts, it scans the start identifier in the non-volatile storage medium to verify the completion status of the block header checksum, the data body checksum, and the commit flag. If a compressed log block fails to be verified or the commit flag is still in the "incomplete" state, it is marked as a corrupted block and skipped, and subsequent compressed log blocks are processed.

[0085] This embodiment effectively solves the problem of data corruption caused by abnormal power loss in the vehicle environment, ensures the atomicity and consistency of log storage, and avoids the entire log file becoming unusable due to the damage of a single log block through block-level fault isolation, significantly improving the reliability of the vehicle log system and meeting automotive-grade functional safety requirements.

[0086] Specifically, the compressed log blocks can be written to non-volatile storage media such as Flash, EEPROM, eMMC, UFS, or SD cards. In terms of storage management strategy, a ring buffer is preferably used as the log storage area, with new compressed log blocks appended sequentially. When the storage area is full, historical ordinary logs that meet aging conditions (such as timeout or exceeding the retention threshold) are prioritized for overwriting. Furthermore, for high-fault-level logs, OTA upgrade failure logs, logs from critical periods before and after abnormal resets, and logs strongly correlated with Diagnostic Trouble Codes (DTCs), the system can assign protection flags, allowing for delayed overwriting or direct writing to a separate protected storage area to ensure reliable retention of critical diagnostic data.

[0087] For in-vehicle devices such as T-BOXs, cockpit domain controllers, or central computing units equipped with eMMC, this method significantly reduces the actual number of bytes written to a single log block through efficient compression. Furthermore, by using block-level independent encapsulation, it completely avoids the problems of full-file recompression and invalid rewriting caused by partial updates in traditional solutions. Under the same log collection frequency and retention strategy, the amount of physical data written to the eMMC is greatly reduced, thereby effectively alleviating the burden of the P / E cycle, garbage collection frequency, and write amplification effect within the eMMC. This has significant benefits in delaying the eMMC's lifespan and extending its effective service life.

[0088] At the underlying implementation level, to perfectly adapt to the aforementioned two-phase commit and power-loss prevention mechanism, this invention features targeted designs for the block header field and verification logic. After writing the data body and DataCRC, the system performs the final commit operation: if the commit flag (CommitFlag) is included in the calculation range of the block header checksum (HeaderCRC), then at this stage, CommitFlag and HeaderCRC are updated synchronously and written back; if an independent commit flag design is used, the calculation range of HeaderCRC does not cover this variable flag, thus allowing the CommitFlag state to be directly flipped without changing the HeaderCRC. During system restart and recovery, the log block is quickly located by scanning the start identifier (Magic Number), and the completion status of HeaderCRC, DataCRC, and CommitFlag is strictly verified. Any log block that fails verification or is in an "incomplete" state is isolated, marked as a corrupted block, and skipped, ensuring that a single block anomaly will never affect the parsing and recovery of subsequent normal log blocks.

[0089] Based on the same inventive concept, this application also provides an in-vehicle log data compression and storage device, see [link to relevant documentation]. Figure 2 As shown, the device can be integrated into an onboard controller. The device includes:

[0090] Encoding table acquisition module 100: used to acquire a preset static Huffman encoding table, wherein the static Huffman encoding table includes Huffman codewords and codeword lengths corresponding to multiple single-byte symbols.

[0091] Cache and Blocking Module 200: Used to divide logs to be stored into log blocks to be compressed.

[0092] The lookup encoding module 300 is used to read the log block to be compressed byte by byte, look up the static Huffman coding table with the byte value as the index, and write the obtained Huffman codewords bit by bit into the bit buffer and concatenate them to form a compressed bit stream.

[0093] Encapsulation and storage module 400: used to encapsulate the compressed bit stream into an independent compressed log block containing a block header and a data body, and record the effective bit length or the number of end padding bits in the block header, and write it to a non-volatile storage medium.

[0094] This embodiment achieves functional decoupling through modular design, with clear responsibilities for each module, improving code reusability and scalability, facilitating rapid deployment and adaptation on different vehicle controllers, and supporting modular verification in functional safety certification, thus reducing the complexity of engineering implementation.

[0095] This application also provides a vehicle, which includes: an on-board controller, the on-board controller including a processor and a non-volatile storage medium; the non-volatile storage medium stores a computer program, and when the computer program is executed by the processor, it implements the on-board log data compression and storage method described in any of the above embodiments.

[0096] For example, the processor includes MCU, SOC, etc., and the non-volatile storage medium includes eMMC, UFS, etc.

[0097] This embodiment integrates the above-mentioned methods into the vehicle controller, enabling the vehicle to possess efficient log compression and reliable storage capabilities. It effectively reduces the storage space occupied by log data, decreases the number of write / erase cycles for flash memory media such as eMMC / UFS, thereby significantly extending the lifespan of the storage media and reducing the vehicle's total lifecycle maintenance costs. Simultaneously, the lightweight compression scheme does not consume excessive computing resources, ensuring that the real-time control tasks of the vehicle controller are not affected, and providing a solid data foundation for fault diagnosis, remote monitoring, and data analysis in intelligent vehicles.

[0098] It should be noted that the vehicles can be, but are not limited to, pure electric vehicles (PEV / BEV), hybrid electric vehicles (HEV), range-extended electric vehicles (REEV), plug-in hybrid electric vehicles (PHEV), new energy vehicles, and gasoline vehicles.

[0099] This application also provides a vehicle log management system, comprising: the aforementioned vehicle, and a cloud server or after-sales diagnostic device communicatively connected to the vehicle; the cloud server or after-sales diagnostic device is equipped with a decompression module, which is used to obtain the compressed log block issued by the vehicle and execute an on-board log data decompression and restoration method to restore the original operating log of the vehicle. This achieves end-to-end collaboration between vehicle-side compressed storage and cloud / diagnostic-side decompression and restoration, supports unified management and efficient analysis of logs from multiple vehicle models and versions, and provides data support for vehicle health monitoring, predictive maintenance, and algorithm iteration optimization.

[0100] In an optional embodiment, the decompression and restoration method specifically includes the following steps:

[0101] Read and Verify: Read the compressed log block, parse the block header, and perform integrity verification (verify the block header checksum). If the verification fails, discard the block.

[0102] Type determination: Determine the storage type based on the encoding type field in the block header: if it is "raw bypass storage", then output the data body directly; if it is "static Huffman compression", then proceed to the decoding process.

[0103] Load the decoding table: Based on the code table version number in the block header, load the corresponding decoding table from the local machine or the cloud (or build the decoding tree).

[0104] Byte-by-byte restoration: According to the effective bit length or the number of end padding bits recorded in the block header, read the compressed bit stream in the data body, restore the original log byte by byte through decoding tree or lookup table, and stop decoding when the effective boundary is reached.

[0105] Length verification: After decoding, the length of the output data is verified using the original length field in the block header to ensure the absolute integrity of the restored data.

[0106] Specifically, when diagnostic tools, production line equipment, or vehicle-mounted devices read compressed log blocks, they first parse the block header field and verify the header checksum (HeaderCRC). If the verification fails, the block is discarded and an error log is recorded, and subsequent data blocks are processed. If the verification passes, the storage type is determined based on the encoding type field (EncodeType): when EncodeType is Raw (raw bypass storage), the data body is directly output as the raw log without decoding; when EncodeType is StaticHuffman (static Huffman compression), the corresponding decoding table is loaded from local storage or a cloud server based on the code table version number (TableVersion), and the corresponding decoding tree structure is constructed. Subsequently, the reading range of the bit stream is precisely controlled according to the valid bit length (ValidBitLength) or the number of padding bits (PadBits) recorded in the block header. The original log data is restored byte by byte through decoding tree traversal or fast table lookup, and decoding stops immediately when the valid bit boundary is reached to avoid out-of-bounds reading or mis-parsed padding bits. After decoding, the actual length of the output data is verified using the RawLength field in the block header to ensure the absolute integrity of the restored data. If the length verification fails, the block is marked as corrupted and an alarm mechanism is triggered.

[0107] The entire decompression process is rigorous and reliable, supporting fully automated processing without manual intervention. Through multiple verification mechanisms, including block header verification, type determination, decoding table version matching, effective boundary control, and length verification, the accuracy and robustness of the decoding process are ensured, significantly improving the efficiency and quality of log analysis and providing a highly reliable data foundation for fault diagnosis, data mining, and system optimization.

[0108] The following analysis and explanation will be based on specific examples.

[0109] Example 1: T-BOX Log Compression and eMMC Lifespan Optimization for Vehicle-Mounted Remote Communication Terminals: In this example, considering the large volume and frequent writes of T-BOX log data, static Huffman compression is used to optimize eMMC lifespan. First, during the offline phase, massive historical logs from the T-BOX are collected in core business scenarios such as network registration, eSIM configuration, APN connection, GNSS positioning, and OTA upgrades. The frequency of occurrence of 256 bytes (0x00~0xFF) is statistically analyzed, a static Huffman tree is constructed, and a global static code table is generated. During T-BOX operation, the system uses a fixed-size (e.g., 2KB) raw data as a log block. The raw log is efficiently converted into a compressed bitstream using a single-byte table lookup method, and a block header containing the code table version number (TableVersion) and a CRC checksum are appended for block-level encapsulation. The encapsulated compressed log blocks are sequentially written to the eMMC's ring buffer. Because each log block is compressed at a high ratio before being written, the amount of physical data written to the eMMC is significantly reduced under the same log collection frequency and retention strategy. This significantly reduces the erase / write burden and write amplification effect within the eMMC, effectively delaying the eMMC's lifespan. During after-sales diagnostics, maintenance tools only need to parse the block header to obtain the TableVersion and load the corresponding static code table to accurately and losslessly restore the original logs for engineers to analyze.

[0110] Example 2: Minimalist Log Compression for Low-Resource Electronic Control Units (ECUs): In this example, a minimally simplified compression architecture is adopted for low-end ECUs with limited RAM and computing resources, such as gateways and body controllers. Only two lightweight mapping tables are fixed in the vehicle-mounted Flash: the code value table (CodeTable

[256] ) and the code length table (LenTable

[256] ). There is no need to dynamically build a Huffman tree in memory, nor is it necessary to maintain complex data structures such as LZ dictionaries, sliding windows, template parsers, or token matching tables. When the system's log cache reaches a set threshold (such as 512 bytes), the compression task is triggered. The compression process only performs basic single-byte table lookup, bit shifting, and bitwise OR operations to generate a compressed bit stream and record the number of padding bits at the end. Thanks to its extremely low computational and memory overhead, this compression task can be flexibly configured to be executed in the low-priority periodic tasks of the RTOS without interfering with hard real-time tasks such as vehicle control. At the same time, its single-byte input interface is highly consistent with the standard byte stream writing, which can seamlessly reuse the original log writing interface and achieve zero-intrusion integration in low-resource environments.

[0111] Example 3: Power Loss Recovery for Automotive-Grade Functional Safety: In this example, for scenarios involving abnormal power loss during vehicle operation (such as momentary battery failure or collision-induced power loss), block-level isolation and a two-phase commit mechanism are employed to ensure the reliability of log data. Assuming the vehicle experiences a sudden power loss while writing a compressed log block, after the system restarts, the log management module will automatically scan the log area in the non-volatile storage medium. If a log block's header checksum (HeaderCRC) or data body checksum (DataCRC) fails verification, or its commit flag (CommitFlag) is still in an "incomplete" state, the system will strictly adhere to the block-level isolation principle, marking the abnormal block as a "corrupted block" and skipping it, without causing the entire log file to crash. Meanwhile, normal log blocks that have already completed the two-phase commit can still be decompressed and read normally based on their TableVersion in the block header and their valid bit length (ValidBitLength) or padding bit count (PadBits). This mechanism ensures that a single-block failure never affects the whole system, preserves critical diagnostic logs prior to the failure to the maximum extent, and fully meets the stringent requirements of automotive-grade functional safety (such as ISO 26262) for data consistency and system robustness.

[0112] Compared with existing technologies, the static Huffman compression, storage, and decompression / restore method for vehicle logs provided by this invention achieves significant benefits in multiple dimensions, including resource overhead, storage lifespan, data reliability, and engineering feasibility, through the deep integration of byte-level streaming encoding, block-level independent encapsulation, and a two-phase commit mechanism. Specifically, these benefits are reflected in the following aspects:

[0113] First, the extremely lightweight design significantly reduces the resource consumption of the vehicle-mounted system, simplifying engineering implementation. Compared to traditional LZ-type dynamic dictionary, field-level, or template-level compression schemes, this invention adopts a single-byte streaming processing architecture. At runtime, it only needs to maintain 256 static mapping tables (CodeTable and LenTable), eliminating the need for dynamically constructing Huffman trees, sliding windows, business dictionaries, longest matching rules, or complex template parsers. Its encoding entry point is merely the most basic byte stream, resulting in extremely low computational and memory overhead. This not only allows compression tasks to be smoothly executed in low-priority cyclic tasks on low-resource ECUs (such as gateways and body controllers) without interfering with hard real-time control logic, but also significantly reduces the integration difficulty and testing costs of mass-production software, perfectly meeting the automotive-grade mass production requirements of T-BOX and safety-related controllers.

[0114] Secondly, this invention significantly reduces the total amount of data written to non-volatile storage media, effectively extending the lifespan of eMMC / Flash hardware. By allocating short codes to high-frequency bytes, this invention drastically reduces the actual number of physical bytes written to disk for logs, effectively extending the retention period of local logs under the same log collection frequency and retention strategy. More importantly, for vehicle controllers (such as cockpit domain controllers and central computing units) that write logs to eMMC for extended periods, the sharp reduction in the amount of data written to disk directly reduces the total volume written to eMMC by log services (TBW), thereby significantly alleviating the burden of block erase / write (P / E cycle), write amplification effect, and garbage collection (GC) trigger frequency within the eMMC. This has extremely significant economic and engineering value in delaying eMMC lifespan consumption and extending its effective service life.

[0115] Third, the innovative block-level encapsulation and boundary control mechanism completely solves the problems of variable-length code disk write-to-disk and data recovery. This invention stores the compression results in the form of independent log blocks. Each block header precisely records the code table version number (TableVersion), the valid bit length (ValidBitLength) or the number of padding bits (PadBits, values ​​0 to 7), and a double CRC checksum. This design not only completely solves the byte alignment problem when variable-length Huffman bit streams are written to disk, enabling the decoding end to accurately identify the end padding bits and avoid misinterpretations, but also achieves strict physical isolation of single-block failures—even if a single log block is damaged due to abnormal power loss or media aging, it will never trigger an "avalanche effect" that renders all subsequent logs unusable, greatly improving the recoverability and data integrity of the compressed log.

[0116] Fourth, it possesses excellent multi-version compatibility and universal applicability across all scenarios. In terms of universality, the single-byte encoding scheme naturally covers all byte values ​​from 0x00 to 0xFF, handling ASCII, UTF-8, hexadecimal text, and mixed logs and binary structured logs uniformly. When log formats or module fields change, there is usually no need to synchronously modify the underlying parsing rules. For data blocks with high randomness or already compressed, the system can automatically trigger bypass storage (Raw mode) based on benefit assessment, ensuring that the compression ratio always has a positive benefit. Regarding compatibility, by embedding the TableVersion field in the block header, after-sales diagnostic tools or cloud platforms can accurately load the corresponding decoding table based on the vehicle model, controller hardware, or software version, achieving seamless compatibility and smooth upgrades across vehicle models and generations of software versions.

[0117] In summary, this invention achieves efficient compression and secure storage of vehicle logs with extremely low system resource overhead. It demonstrates outstanding technical advantages and broad industrialization prospects in improving log analysis efficiency, protecting the lifespan of vehicle storage hardware, and meeting automotive-grade functional safety requirements.

[0118] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for compressing and storing vehicle log data, characterized in that, include: Obtain a preset static Huffman coding table, wherein the static Huffman coding table includes Huffman codewords and codeword lengths corresponding to each of multiple single-byte symbols; Divide the logs to be stored into log blocks to be compressed; The log block to be compressed is read byte by byte, and the current byte value is used as an index to query the static Huffman coding table to obtain the corresponding Huffman codeword and codeword length. The acquired Huffman codewords are written into the bit buffer according to the codeword length, so that multiple variable-length codewords are continuously concatenated to form a compressed bit stream. The compressed bitstream is encapsulated into independent compressed log blocks and written to a non-volatile storage medium; the compressed log block includes a block header and a data body, and the block header records at least the effective bit length or the number of trailing padding bits.

2. The vehicle log data compression and storage method according to claim 1, characterized in that, The process of obtaining the preset static Huffman coding table includes an offline table creation process: Collect historical log samples from the target controller and convert them into byte streams. Count the frequency of occurrence of 256 single-byte symbols from 0x00 to 0xFF. If a single-byte symbol does not appear in the historical log samples, set a preset minimum frequency for that single-byte symbol. Based on the statistical results, a Huffman tree is constructed using standard Huffman coding or limited-length standard Huffman coding to generate the static Huffman coding table; when the codeword length generated by the initial Huffman tree exceeds the preset maximum code length limit, the code table is regenerated through code length rebalancing or standard code length adjustment.

3. The method for compressing and storing vehicle log data according to claim 1, characterized in that, The step of writing the acquired Huffman codewords into a bit buffer according to the codeword length, and then concatenating multiple variable-length codewords to form a compressed bit stream, specifically includes: When the bit buffer is full (8 bits), a compressed byte is output as part of the data body. When the log block to be compressed ends and the last output byte is less than 8 bits, padding bits are used to complete the block, and the number of padding bits at the end is recorded in the block header. The value of the number of padding bits at the end ranges from 0 to 7. Alternatively, the total number of actual effective bits of the compressed bit stream is recorded in the block header as the effective bit length.

4. The method for compressing and storing vehicle log data according to claim 1, characterized in that, Before encapsulating the compressed bitstream into independent compressed log blocks, a compression benefit judgment step is also included: judging whether the length of the compressed data body plus the block header overhead is not less than the original bypass storage length, or whether the compression saving ratio is lower than a preset threshold. If so, the compression result is discarded, and the log block to be compressed is directly encapsulated in the original data format, and marked as original bypass storage in the encoding type field of the block header; If not, then perform the step of encapsulating the compressed bitstream into independent compressed log blocks and mark it as static Huffman compression in the encoding type field.

5. The method for compressing and storing vehicle log data according to claim 1, characterized in that, The header of the compressed log block also Includes the following fields: Starting marker, used for scanning and positioning; The code table version number is used to match the corresponding decoding table when exporting and decompressing the log; Original length, used to verify the length of the original log after decompression; Storage length refers to the actual length of the stored data body; Serial number and / or timestamp, used for sorting and fault time location; The block header checksum and the data body checksum are used for integrity verification of the block header and the data body, respectively. A commit flag is used to indicate whether the compressed log block is in an incomplete or complete state.

6. The method for compressing and storing vehicle log data according to claim 1, characterized in that, The process of encapsulating the compressed bitstream into independent compressed log blocks and writing them to non-volatile storage media employs a two-phase commit and abnormal power-loss recovery mechanism, specifically including: First, write a block header with the commit flag set to incomplete. Then, write the data body and the data body checksum. Finally, update the commit flag to complete. If the commit flag is used to verify the block header checksum, then update the commit flag and the block header checksum simultaneously. After the system restarts, it scans the start identifier in the non-volatile storage medium to verify the completion status of the block header checksum, the data body checksum, and the commit flag. If a compressed log block fails to be verified or the commit flag is still incomplete, it is marked as a corrupted block and skipped, and subsequent compressed log blocks are processed.

7. A vehicle-mounted log data compression and storage device, characterized in that, include: The encoding table acquisition module is used to acquire a preset static Huffman encoding table, which includes the Huffman codewords and codeword lengths corresponding to each of the multiple single-byte symbols. The caching and segmentation module is used to divide the logs to be stored into log blocks to be compressed; The lookup encoding module is used to read the log block to be compressed byte by byte, look up the static Huffman encoding table with the byte value as the index, and write the obtained Huffman codewords bit by bit into the bit buffer and concatenate them to form a compressed bit stream. The encapsulation and storage module is used to encapsulate the compressed bit stream into an independent compressed log block containing a block header and a data body, and to record the effective bit length or the number of trailing padding bits in the block header and write it to a non-volatile storage medium.

8. A vehicle, characterized in that, include: An on-board controller, the on-board controller including a processor and a non-volatile storage medium; The non-volatile storage medium stores a computer program, which, when executed by the processor, implements the vehicle log data compression and storage method as described in any one of claims 1 to 6.

9. A log management system, characterized in that, include: The vehicle as described in claim 8, and a cloud server or after-sales diagnostic device communicatively connected to the vehicle; The cloud server or after-sales diagnostic equipment is equipped with a decompression module. The decompression module is used to obtain the compressed log block issued by the vehicle and execute the vehicle log data decompression and restoration method to restore the original operating log of the vehicle.

10. The log management system according to claim 9, characterized in that, The decompression and restoration method includes: reading the compressed log block, parsing the block header and performing integrity verification; The storage type is determined based on the encoding type field in the block header: if it is raw bypass storage, the data body is directly output; if it is static Huffman compression, the corresponding decoding table is loaded according to the code table version number in the block header. According to the effective bit length or the number of end padding bits recorded in the block header, read the compressed bit stream in the data body, and restore the original log byte by byte by decoding tree or lookup table; After decoding is complete, the output length is verified using the original length field in the block header.

Images