Data compression method, apparatus, device, chip, vehicle, medium, and program product

Abstract

The present disclosure provides a data compression method, an apparatus, a device, a chip, a vehicle, a medium, and a program product. The method comprises: using a CAN identifier to parse CAN packet data to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table; extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the data types, combining the data columns corresponding to the data types into a signal column, and extracting timestamps from the timestamp mapping table to generate a timestamp column; coding the signal column and the timestamp column as intermediate coded data, and compressing the intermediate coded data into intermediate compressed data; and combining the intermediate compressed data into a columnar compressed file. The technical solution provided in the present disclosure solves the problems of insufficient structuring and low compression rate of data compression of packet data, thereby improving the structuring degree and the compression rate of data compression.

Description

Data compression method, device, equipment, chip, vehicle, medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application No. 2024101909477 filed on February 20, 2024, filed by Beijing Rockwell Technology Co., Ltd., and entitled “A data compression method, device, electronic device, chip and medium,” the full text of which is incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of big data technology, and in particular to a data compression method, apparatus, device, chip, vehicle, medium, and program product. Background Art

[0004] In the big data world, the compression process for Controller Area Network (CAN) messages involves first compressing the raw data into a binary file, then further compressing the binary file using zip compression. While this process achieves compression of the raw data, the data compression scheme lacks structure and the compression rate is low. Summary of the Invention

[0005] The present disclosure provides a data compression method, apparatus, device, chip, vehicle, medium and program product, which improves the structured degree and compression rate of data compression by table building, encoding, compressing and combining CAN message data into columnar storage files that support full cloud parsing.

[0006] The present disclosure first provides a data compression method, which includes:

[0007] Use CAN identifiers to parse CAN message data and obtain signal value mapping table, identifier mapping table and timestamp mapping table;

[0008] Extract different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, merge the data columns corresponding to each data type into a signal column, and extract timestamps from the timestamp mapping table to generate a timestamp column;

[0009] The coded signal column and the time stamp column are intermediate coded data and compressed into intermediate compressed data;

[0010] Combine intermediate compressed data into columnar compressed files.

[0011] In some embodiments, CAN message data is parsed using a CAN identifier to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table, including:

[0012] Use CAN identifiers to parse CAN message data and build a signal value mapping table based on the CAN message data;

[0013] If the actual data amount corresponding to the CAN identifier in the signal value mapping table is less than the CAN memory, the message signal value corresponding to the CAN identifier is stored in the multiplexed message memory, and the signal value mapping table is updated; wherein the CAN memory is the memory allocated to the data corresponding to the CAN identifier; and the message memory is the memory for the message signal value;

[0014] If the actual data volume is greater than or equal to the CAN memory, a new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated;

[0015] The updated signal value mapping table is traversed through the CAN identifier, and an identifier mapping table and a timestamp mapping table are generated according to the CAN identifier.

[0016] In some embodiments, the signal value mapping table is a mapping between CAN identifiers and corresponding message signal values, the identifier mapping table is a mapping between CAN identifiers and corresponding data amounts, and the timestamp mapping table is a mapping between CAN identifiers and corresponding data timestamps.

[0017] In some embodiments, extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merging the data columns corresponding to the respective data types into a signal column, and extracting timestamps from the timestamp mapping table to generate a timestamp column includes:

[0018] In the updated signal value mapping table and identifier mapping table, corresponding data is extracted using different data types, and data columns corresponding to the data types are generated;

[0019] Combine all data columns into a signal column;

[0020] In the timestamp mapping table, extract the timestamp to generate a timestamp column.

[0021] In some embodiments, the coded signal column and the timestamp column are intermediate coded data and compressed into intermediate compressed data, including:

[0022] For different data types in the signal sequence, use corresponding encoding algorithms to encode into first encoded data;

[0023] Encoding the timestamp column as second encoded data using run-length encoding;

[0024] Combining the first coded data and the second coded data to obtain intermediate coded data;

[0025] Compress the intermediate encoded data into intermediate compressed data.

[0026] In some embodiments, combining the intermediate compressed data into a columnar compressed file includes:

[0027] Split the intermediate compressed data into one or more data blocks;

[0028] Combine one or more data blocks into columnar compressed files; columnar compressed files include storage files in the ORC format or the Parquet format.

[0029] In some embodiments, combining one or more data blocks into a columnar compressed file includes:

[0030] Traverse the intermediate compressed data stored in one or more data blocks, and save the one or more data blocks as columnar storage files according to the order of timestamps in the intermediate compressed data.

[0031] In some embodiments, the method is applied to a vehicle-cloud integration scenario; before parsing CAN message data using a CAN identifier, the method further includes:

[0032] The CAN message data is collected through the CAN bus deployed in the vehicle's intelligent cockpit domain controller, central threshold controller and autonomous driving domain controller to obtain the CAN message data; wherein, the vehicle is used to execute any of the data compression methods described above to obtain the column-type compressed file.

[0033] The present disclosure also provides a data compression device, which includes:

[0034] A table building module is configured to parse CAN message data using the CAN identifier to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table;

[0035] a column generation module configured to extract different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merge the data columns corresponding to the respective data types into a signal column, and extract timestamps from the timestamp mapping table to generate a timestamp column;

[0036] an encoding and compression module configured to encode the signal column and the time stamp column into intermediate encoded data and compress the encoded data into intermediate compressed data;

[0037] The columnar compression module is configured to combine intermediate compressed data into columnar compressed files. The columnar compressed files support full-data parsing in the cloud.

[0038] In some embodiments, the table building module is configured to parse CAN message data using the CAN identifier and construct a signal value mapping table based on the CAN message data; if the actual data volume corresponding to the CAN identifier in the signal value mapping table is less than the CAN memory, the multiplexing message memory is used to store the message signal value corresponding to the CAN identifier and update the signal value mapping table; wherein, the CAN memory is the memory allocated to the data corresponding to the CAN identifier; the message memory is the memory of the message signal value;

[0039] If the actual data volume is greater than or equal to the CAN memory, a new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; the updated signal value mapping table is traversed through the CAN identifier, and an identifier mapping table and a timestamp mapping table are generated according to the CAN identifier.

[0040] In some embodiments, the signal value mapping table is a mapping of the CAN identifier and the corresponding message signal value; the identifier mapping table is a mapping of the CAN identifier and the corresponding data volume; and the timestamp mapping table is a mapping of the CAN identifier and the corresponding data timestamp.

[0041] In some embodiments, the column encoding module is configured to extract corresponding data using different data types in the updated signal value mapping table and the identifier mapping table, and generate data columns corresponding to the data types; merge all the data columns into the signal column; and extract the timestamp in the timestamp mapping table to generate a timestamp column.

[0042] In some embodiments, the encoding and compression module is configured to encode different data types in the signal column into first encoded data using corresponding encoding algorithms; encode the timestamp column into second encoded data using run-length encoding; merge the first encoded data and the second encoded data to obtain the intermediate encoded data; and compress the intermediate encoded data into the intermediate compressed data.

[0043] In some embodiments, the columnar compression module is configured to split the intermediate compressed data into one or more data blocks; combine the one or more data blocks into the columnar compressed file; and the columnar compressed file includes a storage file in orc format or a storage file in parquet format.

[0044] In some embodiments, the column compression module is configured to traverse the intermediate compressed data stored in the one or more data blocks, and save the one or more data blocks as the column storage file according to the order of timestamps in the intermediate compressed data.

[0045] In some embodiments, the data compression device is applied to a vehicle-cloud integration scenario;

[0046] The data compression device also includes an acquisition module, which is configured to collect message data through the CAN bus deployed in the vehicle's intelligent cockpit domain controller, central threshold controller and autonomous driving domain controller to obtain the CAN message data; wherein, the vehicle is configured to execute the data compression method described in any one of claims 1 to 7 to obtain a columnar compressed file.

[0047] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data compression method as described in any of the preceding items.

[0048] The embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the data compression method as described in any of the above items.

[0049] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the data compression method as described in any of the above items when executed by a processor.

[0050] An embodiment of the present disclosure also provides a chip, comprising at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, the processor communicates with the communication interface and implements the data compression method as described in any of the preceding items through logic circuits or execution code instructions.

[0051] The embodiment of the present disclosure also provides a vehicle, comprising the data compression device as described above.

[0052] In summary, the data compression method proposed in the present invention uses CAN identifiers to parse CAN message data to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table; extracts different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, and merges the data columns corresponding to each data type into a signal column, extracts timestamps from the timestamp mapping table to generate timestamp columns, and organizes different types of data separately, thereby improving the degree of structuring of data compression; encodes signal columns and timestamp columns as intermediate encoded data, and compresses them into intermediate compressed data, thereby improving the compression rate of data compression; combines the intermediate compressed data into a columnar compressed file, thereby improving the degree of structuring and compression rate of data compression.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0055] FIG1 is a flow chart of a data compression method according to an embodiment of the present disclosure;

[0056] FIG2 is a flow chart of a method for parsing CAN message data using a CAN identifier to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table according to an embodiment of the present disclosure;

[0057] FIG3 is a schematic diagram of parsing CAN message data using a CAN identifier according to an embodiment of the present disclosure;

[0058] FIG4 is a flowchart of an embodiment of the present disclosure for extracting different data types from a signal value mapping table and an identifier mapping table to generate data columns corresponding to each data type, merging the data columns corresponding to each data type into a signal column, and extracting timestamps from a timestamp mapping table to generate a timestamp column;

[0059] FIG5 is a flow chart of a method for encoding a signal column and a timestamp column as intermediate encoded data and compressing them into intermediate compressed data according to an embodiment of the present disclosure;

[0060] FIG6 is a flow chart of a method for combining intermediate compressed data into a column-type compressed file according to an embodiment of the present disclosure;

[0061] FIG7 is a schematic diagram of constructing an orc column storage file according to an embodiment of the present disclosure;

[0062] FIG8 is a flow chart of a method for combining one or more data blocks into a columnar storage file according to an embodiment of the present disclosure;

[0063] FIG9 is a flow chart of data compression according to an embodiment of the present disclosure;

[0064] FIG10 is a schematic diagram of data compression for a vehicle-cloud integration scenario according to an embodiment of the present disclosure;

[0065] FIG11 is a schematic structural diagram of a data compression device according to an embodiment of the present disclosure;

[0066] FIG12 is a block diagram of an electronic device for implementing the data compression method of the present disclosure according to an exemplary embodiment;

[0067] FIG13 is a schematic structural diagram of a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar components or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0069] First, a brief introduction to the relevant terms in this disclosure is given:

[0070] CAN: A serial communication protocol bus for real-time applications that uses twisted-pair cables to transmit signals. The CAN protocol is used for communication between various components in a car; the specific information transmitted over the CAN bus is called a CAN message. CAN message data consists of the following parts:

[0071] Start of Frame (SOF): marks the beginning of the message;

[0072] Identifier: used to specify the priority and destination address of the message;

[0073] Data Length Code (DLC): Indicates the number of bytes in the data field;

[0074] Data Field: Contains the actual data to be transmitted;

[0075] Cyclic Redundancy Check (CRC): used to detect errors in message transmission;

[0076] Acknowledge bit (ACK): used by the receiving node to confirm that the message has been successfully received;

[0077] End of Frame (EOF): marks the end of the message;

[0078] Intermission: Provides a certain time interval between messages to ensure that the bus is in an idle state before sending the next message.

[0079] Each device equipped with a CAN bus can receive and send multiple messages with different identifiers or serial numbers (IDs). For example, there are three devices on a CAN bus, namely A, B, and C.

[0080] Device A has receiving ID: 202, 302; sending ID: 101, 102;

[0081] Device B has receiving ID: 101, 303; sending ID: 201, 202;

[0082] Device C has receiving IDs: 101, 102, 201; sending IDs: 302, 303;

[0083] If device A sends a message with ID 101, both devices B and C can receive the message because they can both receive the message with ID 101.

[0084] If device A sends a message with ID 102, only device C can receive the message because only device C has received the message with ID 102.

[0085] If device B sends a message with ID 202, only device A can receive the message because only device A can receive the message with ID 202.

[0086] From the above process, it can be seen that CAN ID is actually a message filter. When the ID of the message on the bus matches the message ID that the receiving end can receive, the receiving end can receive the message corresponding to the above ID.

[0087] The ID in CAN is used to define the meaning of the corresponding message, and each ID has its own specific meaning.

[0088] For example, 101 is a voltage signal and 201 is a current signal. When device A wants to send a voltage signal, it uses a message with ID 101. After devices B and C receive the message with ID 101, they can determine that the signal is a voltage signal.

[0089] In related technologies, CAN message data is first compressed into binary format, then packaged using zip and uploaded to the cloud. Despite this data compression, bandwidth usage remains high. Furthermore, because the message is compressed into binary data, the computing power required to parse the binary data on the cloud is high. Therefore, these related-art CAN message data compression methods suffer from insufficient structure and low compression rates.

[0090] The method proposed in this disclosure is applicable to data compression tasks, a task with a wide range of scenarios, including those in the automotive and industrial automation sectors. In these sectors, data compression technology can improve data transmission efficiency and reduce communication latency. In the automotive industry, the CAN bus is used to connect various electronic control units (ECUs) within a vehicle, enabling rapid information transmission. As the number of sensors and control units in vehicle systems increases, the demand for data transmission also grows. Data compression technology can reduce the bandwidth required for transmission without sacrificing information quality, thereby improving overall system performance. In industrial automation, the CAN bus is used to enable real-time communication between devices. Data compression can help optimize data transmission on production lines, reduce latency, and ensure rapid control system response, which is crucial for maintaining production efficiency and safety. Furthermore, data compression technology has extensive applications in other scenarios requiring efficient and reliable communication, such as aerospace, medical equipment, and smart homes. By reducing data size, data processing speed can be increased, data storage pressure can be reduced, and effective communication can be facilitated in bandwidth-constrained environments. The technical solutions provided in the embodiments of this disclosure are not limited to specific application scenarios.

[0091] The data compression method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0092] FIG1 is a flow chart of a data compression method provided by an embodiment of the present disclosure. In the embodiment shown in FIG1 , the data compression method includes:

[0093] Step 101: Use the CAN identifier to parse CAN message data to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table.

[0094] In this embodiment, the CAN identifier is the identification number of the CAN bus message data, that is, the CAN ID; illustratively, the CAN identifier can be contained in a DBC (DataBase CAN or CAN DataBase Container); wherein, DBC is a standard format for describing data transmission in a CAN network, which defines in detail the structure of the CAN message and the configuration file of the specific information of each signal, including multiple attribute information, such as the signal list corresponding to the CAN ID and the Can ID, and the signal list includes a starting value, starting position, length and data type of each signal in the CAN message data frame.

[0095] The signal value mapping table refers to a mapping table consisting of CAN IDs and their corresponding message signal values. The identifier mapping table refers to a mapping table consisting of CAN IDs and the data volume of their corresponding message signal values. The timestamp mapping table refers to a mapping table consisting of CAN IDs and their corresponding data timestamps.

[0096] Parsing CAN message data usually requires the use of software and hardware tools, such as a CAN analyzer or corresponding software application. These tools can capture the data stream on the CAN bus and decode it according to the structure of the CAN message data, thereby constructing the decoded data into a signal value mapping table, an identifier mapping table, and a timestamp mapping table to provide data support for data compression.

[0097] Step 102: extract different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, merge the data columns corresponding to each data type into a signal column, and extract timestamps from the timestamp mapping table to generate a timestamp column.

[0098] In this embodiment, a data column is a data sequence formed by storing data of the same data type together, and a signal column is a data structure that can merge and store data column values ​​into one column. A timestamp column is a data structure that can store CAN identifiers with the same timestamp into one column. After establishing a signal value mapping table, an identifier mapping table, and a timestamp mapping table, signal values ​​of the same data type are summarized as a whole in the signal value mapping table and the identifier mapping table to form a data column. For example, all signal values ​​of integer data type in the above-mentioned signal mapping table and the identifier mapping table are stored as one data column, and all floating-point signal values ​​in the signal value mapping table are stored as another data column. Signal values ​​of Boolean, string, character, etc. types can also be stored separately as data columns, and all data columns are merged to form a signal column. The data types that support the formation of signal columns are all data types supported by computers. The present disclosure does not limit the data types.

[0099] Columns formed by different types of signal values ​​are treated as data columns, all data columns are merged into signal columns, and the timestamps in the timestamp mapping table are extracted and saved as timestamp columns. This generates signal columns and timestamp columns, improving the structured storage of data.

[0100] Step 103: The coded signal sequence and the time stamp sequence are used as intermediate coded data and compressed into intermediate compressed data.

[0101] In this embodiment, the intermediate coded data is the data after the CAN message is sorted into the above three mapping tables and encoded according to different data types and timestamps. The intermediate compressed data is the data after the intermediate coded data is compressed. For the signal columns and timestamp columns generated according to different data types, encoding is performed according to the corresponding encoding algorithm to form intermediate coded data, and the intermediate coded data is compressed to obtain intermediate compressed data. For example, for integer (int) variables, the zigzag encoding algorithm is used for encoding; for double-precision floating-point (double) variables, the gorilla encoding algorithm is used for encoding; for Boolean (bool) variables, bit-by-bit encoding is used for encoding; for string variables, dictionary encoding is used for encoding. For the timestamps of the signal values ​​in the above three mapping tables, the Run-Length Encoding (RLE) algorithm is used for encoding. The coded data obtained by the above encoding is merged as intermediate coded data, and the intermediate coded data is compressed into intermediate compressed data by the Z standard compression (ZSTD) algorithm, thereby improving the data compression rate.

[0102] Step 104: combine the intermediate compressed data into a column-type compressed file.

[0103] In this embodiment, a columnar compressed file refers to a file format that stores and compresses data in columns. Its core advantage is that it can effectively reduce storage space requirements and support full-data parsing on big data platforms or the cloud. Commonly used columnar compressed file formats include optimized row columnar storage (ORC). Combining intermediate compressed data into columnar compressed files further improves the data compression structure and compression rate.

[0104] In summary, according to the data compression method proposed in the present invention, the CAN identifier is used to parse the CAN message data to obtain a signal value mapping table, an identifier mapping table and a timestamp mapping table; different data types in the signal value mapping table and the identifier mapping table are extracted to generate data columns corresponding to each data type, and the data columns corresponding to each data type are merged into a signal column, and the timestamp is extracted from the timestamp mapping table to generate a timestamp column, and different types of data are sorted separately, thereby improving the degree of structuring of data compression; the signal column and the timestamp column are encoded as intermediate encoded data, and compressed into intermediate compressed data, thereby improving the compression rate of data compression; the intermediate compressed data is combined into a columnar compressed file, thereby improving the degree of structuring and compression rate of data compression.

[0105] FIG2 is a flow chart of a method for parsing CAN message data using CAN identifiers to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table according to an embodiment of the present disclosure. FIG2 is a further explanation of step 101 of FIG1 , and based on the embodiment shown in FIG2 , includes the following steps:

[0106] Step 201: parse CAN message data using a CAN identifier, and construct a signal value mapping table based on the CAN message data.

[0107] In this embodiment, the CAN message data includes a CAN identifier and a message signal value sent or received by the device corresponding to the CAN identifier. The CAN message data is parsed using the CAN identifier, and a signal value mapping table is constructed using the CAN identifier and the corresponding message signal value.

[0108] Step 202: If the actual data volume corresponding to the CAN identifier in the signal value mapping table is smaller than the CAN memory, the multiplexed message memory is used to store the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated.

[0109] Among them, CAN memory is the memory allocated to the data corresponding to the CAN identifier; message memory is the memory of the message signal value.

[0110] In this embodiment, CAN memory refers to the memory allocated to the data corresponding to the CAN identifier. The actual data volume of the message signal value of the currently processed CAN message corresponding to the CAN identifier in the signal value mapping table is compared with the memory size of the message corresponding to the CAN identifier in the signal value mapping table.

[0111] In this embodiment, if the actual data volume is less than the CAN memory, it means that the actual data volume of the message signal value of the currently processed CAN message corresponding to the CAN identifier does not exceed the upper limit of the CAN memory. The message memory can be reused to store the message signal value corresponding to the CAN identifier, thereby completing the update of the data in the mapping table. The message memory is the memory allocated in the system for the message signal value.

[0112] Step 203: If the actual data volume is greater than or equal to the CAN memory, a new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated.

[0113] In this embodiment, if the actual data volume is greater than or equal to the CAN memory, it means that the actual data volume of the message signal value of the currently processed CAN message corresponding to the CAN identifier has exceeded the upper limit of the CAN memory, then new memory is opened for the message signal value corresponding to the CAN identifier, for example, a column of the storage variable list is added to save the message signal value, and the signal value mapping table is updated.

[0114] Step 204: traverse the updated signal value mapping table through the CAN identifier, and generate an identifier mapping table and a timestamp mapping table according to the CAN identifier.

[0115] In this embodiment, in the updated signal value mapping table, traversal is performed through the CAN identifier, and the data size in each message signal value corresponding to the CAN identifier is used to construct an identifier mapping table, and the timestamp corresponding to the CAN identifier is used to construct a timestamp mapping table.

[0116] In an embodiment of the present application, the signal value mapping table is a mapping between CAN identifiers and corresponding message signal values, the identifier mapping table is a mapping between CAN identifiers and corresponding data volumes, and the timestamp mapping table is a mapping between CAN identifiers and corresponding data timestamps.

[0117] In the embodiment of the present application, by limiting the signal value mapping table, the identifier mapping table and the timestamp mapping table, the data integrity and comprehensiveness of the signal value mapping table, the identifier mapping table and the timestamp mapping table are improved.

[0118] FIG3 is a schematic diagram of a method for parsing CAN message data using a CAN identifier according to an embodiment of the present disclosure. As shown in the figure, the method includes the following steps:

[0119] S1, parse the data of a CAN ID.

[0120] S2, obtain the signal value list of the CAN ID from the signal value mapping table.

[0121] S3, determining whether the actual data volume of the CAN ID signal value is less than the CAN memory size of the CAN ID.

[0122] For example, if the actual data volume of the CAN ID signal value is less than the CAN memory size of the CAN ID, indicating that the CAN memory size of the CAN ID is sufficient to store the actual data volume of the CAN ID signal value, a row of value sequences is created in the signal corresponding to the CAN ID in the signal value mapping table as the signal value of the CAN ID. This value sequence reuses the space in the signal value list of the CAN ID. Otherwise, the actual data volume of the message signal value of the CAN ID is used as an index to obtain an existing value sequence from the signal sequence set, thereby updating the signal value mapping table.

[0123] Exemplarily, the value sequence may be valueListPtr.

[0124] S4, traverse each signal data corresponding to the parsed CAN ID.

[0125] S5, judging whether the current data is array data.

[0126] For example, if the data is an array, it is parsed into a binary value in array form. A query is performed to determine whether the binary value exists in the binary data set for the CAN ID. If so, the index of the binary value in the binary data set is obtained. If not, the binary value is appended to the binary data set and the index value is obtained. After determining the index value, the index value is stored in the value sequence. If the current data is not an array, the data is parsed as a single value and the value is stored in the value sequence.

[0127] S6, determining whether the data packet parsing is completed.

[0128] For example, if the parsing is not completed, a signal value list corresponding to the CAN ID is obtained and parsed again. If the parsing is completed, step S7 is executed.

[0129] S7, submit data.

[0130] S8, reset the memory corresponding to each CAN ID in the CAN identifier mapping table to 0.

[0131] S9, parse the next data packet.

[0132] Figure 4 is a flowchart of an embodiment of the present disclosure for extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, merging the data columns corresponding to each data type into a signal column, and extracting timestamps from the timestamp mapping table to generate a timestamp column. Figure 4 further illustrates step 102 of Figure 1 and includes the following steps based on the embodiment shown in Figure 4:

[0133] Step 401 : In the updated signal value mapping table and identifier mapping table, corresponding data are extracted using different data types, and data columns corresponding to the data types are generated.

[0134] Illustratively, the signal value mapping table may be updated using the method provided in the aforementioned embodiment, thereby obtaining an updated signal value mapping table.

[0135] In this embodiment, the data column is a data sequence formed by storing data of the same data type together. In the updated signal value mapping table and identifier mapping table, corresponding data is extracted using different data types, and data columns corresponding to different data types can be generated using the extracted data.

[0136] In one implementation of this embodiment, the updated signal value mapping table includes the device number (integer), the data to be processed (floating point) sent by the device, and the switch status of the device (Boolean). For example, three CAN IDs are parsed in the current CAN message, namely 100, 101, and 102. The message with CAN ID 100 may include data such as a speed value of 15.3km / h, an in-car temperature value of 25.8°C, and an air conditioning switch in the on state (True). The message with CAN ID 101 may include data such as a speed of 1200rpm, a window switch in the closed state (False), and a tire pressure of 240kPa. The message with CAN ID 102 may include data such as a speed value of 35.0km / h, a deceleration start state in the on state (True), and a frame stability adjustment parameter of 30. Save the data in the signal value mapping to the corresponding data column according to the first data type and the second data type. If the first data type is integer and the second data type is Boolean, then save 1200, 240, and 30 in the integer data column of the signal column, and save True, False, and True in the Boolean data column of the signal column.

[0137] Step 402: Merge all data columns into a signal column.

[0138] In this embodiment, all data columns corresponding to different data types are merged to form a signal column. For example, the integer data column and the Boolean data column are merged to form a signal column. This facilitates structured storage and compression of data.

[0139] Step 403: In the timestamp mapping table, extract the timestamp to generate a timestamp column.

[0140] In this embodiment, in the timestamp mapping table, the CAN identifier and the timestamp are associated, and the timestamp in the timestamp mapping table is extracted to generate a timestamp column for specifically storing timestamp data.

[0141] In one implementation of this embodiment, for example, the timestamp mapping table is for CAN ID 200, and the timestamps corresponding to the CAN ID include the first timestamp 2020-12-03-14:15:02, the second timestamp 2020-12-03-14:20:52, and the third timestamp 2021-05-08-08:45:32. These three timestamps are extracted from the timestamp mapping table and stored in the timestamp column data list, thereby generating a timestamp column.

[0142] In this embodiment, the updated signal value mapping table, identifier mapping table, and timestamp mapping table are used to generate signal columns and timestamp columns according to different data types, and different types of data are sorted separately, thereby improving the structural degree of data compression.

[0143] FIG5 is a flow chart of a method for encoding a signal sequence and a timestamp sequence as intermediate encoded data and compressing the encoded signal sequence and the timestamp sequence into intermediate compressed data according to an embodiment of the present disclosure. FIG5 is a further explanation of step 103 of FIG1 , and based on the embodiment shown in FIG5 , includes the following steps:

[0144] Step 501 : For different data types in a signal sequence, use corresponding encoding algorithms to encode them into first encoded data.

[0145] In this embodiment, the first coded data is the data obtained by encoding the different data types in the signal sequence. Different data types in the signal sequence are encoded using the encoding algorithm corresponding to the data type to obtain the first coded data. For example, integer (int) variables are encoded using the zigzag encoding algorithm; double-precision floating-point (double) variables are encoded using the gorilla encoding algorithm; Boolean (bool) variables are encoded using bitwise encoding; and string variables are encoded using dictionary encoding. The encoded integer, floating-point, Boolean, and string variables are then the first coded data.

[0146] Step 502: Encode the timestamp column into second encoded data using run-length encoding.

[0147] In this embodiment, the second coded data is the result of encoding the timestamp data. The timestamp sequence is encoded into the second coded data using a run-length encoding algorithm, effectively organizing the sequence of the timestamp data.

[0148] Step 503: merge the first coded data and the second coded data to obtain intermediate coded data.

[0149] In this embodiment, the first coded data and the second coded data are combined to obtain intermediate coded data. For example, the first coded data and the second coded data are combined and then written into a new file as the intermediate coded data.

[0150] Step 504: compress the intermediate encoded data into intermediate compressed data.

[0151] In this embodiment, the data is compressed into intermediate compressed data using the Zstandard (ZSTD) compression algorithm, thereby improving the data compression rate.

[0152] FIG6 is a flow chart of a method for combining intermediate compressed data into a column-type compressed file according to an embodiment of the present disclosure. FIG6 is a specific description of step 104 of FIG1 , based on the embodiment shown in FIG6 , including the following steps:

[0153] Step 601: divide the intermediate compressed data into one or more data blocks.

[0154] In this embodiment, the intermediate compressed data is divided into one or more data blocks according to the Stripe structure of the orc file to facilitate storage in the orc format.

[0155] Step 602: combine one or more data blocks into a columnar compressed file, where the columnar compressed file includes a storage file in the orc format or a storage file in the parquet format.

[0156] In this embodiment, one or more data blocks are combined into a columnar compressed file, where the columnar compressed file includes a storage file in the orc format or a storage file in the parquet format.

[0157] FIG7 is a schematic diagram of constructing an orc column storage file according to an embodiment of the present disclosure. As shown in the figure, the steps are as follows:

[0158] M1, determines whether the CAN ID writer exists.

[0159] If it does not exist, create a structure of all data types contained in the message according to the signal list type of the CAN ID, for example:<vin:string,ts:bright,signal0:int,signal1:bool,…,signaln:char> Create signal types for various CAN ID signals and use the Type structure type to create a writer (Writer); if it exists, execute M2.

[0160] M2, creates a columnar vector storage block based on the CAN memory size of the CAN ID.

[0161] M3, obtains the first field and timestamp of the column vector storage block for storing the timestamp.

[0162] M4, loops from 0 to CAN memory size (currentidSize), index is i.

[0163] M5, obtain the time point (time) of index i in the timestamp mapping table, and store the time point (time) at index i of the data (data) member of the timestamp (timeBatch).

[0164] M6, obtaining a data value list of index i in the row data set of the CAN ID as a value list of the current data row.

[0165] Exemplarily, the data value column of index i may be valueListPtr.

[0166] M7, loops through the elements of the current row value list in the row data set, indexed by j.

[0167] M8, obtains the j+1th member field fields[j+1] in the columnar vector storage block as the vector storage block indexed by the current value list.

[0168] Exemplarily, the vector storage block may be a vectorBatch.

[0169] M9, according to the type of the value of the value list index j, convert the vector into a vector of the corresponding type.

[0170] Illustratively, the vector type of vector may be DoubleVectorBatch, LongVectorBatch, and StringVectorBatch.

[0171] M10, determines whether the data type is an array type.

[0172] If the data type is an array type, then use the value as the index to retrieve the string binary data valueStr from the CAN ID binary data set, store valueStr in position i of the data member of the vector storage block vectorBatch, and store the length of valueStr in position i of the data member of the vector storage block. The length of valueStr is also stored in position i of the length member of the vector storage block. If the data type is not an array type, then store value in position i of the data member of the vector storage block.

[0173] M11, determines whether the traversal of the value list is completed.

[0174] If not, continue traversing, if so, execute M12.

[0175] M12, determines whether the traversal of the CAN memory size is completed.

[0176] If the currentidSize traversal has not ended, then continue to loop from 0 to currentidSize until the currentidSize traversal ends.

[0177] M13, add a columnar vector storage block to the writer for this CAN ID.

[0178] M14, performs column compression.

[0179] M15, generate orc file.

[0180] FIG8 is a flow chart of a method for combining one or more data blocks into a columnar storage file according to an embodiment of the present disclosure.

[0181] FIG8 is a specific description of step 602 of FIG6 , and based on the embodiment shown in FIG8 , includes the following steps:

[0182] Step 801 : traverse the intermediate compressed data stored in one or more data blocks, and save the one or more data blocks as a column-type compressed file according to the order of timestamps in the intermediate compressed data.

[0183] In this embodiment, the intermediate compressed data stored in one or more data blocks is traversed and saved as column-based compressed files based on the order of timestamps in the intermediate compressed data, that is, the order of timestamps corresponding to CAN IDs. Column-based storage effectively reduces the storage space occupied by data, improves the data compression ratio, and supports full-data parsing in the cloud.

[0184] Figure 9 is a flowchart of a data compression method according to an embodiment of the present disclosure. Based on the embodiment shown in Figure 9, the generation of an orc column-based compressed file includes five steps, which are as follows:

[0185] N1, parsing.

[0186] Parsing means that after capturing the CAN message data, the CAN bus configuration (dbc) and LIN bus configuration (ldf) are used to convert the signal in the CAN message data into real message data through the start bit, length, and accuracy.

[0187] N2, create a table.

[0188] The table is built by using real message data to generate signal value mapping table, identifier mapping table and timestamp mapping table. It prepares data for data compression and improves the data structure rate.

[0189] N3, encoding.

[0190] Encoding involves encoding the message signal values ​​in the signal value mapping table, identifier mapping table, and timestamp mapping table generated by table creation based on their data types. For example, zigzag encoding is used for int data; gorilla encoding is used for double data; bitwise encoding is used for bool data; and dictionary encoding is used for string data. Timestamp column data is encoded using RLE encoding. All encoded data is then combined into the encoded data.

[0191] N4, compression.

[0192] Compression is the process of combining all encoded data into encoded data and then using the ZSTD compression algorithm to process it into compressed data, thereby improving the data compression rate.

[0193] N5, combination.

[0194] Combining compressed data into Stripe data represents data divided into multiple segments that can be processed in parallel. Combining Stripe data into ORC columnar storage files achieves columnar compression of message data, improving the data compression rate and enabling full parsing and querying in the cloud.

[0195] The compression method provided in the embodiment of the present application is applied to a vehicle-cloud integration scenario. Before using the CAN identifier to parse the CAN message data, the following operations may be performed:

[0196] CAN message data is obtained by collecting message data through the CAN bus deployed in the vehicle's intelligent cockpit domain controller (Head Unit, HU), central threshold controller (Cross-point Control Unit, XCU) and automatic driving domain controller (Full Self-Driving, FSD).

[0197] The vehicle is used to execute the data compression method provided in any of the previous embodiments to obtain a column-type compressed file.

[0198] Accordingly, the above method can also perform the following operations:

[0199] The vehicle sends a column-compressed file to the cloud associated with the vehicle in the vehicle-cloud integration scenario, so that the cloud can use the column-compressed file.

[0200] Figure 10 is a schematic diagram of a data compression method for vehicle-cloud integration scenarios according to an embodiment of the present disclosure. Based on the embodiment shown in Figure 10, the CAN bus deployed in the HU, XCU, and FSD of the terminal vehicle collects message data and then performs message parsing. After parsing, edge computing is used to provide real-world vehicle data to the business side. Columnar compression can also be performed to create tables, encode, and compress the message data to form an ORC columnar storage file. ORC columnar storage files are structured data, and structured database storage is generated based on the corresponding values ​​of fields. These ORC columnar storage files are sent to the cloud via the data upload function. Because ORC columnar storage files are efficiently compressed data, they save bandwidth costs during the cloud upload process. Using ORC columnar storage files, object storage is constructed in the cloud. This object storage is a time series database. The ORC format can be directly imported into the time series database for memory queries without the need for secondary parsing. Consequently, cloud computing power consumption is minimal, and full data parsing and querying are possible. Furthermore, indexing by timestamp is possible, making it suitable for scenarios such as multi-dimensional queries, signal playback, and signal exploration.

[0201] From the above, it can be seen that the compression method provided in the embodiment of the present application is applied to the vehicle-cloud integration scenario, and the CAN message is associated with the CAN bus deployed in the vehicle's HU, XCU and FSD. In this way, targeted compression processing of the CAN message associated with the CAN bus deployed in the vehicle's HU, XCU and FSD can be achieved.

[0202] The disclosed embodiment proposes a data compression method, which uses a CAN identifier to parse CAN message data to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table; extracts different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, and merges the data columns corresponding to each data type into a signal column, extracts timestamps from the timestamp mapping table to generate a timestamp column, and organizes different types of data separately, thereby improving the degree of structuring of data compression; encodes signal columns and timestamp columns into intermediate encoded data, and compresses them into intermediate compressed data, thereby improving the compression rate of data compression; and combines the intermediate compressed data into a columnar compressed file, thereby improving the degree of structuring and compression rate of data compression.

[0203] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides a data compression device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0204] FIG11 is a schematic diagram of the structure of a data compression device 1100 according to an embodiment of the present disclosure. As shown in FIG11 , the data compression device includes:

[0205] The table building module 1110 is configured to parse CAN message data using the CAN identifier to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table;

[0206] The column generation module 1120 is configured to extract different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merge the data columns corresponding to the respective data types into a signal column, and extract timestamps from the timestamp mapping table to generate a timestamp column;

[0207] The encoding and compression module 1130 is configured to encode the signal column and the time stamp column into intermediate encoded data and compress them into intermediate compressed data;

[0208] The column compression module 1140 is configured to combine the intermediate compressed data into a column compressed file.

[0209] In some embodiments, the table building module 1110 is configured to parse CAN message data using a CAN identifier and construct a signal value mapping table based on the CAN message data; if the actual data amount corresponding to the CAN identifier in the signal value mapping table is less than the CAN memory, reuse the message memory to store the message signal value corresponding to the CAN identifier, and update the signal value mapping table, wherein the CAN memory is the memory allocated to the data corresponding to the CAN identifier; the message memory is the memory for the message signal value;

[0210] If the actual data volume is greater than or equal to the CAN memory, new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; the updated signal value mapping table is traversed through the CAN identifier, and the identifier mapping table and timestamp mapping table are generated according to the CAN identifier.

[0211] In some embodiments, the signal value mapping table is a mapping between CAN identifiers and corresponding message signal values, the identifier mapping table is a mapping between CAN identifiers and corresponding data amounts, and the timestamp mapping table is a mapping between CAN identifiers and corresponding data timestamps.

[0212] In some embodiments, the column generation module 1120 is configured to extract corresponding data using different data types in the updated signal value mapping table and identifier mapping table, and generate data columns corresponding to the data types; merge all data columns into signal columns; and extract timestamps in the timestamp mapping table to generate timestamp columns.

[0213] In some embodiments, the encoding and compression module 1130 is configured to use corresponding encoding algorithms for different data types in the signal column to encode into first encoded data; use run-length encoding to encode the timestamp column into second encoded data;

[0214] The first coded data and the second coded data are combined to obtain intermediate coded data; and the intermediate coded data is compressed into intermediate compressed data.

[0215] In some embodiments, the columnar compression module 1140 is configured to split the intermediate compressed data into one or more data blocks; and combine the one or more data blocks into a columnar compressed file, which includes a storage file in orc format or a storage file in parquet format.

[0216] In some embodiments, the column compression module 1140 is configured to traverse the intermediate compressed data stored in one or more data blocks, and save the one or more data blocks as a column storage file according to the order of timestamps in the intermediate compressed data.

[0217] In some embodiments, the data compression device is applied to a vehicle-cloud integration scenario; the data compression device further includes a collection module and a transceiver module;

[0218] The acquisition module is configured to acquire message data via the CAN bus deployed in the vehicle's intelligent cockpit domain controller, central threshold controller, and autonomous driving domain controller to obtain CAN message data; wherein the vehicle is configured to execute the data compression method as described above to obtain a column-type compressed file;

[0219] The transceiver module is configured to send the column-compressed file to a cloud associated with the vehicle in a vehicle-cloud integration scenario, so that the cloud can use the column-compressed file.

[0220] In summary, the data compression device uses CAN identifiers to parse CAN message data, obtaining a signal value mapping table, an identifier mapping table, and a timestamp mapping table. Different data types are extracted from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type. The data columns corresponding to each data type are then merged into a signal column. Timestamps are extracted from the timestamp mapping table to generate a timestamp column. The signal column and timestamp column are encoded as intermediate encoded data and compressed into intermediate compressed data. The intermediate compressed data is then combined into a column-based compressed file. This device solves the problems of insufficient data compression structure and low compression rate for message data, thereby improving the degree of data structure and compression rate.

[0221] The embodiments provided above in this disclosure describe the methods and devices provided in these embodiments. To implement the various functions in the methods provided in these embodiments, electronic devices may include hardware structures and software modules, and implement these functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Certain of these functions may be implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules.

[0222] FIG12 is a block diagram showing an electronic device 1200 for implementing the above data compression method according to an exemplary embodiment.

[0223] For example, the electronic device 1200 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

[0224] 12 , the electronic device 1200 may include one or more of the following components: a processing component 1202 , a memory 1204 , a power component 1206 , a multimedia component 1208 , an audio component 1210 , an input / output (I / O) interface 1212 , a sensor component 1214 , and a communication component 1216 .

[0225] The processing component 1202 generally controls the overall operation of the electronic device 1200, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1202 may include one or more modules to facilitate interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate interaction between the multimedia component 1208 and the processing component 1202.

[0226] The memory 1204 is configured to store various types of data to support operations on the electronic device 1200. Examples of such data include instructions for any application or method operating on the electronic device 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0227] The power supply component 1206 provides power to the various components of the electronic device 1200. The power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1200.

[0228] The multimedia component 1208 includes a screen that provides an output interface between the electronic device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the electronic device 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0229] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 1200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1204 or transmitted via the communication component 1216. In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.

[0230] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0231] The sensor assembly 1214 includes one or more sensors for providing various aspects of the status assessment of the electronic device 1200. For example, the sensor assembly 1214 can detect the open / closed state of the electronic device 1200, the relative positioning of components, such as the display and keypad of the electronic device 1200. The sensor assembly 1214 can also detect changes in the position of the electronic device 1200 or a component of the electronic device 1200, the presence or absence of user contact with the electronic device 1200, the orientation or acceleration / deceleration of the electronic device 1200, and changes in the temperature of the electronic device 1200. The sensor assembly 1214 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0232] The communication component 1216 is configured to facilitate wired or wireless communication between the electronic device 1200 and other devices. The electronic device 1200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (New Radio) or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0233] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0234] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, and the instructions can be executed by the processor 1220 of the electronic device 1200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0235] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data compression method described in the above embodiment of the present disclosure.

[0236] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the data compression method described in the above embodiment of the present disclosure when a processor is used.

[0237] An embodiment of the present disclosure further provides a vehicle, comprising the data compression device described in the above embodiment.

[0238] FIG13 is a schematic structural diagram of a chip 1300 for implementing the above data compression method according to an exemplary embodiment.

[0239] 13 , the chip 1300 includes at least one communication interface 1301 and a processor 1302 ; the communication interface 1301 is used to receive signals input into the chip 1300 or signals output from the chip 1300 , and the processor 1302 communicates with the communication interface 1301 and implements the data compression method described in the above embodiment through logic circuits or executing code instructions.

[0240] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0241] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with an embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, the illustrative use of the above terms does not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0242] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0243] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (control method), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0244] It should be understood that the various parts of the embodiments of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0245] Those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0246] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0247] Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limitations on the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure. Industrial Applicability

[0248] The present disclosure provides a data compression method, apparatus, device, chip, vehicle, medium, and program product, relating to the field of big data technology. The data compression method comprises: parsing CAN message data using a CAN identifier to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table; extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to each data type, merging the data columns corresponding to each data type into a signal column, extracting timestamps from the timestamp mapping table to generate a timestamp column; encoding the signal column and the timestamp column into intermediate encoded data, and compressing the data into intermediate compressed data; and combining the intermediate compressed data into a column-type compressed file.

Claims

1. A data compression method, comprising: Use CAN identifiers to parse CAN message data and obtain signal value mapping table, identifier mapping table and timestamp mapping table; Extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merging the data columns corresponding to the respective data types into a signal column, and extracting timestamps from the timestamp mapping table to generate a timestamp column; encoding the signal sequence and the time stamp sequence into intermediate encoded data, and compressing the encoded data into intermediate compressed data; The intermediate compressed data are combined into a column-type compressed file.

2. The method according to claim 1, wherein The method of using the CAN identifier to parse the CAN message data to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table includes: Parsing the CAN message data using the CAN identifier, and constructing the signal value mapping table based on the CAN message data; If the actual data amount corresponding to the CAN identifier in the signal value mapping table is less than the CAN memory, the multiplexed message memory is used to store the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; wherein the CAN memory is the memory to which the data corresponding to the CAN identifier is allocated; and the message memory is the memory for the message signal value; If the actual data volume is greater than or equal to the CAN memory, a new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; The updated signal value mapping table is traversed through the CAN identifier, and the identifier mapping table and the timestamp mapping table are generated according to the CAN identifier.

3. The method according to claim 1 or 2, wherein The signal value mapping table is a mapping between the CAN identifier and the corresponding message signal value; the identifier mapping table is a mapping between the CAN identifier and the corresponding data amount; and the timestamp mapping table is a mapping between the CAN identifier and the corresponding data timestamp.

4. The method according to any one of claims 1 to 3, wherein: The extracting different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merging the data columns corresponding to the respective data types into signal columns, and extracting timestamps from the timestamp mapping table to generate timestamp columns includes: Extracting corresponding data from the updated signal value mapping table and the identifier mapping table using different data types, and generating data columns corresponding to the data types; Merging all the data columns into the signal column; In the timestamp mapping table, timestamps are extracted to generate timestamp columns.

5. The method according to any one of claims 1 to 4, wherein: The encoding of the signal sequence and the timestamp sequence into intermediate encoded data and compressing the intermediate compressed data includes: For different data types in the signal sequence, use corresponding encoding algorithms to encode into first encoded data; encoding the timestamp column into second encoded data using run-length encoding; Combining the first coded data and the second coded data to obtain the intermediate coded data; The intermediate encoded data is compressed into the intermediate compressed data.

6. The method according to any one of claims 1 to 5, wherein: Combining the intermediate compressed data into a column-type compressed file includes: dividing the intermediate compressed data into one or more data blocks; The one or more data blocks are combined into the columnar compressed file; the columnar compressed file includes a storage file in the orc format or a storage file in the parquet format.

7. The method according to claim 6, wherein: Combining the one or more data blocks into the column-compressed file includes: The intermediate compressed data stored in the one or more data blocks are traversed, and the one or more data blocks are saved as the column storage file according to the order of timestamps in the intermediate compressed data.

8. The method according to any one of claims 1 to 7, wherein: The method is applied to vehicle-cloud integration scenarios; Before parsing the CAN message data using the CAN identifier, the method further includes: The CAN message data is collected through the CAN bus deployed in the vehicle's intelligent cockpit domain controller, central threshold controller and autonomous driving domain controller to obtain the CAN message data; wherein, the vehicle is used to execute the data compression method according to any one of claims 1 to 7 to obtain the columnar compressed file.

9. A data compression device, comprising: a table building module configured to parse CAN message data using CAN identifiers to obtain a signal value mapping table, an identifier mapping table, and a timestamp mapping table, wherein the signal value mapping table is a mapping between the CAN identifiers and corresponding message signal values, the identifier mapping table is a mapping between the CAN identifiers and corresponding data amounts, and the timestamp mapping table is a mapping between the CAN identifiers and corresponding data timestamps; a column generation module configured to extract different data types from the signal value mapping table and the identifier mapping table to generate data columns corresponding to the respective data types, merge the data columns corresponding to the respective data types into a signal column, and extract timestamps from the timestamp mapping table to generate a timestamp column; an encoding and compression module configured to encode the signal column and the timestamp column into intermediate encoded data, and compress the encoded data into intermediate compressed data; The column compression module is configured to combine the intermediate compressed data into a column compression file, and the column compression file supports full cloud parsing.

10. The data compression device according to claim 9, wherein: The table building module is configured to parse CAN message data using the CAN identifier and build a signal value mapping table based on the CAN message data; If the actual data amount corresponding to the CAN identifier in the signal value mapping table is less than the CAN memory, the multiplexed message memory is used to store the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; wherein the CAN memory is the memory to which the data corresponding to the CAN identifier is allocated; and the message memory is the memory for the message signal value; If the actual data volume is greater than or equal to the CAN memory, a new memory is opened for the message signal value corresponding to the CAN identifier, and the signal value mapping table is updated; the updated signal value mapping table is traversed through the CAN identifier, and an identifier mapping table and a timestamp mapping table are generated according to the CAN identifier.

11. The data compression device according to claim 9 or 10, wherein: The signal value mapping table is a mapping between the CAN identifier and the corresponding message signal value; the identifier mapping table is a mapping between the CAN identifier and the corresponding data amount; and the timestamp mapping table is a mapping between the CAN identifier and the corresponding data timestamp.

12. The data compression device according to any one of claims 9 to 11, wherein: The column encoding module is configured to extract corresponding data using different data types from the updated signal value mapping table and the identifier mapping table, and generate data columns corresponding to the data types; and merge all the data columns into the signal column; In the timestamp mapping table, timestamps are extracted to generate timestamp columns.

13. The data compression device according to any one of claims 9 to 12, wherein: The encoding and compression module is configured to encode different data types in the signal column into first encoded data using corresponding encoding algorithms; encode the timestamp column into second encoded data using run-length encoding; merge the first encoded data and the second encoded data to obtain the intermediate encoded data; and compress the intermediate encoded data into the intermediate compressed data.

14. The data compression device according to any one of claims 9 to 13, wherein: The column compression module is configured to divide the intermediate compressed data into one or more data blocks; and combine the one or more data blocks into the column compressed file; The columnar compressed file includes a storage file in the orc format or a storage file in the parquet format.

15. The data compression device according to claim 14, wherein: The column compression module is configured to traverse the intermediate compressed data stored in the one or more data blocks, and save the one or more data blocks as the column storage file according to the order of timestamps in the intermediate compressed data.

16. The data compression device according to any one of claims 9 to 15, wherein: The data compression device is applied to a vehicle-cloud integration scenario; the data compression device also includes an acquisition module, which is configured to collect message data through the CAN bus deployed in the vehicle's intelligent cockpit domain controller, central threshold controller and autonomous driving domain controller to obtain the CAN message data; wherein, the vehicle is configured to execute the data compression method according to any one of claims 1 to 7 to obtain a columnar compressed file.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being configured to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

20. A chip comprising at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, the processor communicates with the communication interface and implements the method according to any one of claims 1-8 through logic circuits or executing code instructions.

21. A vehicle comprising the data compression device according to any one of claims 9 to 16.

Images