Message recombination compression method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a message recombination compression method and device, equipment and a storage medium, which are applied to the technical field of computers, and the method comprises the following steps: obtaining a plurality of continuous messages, and analyzing each message to obtain a plurality of fields in each message; obtaining classification information of the fields, and grouping the fields according to the classification information to obtain field groups; extracting numerical value distribution information of each field group, and determining a target compression mode of the current field group based on the numerical value distribution information; compressing the current field group according to the target compression mode to obtain a compressed field; message collection information is extracted from the message, and file header information is obtained through calculation according to the message collection information; and assembling the file header information and the compressed field to obtain a packaged file of the message, and according to the method and the device, through disassembling and recombining the continuous message, filtering and compressing the duplicate value and the similar value, the requirement of a storage space is remarkably reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a message reassembly and compression method, device, equipment and storage medium. Background Art

[0002] GB32960 is a vehicle message format defined in the Chinese national standard "Technical Requirements for Electric Vehicle Remote Service and Management System Part 3: On-board Terminal" (GB / T 32960.3-2016). This standard specifies the data collection, transmission and storage requirements of on-board terminals in electric vehicle remote service and management systems. It is mainly used to monitor and manage the operating status, battery status, fault information, etc. of electric vehicles.

[0003] Since vehicles continuously generate a large amount of data during operation, vehicles that support GB32960 message collection usually collect a batch of data every 10 seconds and assemble them into messages for transmission. The vehicle's status data over a period of time often contains a large number of repeated indicator values. If these messages are stored directly, a large amount of storage space will be occupied, significantly reducing the effective use of storage space. In addition, when analyzing a large amount of repeated original message data, a lot of time and computing resources are required to extract effective information from the data, thereby reducing processing efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a message reorganization and compression method, device, equipment and storage medium to solve the technical problems in the prior art that directly storing vehicle messages has a large amount of duplicate data, occupies a large amount of storage resources, and cannot locate valid data. The specific technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, a method for reassembling and compressing a message is provided, the method comprising:

[0006] Obtaining a plurality of consecutive messages, parsing each of the messages, and obtaining a plurality of fields in each of the messages;

[0007] Acquire classification information of the field, group the fields according to the classification information, and obtain field groups;

[0008] Extracting the value distribution information of each of the field groups, and determining the target compression mode of the current field group based on the value distribution information;

[0009] Compressing the current field group according to the target compression mode to obtain a compressed field;

[0010] Extracting message collection information from the message, and calculating and obtaining file header information according to the message collection information;

[0011] Assemble the file header information and the compression field to obtain a packaged file of the message.

[0012] Optionally, the compression method includes fixed value compression, continuous value compression and discrete value compression, and the extracting the value distribution information of each field group and determining the target compression method of the current field group based on the value distribution information includes:

[0013] Counting the values ​​in the current field group to obtain the value distribution information of the current field group;

[0014] If the value distribution information indicates that the value is a fixed value, confirming that the target compression mode of the current field grouping is fixed value compression;

[0015] If the value distribution information indicates that the value is a continuous value, confirming that the target compression mode of the current field grouping is continuous value compression;

[0016] If the numerical distribution information indicates that the numerical value is a discrete value, it is confirmed that the target compression method of the current field grouping is discrete value compression.

[0017] Optionally, counting the values ​​in the current field group to obtain the value distribution information of the current field group includes:

[0018] Counting the values ​​in the current field group, and if all the values ​​in the current field group are equal, confirming that the value distribution information indicates that the value is a fixed value;

[0019] If the values ​​in the current field group are not equal, calculating the variance of the current field group;

[0020] If the variance is less than or equal to a first preset threshold, confirming that the value distribution information indicates that the value is a continuous value;

[0021] If the variance is greater than the first preset threshold, it is confirmed that the numerical distribution information indicates that the numerical value is a discrete value.

[0022] Optionally, when it is detected that the target compression mode is fixed value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes:

[0023] Obtaining a fixed number of occupied bits corresponding to the target compression mode and a first target value of the current field group;

[0024] Calculate the first occupied bit number of the first target value;

[0025] The compressed field is obtained based on the fixed number of occupied bits, the first number of occupied bits, and the first target value.

[0026] Optionally, when it is detected that the target compression mode is continuous value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes:

[0027] Obtaining a fixed number of occupied bits corresponding to the target compression mode and a second target value of the current field group;

[0028] Taking the minimum value among the second target values ​​as the first offset, subtracting the first offset from all the second target values ​​to obtain the offset second target values;

[0029] Calculate a second number of occupied bits based on the maximum value of the second target values ​​after the shift;

[0030] Using continuous and equal values ​​in the second target value after the shift to form a first subgroup, removing groups equal to a default value from the first subgroup, where the default value is the mode of the second target value after the shift;

[0031] From the remaining first subgroups, a group containing the largest number of values ​​is obtained as a second subgroup, and a third number of occupied bits of the second subgroup is calculated;

[0032] Creating a segment record for each of the remaining first subgroups, wherein the segment record includes a value and a number of values ​​included in the remaining first subgroups;

[0033] The compressed field is obtained based on the fixed number of occupied bits, the second number of occupied bits, the third number of occupied bits and the segment record.

[0034] Optionally, when it is detected that the target compression mode is discrete value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes:

[0035] Obtaining a fixed number of occupied bits corresponding to the target compression method, and removing duplicates from all values ​​in the current field group to obtain a third target value;

[0036] Taking the minimum value among the third target values ​​as the second offset, subtracting the second offset from all the third target values ​​to obtain the second offset;

[0037] Calculate a fourth number of occupied bits based on a maximum value among the second offsets;

[0038] Mapping the third target value to a target index, calculating the fifth occupied bit number of the target index, and listing the target index corresponding to each third target value;

[0039] The compressed field is obtained based on the fixed number of occupied bits, the fourth number of occupied bits, the fifth number of occupied bits, and the target index.

[0040] Optionally, extracting message collection information from the message, and calculating and obtaining file header information according to the message collection information includes:

[0041] Extracting message collection information of each message from the plurality of messages, wherein the message collection information includes message collection time and message collection quantity;

[0042] The maximum value of the message collection time is used as the start timestamp, and the minimum value of the message collection time is used as the end timestamp;

[0043] Obtaining the sixth number of occupied bits of the compression field, and taking the maximum value of the sixth number of occupied bits as the number of occupied bits of the message;

[0044] Extracting a collection time interval from the message collection time, and taking the mode of the collection time intervals as the message collection interval;

[0045] Adjacent messages whose collection time interval is less than a preset interval threshold are used to form a third sub-group, and the number of groups of the third sub-group is obtained;

[0046] Calculating a time offset and a number of messages of each third subgroup, wherein the time offset is obtained based on a difference between a first acquisition time in the third subgroup and the start timestamp;

[0047] The file header information is generated, wherein the file header information includes the start timestamp, the end timestamp, the number of bits occupied by the message, the message collection interval, the number of groups, the time offset of each third sub-group, and the number of messages.

[0048] According to a second aspect of an embodiment of the present application, a message reassembly and compression device is provided, the device comprising:

[0049] A message parsing module, used to obtain a plurality of consecutive messages, parse each of the messages, and obtain a plurality of fields in each of the messages;

[0050] A field grouping module, used for obtaining classification information of the fields, grouping the fields according to the classification information, and obtaining field groups;

[0051] A compression mode determination module, used for extracting the value distribution information of each field group, and determining the target compression mode of the current field group based on the value distribution information;

[0052] A field compression module, used for compressing the current field group according to the target compression mode to obtain a compressed field;

[0053] A file header calculation module, used to extract message collection information from the message, and calculate and obtain file header information according to the message collection information;

[0054] The compression assembly module is used to assemble the file header information and the compression field to obtain a packaged file of the message.

[0055] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the message reassembly and compression method as described in any one of the first aspects.

[0056] According to the fourth aspect of an embodiment of the present application, a non-volatile readable storage medium is also provided. When the instructions in the non-volatile readable storage medium are executed by a processor of an electronic device, the electronic device can execute the message reorganization and compression method as described in any one of the first aspects.

[0057] The message reorganization and compression method provided in the embodiment of the present application obtains several consecutive messages, parses each message, obtains several fields in each message; obtains the classification information of the field, groups the fields according to the classification information, and obtains the field grouping; extracts the numerical distribution information of each field grouping, determines the target compression method of the current field grouping based on the numerical distribution information; compresses the current field grouping according to the target compression method to obtain the compressed field; extracts the message collection information from the message, calculates and obtains the file header information according to the message collection information; assembles the file header information and the compressed field to obtain the packaged file of the message. The present application parses a batch of consecutive messages, regroups them according to the classification information of the field, adopts different compression strategies (such as the compression method of fixed value, continuous value, and discrete value fields) for the numerical characteristics in each field group, and performs the maximum data compression. By parsing and reorganizing the collected messages, duplicate values ​​and similar values ​​are filtered and compressed, the storage space requirement is significantly reduced, and by removing a large amount of duplicate data, it helps to improve the positioning rate of effective information, and facilitates the analysis of the trend of a certain indicator in a certain period of time. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a flowchart of the steps of a message reassembly and compression method provided in an embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of a packaged file in a message reassembly and compression method provided in an embodiment of the present application;

[0061] Figure 3 It is a structural block diagram of a message reassembly and compression device provided in an embodiment of the present application;

[0062] Figure 4 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, some technical features involved in the embodiments of the present application are explained and illustrated below:

[0065] GB32960 is the abbreviation of China's national standard "Technical Requirements for Remote Service and Management Systems for Electric Vehicles" (GB / T32960). This standard specifies the technical requirements for remote service and management systems for electric vehicles, and is mainly used to regulate the remote service and management systems for electric vehicles, ensuring the accurate collection, transmission and storage of vehicle data, so as to facilitate applications such as vehicle status monitoring, fault diagnosis, and remote control.

[0066] Vehicle messages refer to various data information collected and transmitted by the vehicle terminal during the operation of the vehicle. These data information usually include the vehicle's operating status, battery status, fault information, location information, etc. GB32960 vehicle messages refer to vehicle status data collected and sent by the electric vehicle's onboard terminal in accordance with the format and content specified in GB32960.

[0067] Parquet is an efficient columnar storage format that is widely used in big data processing and analysis. It supports multiple data types, compression algorithms, and partitioned storage, and is suitable for storing and querying large-scale data.

[0068] The Vehicle Identification Number (VIN) is a unique identification code for a vehicle, similar to the vehicle's "ID number". The VIN code consists of 17 characters and contains important information such as the vehicle's manufacturer, model, year of production, location of production, engine type, etc. The VIN code is unique worldwide and is used to identify and track vehicles.

[0069] As an example, GB32960 message, as a vehicle message under the GB32960 standard, can be used to monitor and manage the operating status, battery status, fault information, etc. of electric vehicles. Usually, vehicles that support the collection of this message will collect a batch of data every 10 seconds and assemble it into a message for transmission. Since the vehicle will continuously generate a large amount of data during operation, the data collected by each vehicle message often contains a large number of identical indicator values. If the original message is stored directly, it will take up a lot of storage space, and a large amount of original message data will require a lot of extra time and computing resources to extract useful information when performing data analysis, making it difficult to quickly locate data for a specific time period or specific indicators.

[0070] One of the core invention points of the present application is to parse and reorganize a batch of continuous messages of the same vehicle to complete packaged storage. By parsing a batch of continuous messages, they are regrouped according to the classification information of the fields, and different compression strategies (such as fixed, continuous, and other types of field compression methods) are adopted for the numerical characteristics within each field group to maximize data compression. By parsing and reorganizing the collected messages, filtering and compressing duplicate values ​​and similar values, the demand for storage space is significantly reduced, and by removing a large amount of duplicate data, it helps to improve the positioning rate of effective information.

[0071] Reference Figure 1 , shows a flowchart of a method for reassembling and compressing a message provided in an embodiment of the present application, which may specifically include the following steps:

[0072] S101, obtaining a plurality of consecutive messages, parsing each message, and obtaining a plurality of fields in each message.

[0073] In an embodiment of the present application, compression of vehicle messages is completed by parsing and reorganizing a batch of continuous vehicle messages, wherein the data collection time range of the continuous vehicle messages can be reasonably set according to the online time and data volume of the vehicle. For example, considering the usual online time of the vehicle and the size of the packaged file after the message is compressed and packaged, the collection time range of the continuous message can be set to one month, that is, the vehicle messages of the same vehicle within one month are collected as compressed data.

[0074] Several consecutive message data are obtained from the vehicle terminal or data storage system, and each message is parsed into multiple fields using a parsing algorithm, wherein each field represents a different state or parameter of the vehicle. For example, assuming that there are 100 original messages, each message consists of 20 fields. By parsing a vehicle message, 20 fields can be obtained. After parsing each message, a total of 2000 fields can be obtained. Preferably, the message reorganization and compression method proposed in the present application is not only applicable to GB32960 standard messages, but can also be widely used in messages of other standards, such as ISO 15118, UDS, etc., and even various types of data transmission and storage scenarios, which are not limited by the present application.

[0075] S102, obtaining classification information of fields, grouping the fields according to the classification information, and obtaining field groups.

[0076] After the message is disassembled, all the disassembled fields can be regrouped according to the classification information of the field. For example, the classification information of the field may include the field name, field function, field source, etc. For example, according to the field name, the "number of single cell batteries" fields in all messages can be grouped into the same group, and the "vehicle speed" fields can be grouped into the same group.

[0077] S103, extracting the value distribution information of each field group, and determining the target compression mode of the current field group based on the value distribution information.

[0078] As an example, for each field grouping, the value distribution information of the field grouping can be obtained by extracting and analyzing the values ​​in the group. Usually, the value distribution information may include: data type, data range, and frequency distribution, wherein the data type may indicate whether the current value is a numeric type (integer, floating point number), text type, or other types, the data range may indicate the maximum and minimum values ​​of the current value, and the frequency distribution may indicate the frequency of occurrence of different values, especially whether there are many repeated values, or whether there is an obvious pattern (such as a normal distribution).

[0079] According to the numerical distribution information of each field grouping, a compression method suitable for the current field grouping can be selected as the target compression method under the premise of increasing the compression degree. Exemplarily, the compression methods can include fixed value compression, continuous value compression and discrete value compression, which correspond to field groups with fixed values, continuous values ​​and discrete values, respectively. By extracting the numerical distribution information of each field grouping and determining the target compression method based on this information, the efficiency and effect of data compression can be significantly improved.

[0080] In a possible embodiment, the target compression mode may be selected through the following steps:

[0081] Count the values ​​in the current field group and obtain the value distribution information of the current field group;

[0082] If the value distribution information indicates that the value is a fixed value, confirming that the target compression method of the current field grouping is fixed value compression;

[0083] If the value distribution information indicates that the value is a continuous value, confirming that the target compression method of the current field grouping is continuous value compression;

[0084] If the value distribution information indicates that the value is a discrete value, it is determined that the target compression method of the current field grouping is discrete value compression.

[0085] In an embodiment of the present application, for each field grouping, all values ​​in the current field grouping can be counted, and based on the statistical results, it can be determined whether the value is a fixed value, a continuous value or a discrete value, and the corresponding compression method can be selected according to the distribution information of the value.

[0086] Specifically, if all the values ​​in the current field grouping are equal, such as in the field grouping "number of single batteries", the number of batteries in the same vehicle is usually a fixed value, and the values ​​obtained within a period of time are all the same value, it can be confirmed that the numerical distribution information represents the value as a fixed value; if the values ​​in the current field grouping are not all equal, for example, the variance of the current field grouping can be calculated to further judge the numerical distribution of the field grouping. Variance is a quantity used in statistics to measure the degree of discreteness of data distribution, which can reflect the difference between each data point in the data set and its average (i.e., mean). Specifically, the larger the variance, the more dispersed the data point, that is, the greater the fluctuation or change; the smaller the variance, the closer the data point is to the mean, that is, the smaller the change, and the more stable the data. Preferably, other statistical indicators can also be used to analyze the numerical distribution information in the field grouping, such as the difference between continuous values, the mean, square difference, quantile, etc. of the field grouping. This application uses variance as an example to illustrate the acquisition of numerical distribution information, and does not limit it.

[0087] If the variance of the current field grouping is less than or equal to the first preset threshold, it indicates that the difference between the values ​​of the field grouping is small, or the values ​​of the field grouping do not change frequently. If the variance of the current field grouping is greater than the first preset threshold, it indicates that the difference between the values ​​of the field grouping is large, or the values ​​of the field grouping change frequently. The first preset threshold is used to set the range of change of the values ​​in the current field grouping, which can be set according to actual needs. For values ​​with small differences or infrequent changes, they can be considered as continuous values, such as the battery temperature of the vehicle. There may be fields with large values, but the field values ​​are usually continuous and do not change frequently. For values ​​with large differences or frequent changes, they can be considered as discrete values, such as the vehicle speed, current and other values, which change frequently during the use of the vehicle.

[0088] S104, compressing the current field group according to the target compression method to obtain a compressed field.

[0089] Compressing the current field group according to the target compression method can significantly reduce the amount of data and improve the efficiency of data transmission and storage. For field groups with fixed values, continuous values, and discrete values, the compression methods can include fixed value compression, continuous value compression, and discrete value compression. The current field group is compressed according to the selected target compression method.

[0090] Specifically, refer to Figure 2 , showing a schematic diagram of a packaged file in a message reassembly and compression method provided in an embodiment of the present application, Figure 2 The field group 1 in can be used as an example of fixed value compression. When it is detected that the target compression method is fixed value compression, since the values ​​in the field group are equal, only the data length, value distribution type, number of bits occupied by the value and the value can be saved. Among them, the data length and value distribution type are fixed storage items. Each field group needs to store the data length and value distribution type of the group. For example, the value distribution type can be determined by the target compression method. For example, corresponding to fixed value compression, the value distribution type of the field group is a fixed value. The value distribution type is stored in each compressed field group. A fixed-length storage can be allocated for the value distribution type. The number of bits, such as 2 bits of storage bit can be allocated for the value distribution type; the number of bits occupied by the value, that is, the first number of bits occupied, can also be determined by obtaining the value of the field grouping, such as the number of bits occupied by the value can be set to 5 bits. Since the values ​​in the field grouping are fixed values, all values ​​are equal, and the values ​​can be filled in once. The compressed field grouping consists of data length, value distribution type, number of bits occupied by the value and the value, wherein the data length can be calculated through the fixed number of bits occupied by the value distribution type, the first number of bits occupied and the value. Preferably, the data length can be 2+5+x (bit), wherein the value of x is determined by the value.

[0091] Reference Figure 2 , Figure 2 The field grouping 2 in can be used as an example of continuous value compression. When it is detected that the target compression method is continuous value compression, compression operations such as offset calculation and deduplication can be performed on the field grouping. Specifically, the two fixed storage items of data length and value distribution type are first obtained. Similarly, a 2-bit storage bit number can be allocated to the value distribution type, and the minimum value is obtained from the second target value of the current field grouping, and the minimum value is used as the first offset, wherein the second target value represents all values ​​of the current field grouping, and the first offset is subtracted from all second target values ​​to obtain the offset second target value. By recording the first offset and subtracting the first offset from all values, all values ​​can be restored on the basis of which the values ​​in the field grouping occupy a smaller storage space.

[0092] Select the maximum value from the second target value after the offset, and calculate the second number of digits occupied by the maximum value. After the value of the field group is subtracted from the first offset, each remaining difference is different. The maximum difference is used as the basis to calculate the second number of digits occupied, and all remaining differences can be stored. The continuous and equal values ​​in the second target value after the offset are grouped into the same group, which can be called the first sub-group. The groups in the first sub-group that are equal to the default value are filtered out, where the default value refers to the mode in the second target value after the offset, that is, the value that appears the most times in the second target value after the offset. Using the default value to filter the first sub-group can effectively remove duplicate field groups and reduce duplicate data storage; from the first sub-group remaining after filtering, select the group containing the largest number of values, which can be called the second sub-group, and calculate the number of sub-group digits occupied by the second sub-group, which is recorded as the third number of digits occupied. Based on the same principle, after filtering out duplicate data, the group with the most data is obtained from each sub-group to calculate the number of digits occupied. The third number of digits occupied is sufficient to store any remaining sub-groups.

[0093] A segment may also be created for each remaining first subgroup, and a corresponding segment record may be generated, such as Figure 2As shown, the segment record may include the numerical value contained in the first sub-group, the number of numerical values ​​contained, and may also include the position of the sub-group in all numerical values ​​of the current field group. The compressed field grouping is composed of data length, value distribution type, first offset, number of occupied bits of the numerical value, default value, number of sub-group bits, several segments and segment records, wherein the data length can be calculated by the fixed number of occupied bits of the value distribution type, the second number of occupied bits of the numerical value, the third number of occupied bits of the sub-group and the segment record. Preferably, the data length can be 2+4*8+5+x+5+(x+3*8+y)*n(bit), wherein the number of bits occupied by the first offset is 4B, which is equal to 4 8 bits, the number of storage bits allocated to the sub-group is the same as the number of bits of the numerical value, which is 5 bits, y represents the number of data contained in the sub-group, and n represents the number of segment records.

[0094] Figure 2 The field group 3 in can be used as an example of discrete value compression. When it is detected that the target compression mode is discrete value compression, the field group can be subjected to compression operations such as offset calculation and deduplication. Specifically, the two fixed storage items of data length and value distribution type are first obtained. Similarly, a 2-bit storage bit number can be allocated for the value distribution type. The field group is subjected to numerical deduplication. For the convenience of distinction, the deduplication value is used as the third target value, and the minimum value is extracted from the third target value as the second offset. The second offset is subtracted from all third target values ​​to obtain the offset third target value. By recording the second offset and subtracting the second offset from all third target values, the values ​​in the field group can occupy a smaller storage space on the basis of restoring all values. The maximum value is selected from the offset third target value, and the fourth occupied bit number occupied by the maximum value is calculated. After the value of the field group is subtracted from the second offset, each remaining difference is different. The largest difference is used as a basis to calculate the fourth occupied bit number, and all remaining differences can be stored.

[0095] The offset third target value is stored as an array, and the index subscript position of each value in the field group is recorded, where the index subscript position is counted from 0. For example, the number of arrays is recorded as m, and 2B of storage space can be allocated for the number of arrays to realize mapping the offset third target value to the target index. At the same time, the fifth occupied bit of the target index is calculated, and the target indexes corresponding to each offset third target value can be listed one by one, and the storage space of each listed array is recorded as xbit, and the storage space of each listed index subscript is recorded as ybit. Thus, the compressed field group consists of data The data length is composed of the fixed number of occupied bits of the value distribution type, the fourth number of occupied bits of the third target value after the offset, the fifth number of occupied bits of the target index and the array and index subscript of the enumeration. Preferably, the data length can be 2+4*8+5+2*8+(x+y)*m(bit), wherein the number of bits occupied by the second offset is 4B, which is equal to 4 8-bits, and 2B of storage space is allocated for the number of arrays, which is equivalent to 2 8-bits.

[0096] S105, extracting message collection information from the message, and calculating and obtaining file header information according to the message collection information.

[0097] By parsing the message, the message collection information can be extracted from the message and the file header information can be calculated, which can reduce the storage and transmission of redundant data. For example, repeated message collection information can be stored only once in the file header information, such as the frame number, and subsequent data only needs to store the difference part, so that the compressed data occupies less storage space and bandwidth, improving the efficiency of data transmission and storage.

[0098] Specifically, Figure 2 As shown in the file header information, the message collection information of each message can be extracted from a batch of continuous messages, wherein the message collection information may include the vehicle frame number, message collection time and message collection quantity, and optionally, the message collection information may also include vehicle status, vehicle location and other information.

[0099] The maximum value of all message collection times is used as the start timestamp, and the minimum value is used as the end timestamp. The units of message collection time and timestamp are seconds. The data length of all compressed fields is obtained, which can be called the sixth occupied bit number. The maximum value is selected from the sixth occupied bit number as the message occupied bit number.

[0100] By sorting all the message collection times, the collection time interval between two consecutive message collections can be extracted from the message collection time, and the mode with the most occurrences in the collection time interval is used as the message collection interval; at the same time, the continuous messages, or adjacent messages whose collection time interval is less than the preset interval threshold, are grouped into a third sub-group, and the number of groups of the third sub-group is obtained, wherein the preset time threshold can be set as needed. The time offset of the third sub-group is obtained by subtracting the start timestamp from the collection time of the first message in each third sub-group, the number of messages in the third sub-group is counted, and the time offset and the number of messages are recorded in correspondence with the third sub-group. The generated file header information consists of the start timestamp, the end timestamp, the number of bits occupied by the message, the message collection interval, the number of groups and the group record of each third sub-group, wherein the group record refers to the time offset and the number of messages of the third sub-group.

[0101] S106, assembling the file header information and the compression field to obtain a packaged file of the message.

[0102] According to all the data obtained by sorting in steps S101 to S106, the file header information and all the compressed field contents are written in sequence, that is, the assembly of the file header information and the compressed fields is completed, and a batch of continuous message packaging files are obtained, thereby realizing the compression and packaging of the messages.

[0103] With the increase of vehicle electronic control units (ECUs) and the complexity of their functions, the amount of data transmitted within vehicles has increased dramatically. Vehicle message compression is an important means to improve vehicle data transmission efficiency and reduce bandwidth usage. Compressed messages take up less storage space, which is conducive to long-term data storage and management. At the same time, compressed data transmission speed is faster, especially when network bandwidth is limited or unstable, which can improve the reliability and efficiency of data transmission. Therefore, compression technology can effectively reduce the amount of transmitted data, speed up data transmission, and reduce energy consumption and costs.

[0104] The present application performs maximum data compression by parsing and reorganizing a batch of continuous messages, regrouping them according to the classification information of the fields, and adopting different compression strategies (such as compression methods for fixed value, continuous value, and discrete value fields) according to the numerical characteristics within each field group. By parsing and reorganizing the collected messages, filtering and compressing duplicate values ​​and similar values, the demand for storage space is significantly reduced, and by removing a large amount of duplicate data, it helps to improve the location rate of effective information.

[0105] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0106] Reference Figure 3 , shows a structural block diagram of a message reassembly and compression device provided in an embodiment of the present application, which may specifically include the following modules:

[0107] The message parsing module 301 is used to obtain a number of consecutive messages, parse each message, and obtain a number of fields in each message;

[0108] A field grouping module 302 is used to obtain classification information of the fields, group the fields according to the classification information, and obtain field groups;

[0109] A compression mode determination module 303 is used to extract the value distribution information of each field group and determine the target compression mode of the current field group based on the value distribution information;

[0110] A field compression module 304 is used to compress the current field group according to the target compression mode to obtain a compressed field;

[0111] A file header calculation module 305 is used to extract message collection information from the message and calculate the file header information according to the message collection information;

[0112] The compression assembly module 306 is used to assemble the file header information and the compression field to obtain the packaged file of the message.

[0113] Furthermore, the compression method may include fixed value compression, continuous value compression and discrete value compression, and the compression method determination module 303 includes:

[0114] The field value statistics module is used to count the values ​​in the current field group and obtain the value distribution information of the current field group;

[0115] A fixed value compression determination module, used for confirming that the target compression mode of the current field group is fixed value compression if the value distribution information indicates that the value is a fixed value;

[0116] A continuous value compression determination module, used for confirming that the target compression mode of the current field grouping is continuous value compression if the value distribution information indicates that the value is a continuous value;

[0117] The discrete value compression determination module is used to confirm that the target compression method of the current field grouping is discrete value compression if the numerical distribution information indicates that the numerical value is a discrete value.

[0118] Furthermore, the field value statistics module also includes:

[0119] The fixed value statistics module is used to count the values ​​in the current field group. If all the values ​​in the current field group are equal, it is confirmed that the value distribution information represents that the value is a fixed value;

[0120] A variance calculation module is used to calculate the variance between consecutive values ​​if the values ​​in the current field group are not equal;

[0121] A continuous value statistics module, used for confirming that the value represented by the value distribution information is a continuous value if the variance is less than or equal to a first preset threshold;

[0122] The discrete value statistics module is used to confirm that the value represented by the numerical distribution information is a discrete value if the variance is greater than a first preset threshold.

[0123] Further, when it is detected that the target compression mode is fixed value compression, the field compression module 304 includes:

[0124] A first target value acquisition module, used to acquire the fixed number of occupied bits corresponding to the target compression method and the first target value of the current field group;

[0125] A first digit calculation module, used to calculate a first occupied digit of a first target value;

[0126] The fixed value field compression module is used to obtain a compressed field based on the fixed number of occupied bits, the first number of occupied bits and the first target value.

[0127] Further, when it is detected that the target compression mode is continuous value compression, the field compression module 304 includes:

[0128] A second target value acquisition module, used to acquire the fixed number of occupied bits corresponding to the target compression mode and the second target value of the current field group;

[0129] A first offset calculation module, used for extracting a first offset from a second target value, subtracting the first offset from all second target values, and obtaining a second target value after the offset;

[0130] A second digit calculation module, used for calculating the second occupied digit based on the maximum value of the second target value after the shift;

[0131] A subgroup deduplication module, configured to form a first subgroup using continuous and equal values ​​in the second target value after the shift, and remove groups equal to a default value from the first subgroup, where the default value is the mode of the second target value after the shift;

[0132] A third digit calculation module, used for obtaining the group containing the largest number of values ​​from the remaining first subgroups as the second subgroup, and calculating the third occupied digit of the second subgroup;

[0133] A segment record creation module, used to create a segment record for each remaining first subgroup, the segment record including the value and the number of values ​​contained in the remaining first subgroup;

[0134] The continuous value field compression module is used to obtain a compressed field based on the fixed number of occupied bits, the second number of occupied bits, the third number of occupied bits and the segmented records.

[0135] Further, when it is detected that the target compression mode is discrete value compression, the field compression module 304 includes:

[0136] A third target value acquisition module is used to acquire the fixed number of occupied bits corresponding to the target compression method, and remove duplicates from all values ​​in the current field group to obtain a third target value;

[0137] A second offset calculation module, used for taking the minimum value among the third target values ​​as the second offset, subtracting the second offset from all the third target values, and obtaining the offset third target values;

[0138] a fourth digit calculation module, used for calculating a fourth occupied digit based on a maximum value of the third target values ​​after the shift;

[0139] A fifth digit calculation module, used for mapping the offset third target value to a target index, calculating the fifth occupied digit of the target index, and listing the target index corresponding to each offset third target value;

[0140] The discrete value field compression module is used to obtain a compressed field based on the fixed number of occupied bits, the fourth number of occupied bits, the fifth number of occupied bits and the target index.

[0141] Furthermore, the file header calculation module 305 also includes:

[0142] The collection information extraction module is used to extract the message collection information of each message from a plurality of messages, and the message collection information includes the message collection time and the message collection quantity;

[0143] A timestamp acquisition module is used to use the maximum value of the message collection time as the start timestamp and the minimum value of the message collection time as the end timestamp;

[0144] A sixth digit calculation module, used to obtain the sixth occupied digit of the compressed field, and take the maximum value of the sixth occupied digit as the message occupied digit;

[0145] A collection interval acquisition module is used to extract the collection time interval from the message collection time, and use the majority of the collection time interval as the message collection interval;

[0146] A third sub-group acquisition module, configured to form a third sub-group by collecting adjacent messages whose time interval is less than a preset interval threshold, and obtain the number of groups of the third sub-group;

[0147] A time offset calculation module, used to calculate the time offset and the number of messages of each third sub-group, the time offset being obtained based on the difference between the first acquisition time and the start timestamp in the third sub-group;

[0148] The file header generation module is used to generate file header information, which includes a start timestamp, an end timestamp, the number of bits occupied by the message, a message collection interval, the number of groups, and the time offset and number of messages of each third sub-group.

[0149] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0150] refer to Figure 4 The embodiment of the present application further provides an electronic device, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0151] Memory 403, used for storing computer programs;

[0152] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:

[0153] Obtain several consecutive messages, parse each message, and obtain several fields in each message;

[0154] Get the classification information of the field, group the fields according to the classification information, and obtain the field grouping;

[0155] Extracting the value distribution information of each field group, and determining the target compression method of the current field group based on the value distribution information;

[0156] Compress the current field group according to the target compression method to obtain a compressed field;

[0157] Extracting message collection information from the message, and calculating and obtaining file header information according to the message collection information;

[0158] Assemble the file header information and compression fields to obtain the packaged file of the message.

[0159] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0160] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0161] The embodiment of the present application also provides a non-volatile readable storage medium. When the instructions in the non-volatile readable storage medium are executed by the processor of the electronic device, the electronic device can perform the various processes of the above-mentioned message reorganization and compression method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0162] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0163] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A message reassembly and compression method, characterized in that: The method comprises: Obtaining a plurality of consecutive messages, parsing each of the messages, and obtaining a plurality of fields in each of the messages; Acquire classification information of the field, group the fields according to the classification information, and obtain field groups; Extracting the value distribution information of each of the field groups, and determining the target compression mode of the current field group based on the value distribution information; Compressing the current field group according to the target compression mode to obtain a compressed field; Extracting message collection information from the message, and calculating and obtaining file header information according to the message collection information; Assemble the file header information and the compression field to obtain a packaged file of the message.

2. The method according to claim 1, characterized in that The compression method includes fixed value compression, continuous value compression and discrete value compression, and the extracting of the value distribution information of each field group and determining the target compression method of the current field group based on the value distribution information include: Counting the values ​​in the current field group to obtain the value distribution information of the current field group; If the value distribution information indicates that the value is a fixed value, confirming that the target compression mode of the current field grouping is fixed value compression; If the value distribution information indicates that the value is a continuous value, confirming that the target compression mode of the current field grouping is continuous value compression; If the numerical distribution information indicates that the numerical value is a discrete value, it is confirmed that the target compression method of the current field grouping is discrete value compression.

3. The method according to claim 2, characterized in that The counting of the values ​​in the current field group to obtain the value distribution information of the current field group includes: Counting the values ​​in the current field group, and if all the values ​​in the current field group are equal, confirming that the value distribution information indicates that the value is a fixed value; If the values ​​in the current field group are not equal, calculating the variance of the current field group; If the variance is less than or equal to a first preset threshold, confirming that the value distribution information indicates that the value is a continuous value; If the variance is greater than the first preset threshold, it is confirmed that the numerical distribution information indicates that the numerical value is a discrete value.

4. The method according to claim 1, characterized in that: In the case where it is detected that the target compression mode is fixed value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes: Obtaining a fixed number of occupied bits corresponding to the target compression mode and a first target value of the current field group; Calculate the first occupied bit number of the first target value; The compressed field is obtained based on the fixed number of occupied bits, the first number of occupied bits, and the first target value.

5. The method according to claim 1, characterized in that In the case where it is detected that the target compression mode is continuous value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes: Obtaining a fixed number of occupied bits corresponding to the target compression mode and a second target value of the current field group; Taking the minimum value among the second target values ​​as the first offset, subtracting the first offset from all the second target values ​​to obtain the offset second target values; Calculate a second number of occupied bits based on the maximum value of the second target values ​​after the shift; Using continuous and equal values ​​in the second target value after the shift to form a first subgroup, removing groups equal to a default value from the first subgroup, where the default value is the mode of the second target value after the shift; From the remaining first subgroups, a group containing the largest number of values ​​is obtained as a second subgroup, and a third number of occupied bits of the second subgroup is calculated; Creating a segment record for each of the remaining first subgroups, wherein the segment record includes a value and a number of values ​​included in the remaining first subgroups; The compressed field is obtained based on the fixed number of occupied bits, the second number of occupied bits, the third number of occupied bits and the segment record.

6. The method according to claim 1, characterized in that In the case where it is detected that the target compression mode is discrete value compression, compressing the current field group according to the target compression mode to obtain a compressed field includes: Obtaining a fixed number of occupied bits corresponding to the target compression method, and removing duplicates from all values ​​in the current field group to obtain a third target value; Taking the minimum value among the third target values ​​as the second offset, subtracting the second offset from all the third target values ​​to obtain the offset third target values; Calculate a fourth number of occupied bits based on the maximum value of the third target values ​​after the shift; Mapping the offset third target value to a target index, calculating the fifth occupied bit number of the target index, and listing the target index corresponding to each offset third target value; The compressed field is obtained based on the fixed number of occupied bits, the fourth number of occupied bits, the fifth number of occupied bits, and the target index.

7. The method according to claim 1, characterized in that The extracting the message collection information from the message, and calculating and obtaining the file header information according to the message collection information, includes: Extracting message collection information of each message from the plurality of messages, wherein the message collection information includes message collection time and message collection quantity; The maximum value of the message collection time is used as the start timestamp, and the minimum value of the message collection time is used as the end timestamp; Obtaining the sixth number of occupied bits of the compression field, and taking the maximum value of the sixth number of occupied bits as the number of occupied bits of the message; Extracting a collection time interval from the message collection time, and taking the mode of the collection time intervals as the message collection interval; Adjacent messages whose collection time interval is less than a preset interval threshold are used to form a third sub-group, and the number of groups of the third sub-group is obtained; Calculating a time offset and a number of messages of each third subgroup, wherein the time offset is obtained based on a difference between a first acquisition time in the third subgroup and the start timestamp; Generate the file header information, wherein the file header information includes the start timestamp, the end timestamp, the number of bits occupied by the message, the message collection interval, the number of groups, the time offset of each third sub-group, and the number of messages.

8. A message reassembly and compression device, characterized in that: The device comprises: A message parsing module, used to obtain a plurality of consecutive messages, parse each of the messages, and obtain a plurality of fields in each of the messages; A field grouping module, used for obtaining classification information of the fields, grouping the fields according to the classification information, and obtaining field groups; A compression mode determination module, used for extracting the value distribution information of each field group, and determining the target compression mode of the current field group based on the value distribution information; A field compression module, used for compressing the current field group according to the target compression mode to obtain a compressed field; A file header calculation module, used to extract message collection information from the message, and calculate and obtain file header information according to the message collection information; The compression assembly module is used to assemble the file header information and the compression field to obtain a packaged file of the message.

9. An electronic device, comprising: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the message reassembly and compression method described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, when instructions in the non-volatile readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the message reassembly and compression method as described in any one of claims 1 to 7.

Images