A data processing method, device, vehicle, cloud service platform, and system

Abstract

The application provides a data processing method, device, vehicle, cloud service platform and system, and belongs to the technical field of data processing. The data processing method comprises: performing lightweight processing on a target string key name in a target data frame collected on a target device; generating a compressed target data frame corresponding to the target data frame based on inter-frame difference information of adjacent target data frames; and uploading the compressed target data frame to a cloud service platform. The application aims to reduce the amount of data uploaded to the cloud, and can be applied to intelligent vehicles or intelligent network connection scenarios.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and more specifically, to a data processing method, device, vehicle, cloud service platform, and system. Background Technology

[0002] With the development of technology, terminal devices can usually upload data to the cloud platform during operation. For example, with the rapid development of automotive technology, intelligent connected vehicle technology has now entered the stage of large-scale implementation. The number of vehicles connected to the cloud platform is rapidly evolving to millions or even tens of millions. Moreover, with the enrichment of the single-vehicle perception hardware matrix and the increase in sampling frequency, vehicles need to upload massive amounts of data to the cloud platform during operation.

[0003] However, the current common method for vehicle-to-everything (V2X) data transmission is to compress and report the data at fixed intervals. But as the amount of data from connected vehicles and the amount of data collected by the vehicle end increase, adopting the full data reporting mode will result in a huge amount of data transmission and bandwidth consumption. Summary of the Invention

[0004] This application provides a data processing method, device, vehicle, cloud service platform, and system, aiming to reduce the amount of data uploaded to the cloud.

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising: Lightweighting the target string key name in the target data frame collected on the target device includes obtaining the lightweight identifier corresponding to the target string key name in the target data frame, and replacing the target string key name in the target data frame with the lightweight identifier corresponding to the target string key name, wherein any target string key name has a corresponding lightweight identifier preset. Based on the inter-frame difference information of adjacent target data frames, a compressed target data frame corresponding to the target data frame is generated; Upload the compressed target data frame to the cloud service platform.

[0006] Optionally, generating a compressed target data frame corresponding to the target data frame based on the inter-frame difference information of adjacent target data frames includes: Based on the frame sequence number of the target data frame, the target data frames are grouped together, and each group of target data frames includes one key target data frame and at least one residual target data frame. The full data of the key target data frame is compressed to generate a compressed target data frame corresponding to the key target data frame; Differential coding is performed on any residual target data frame to determine the differential data of the residual target data frame, and the differential data is compressed to generate the compressed target data frame corresponding to the residual target data frame. The differential data is used to characterize the inter-frame difference information between the residual target data frame and its previous target data frame.

[0007] Optionally, grouping the target data frames according to their frame sequence numbers includes: Based on the increasing order of the frame sequence number, determine in turn whether the frame sequence number of any target data frame satisfies the condition that its frame sequence number modulo zero with respect to the preset key frame interval parameter. If the modulus is zero, the target data frame is a critical target data frame; If the modulus is not zero, the target data frame is a residual target data frame; Any critical target data frame is grouped together with each residual target data frame that follows it.

[0008] Optionally, when compressing the full data of the key target data frame or the differential data of any residual target data frame, compression is performed based on a preset compression algorithm. The preset compression algorithm includes the ZSTD standard compression algorithm.

[0009] Optionally, uploading the compressed target data frame to the cloud service platform includes: The fragment topic corresponding to the target device is determined based on the unique identifier of the target device; Based on the Message Queue Telemetry Transport (MQTT) protocol, the compressed target data frame is published to the shard topic, so that the cloud service platform can obtain the compressed target data frame by subscribing to the shard topic.

[0010] Secondly, embodiments of this application provide a data processing method applied to a cloud service platform, the method comprising: Acquire compressed target data frames from the target device; The compressed target data frame is decompressed to restore the target data frame of the target device.

[0011] Optionally, acquiring the compressed target data frame of the target device includes: Any Flink task subscribes to an MQTT shard topic and sequentially retrieves each group of compressed target data frames from the target device, storing them in the buffer corresponding to that target device.

[0012] Optionally, decompressing the compressed target data frame to recover the target data frame of the target device includes: In the buffer corresponding to the target device, when the compressed target data frame is identified as a compressed target data frame corresponding to the key target data frame, the compressed target data frame is decompressed to obtain the key target data frame. When the compressed target data frame is identified as the compressed target data frame corresponding to the residual target data frame, the compressed target data frame is decompressed to obtain the differential data of the residual target data frame. Based on the differential data of the residual target data frame and its previous target data frame, the residual target data frame is obtained.

[0013] Optionally, the method further includes: Each set of target data frames obtained from the decompression of the target device is sent to a Kafka message queue for temporary storage; Each set of compressed target data frames from the target device is sent to a Kafka message queue for persistent storage.

[0014] Optionally, the method further includes: Based on each set of target data frames temporarily stored, anomaly detection is performed on the target monitoring data preset by the target device. When it is determined that there is an anomaly in the target monitoring data, an anomaly control command is sent to the target device. The anomaly detection includes one or more of the following: threshold static detection, statistical analysis detection, and large model inference detection. The anomaly control commands include one or more of the following: basic snapshot capture command, fine log return command, full raw log capture command, terminal system restart command, and parameter adjustment command.

[0015] Optionally, the step of performing anomaly detection on the target monitoring data preset by the target device based on each set of target data frames temporarily stored includes: Determine whether the target monitoring data in each set of target data frames temporarily stored is greater than the threshold corresponding to the target monitoring data; When the target monitoring data is greater than the threshold corresponding to the target monitoring data, an anomaly score of the target monitoring data in the sliding window is calculated, wherein the anomaly score is determined based on the standard deviation and / or slope change of the target monitoring data in the sliding window; When the anomaly score of the target monitoring data exceeds the preset confidence interval of the target monitoring data, a multidimensional time series correlation analysis is performed on the target monitoring data and the preset associated data of the target monitoring data to determine whether there is anomaly in the target monitoring data.

[0016] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the data processing method as described in the first aspect of the embodiments.

[0017] Fourthly, embodiments of this application provide a vehicle for performing the data processing method described in the first aspect of the embodiments, or including the electronic equipment described in the third aspect of the embodiments.

[0018] Fifthly, embodiments of this application provide a cloud service platform, which is used to execute the data processing method described in the second aspect of the embodiments.

[0019] Sixthly, embodiments of this application provide a data processing system, which includes the vehicle described in the fourth aspect of the embodiment and the cloud service platform described in the fifth aspect of the embodiment.

[0020] Beneficial effects: In the data processing method provided in this application embodiment, the target string key name in the target data frame collected on the target device is first lightweighted by replacing the target string key name in the target data frame with the lightweight identifier corresponding to the target string key name, thereby reducing the size of the data frame. Then, based on the inter-frame difference information of adjacent target data frames, a compressed target data frame corresponding to the target data frame is generated and uploaded to the cloud service platform. Compared with the full data compression and upload of each data frame, compression based on the inter-frame difference information of the target data frame can significantly reduce the amount of data uploaded by the device to the cloud, and can be applied in intelligent vehicle or intelligent connected vehicle scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the steps of a data processing method proposed in an embodiment of this application; Figure 2 This is a flowchart illustrating the steps for generating a compressed target data frame according to an embodiment of this application; Figure 3 This is a flowchart of the steps of a data processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the Flink distributed stream processing architecture proposed in one embodiment of this application; Figure 5 This is a functional block diagram of a data processing apparatus according to an embodiment of this application; Figure 6 This is a functional block diagram of a data processing apparatus provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device according to an embodiment of this application; Figure 8 This is a schematic diagram of a readable storage medium proposed in an embodiment of this application; Figure 9 This is a schematic diagram of a computer program product proposed in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] SOC: State of Charge, battery state of charge / remaining capacity; CAN: Controller Area Network; IMU: Inertial Measurement Unit; GNSS track: Global Navigation Satellite System; ID: Identifier, identifier or identity identifier; MQTT: Message Queuing Telemetry Transport (a lightweight Internet of Things communication protocol).

[0026] With the development of technology, terminal devices can usually upload data to the cloud platform during operation. The cloud platform can then monitor, analyze, and optimize based on this data. For example, with the rapid development of automotive technology, intelligent connected vehicle technology has now entered the stage of large-scale implementation, and the number of vehicles connected to the cloud platform is rapidly evolving to millions or even tens of millions.

[0027] With the enrichment of the single-vehicle perception hardware matrix and the increase in sampling frequency, massive amounts of high-frequency time-series data and heterogeneous perception data are continuously generated during vehicle operation. Information that must be reported to the cloud in real time must be included. This data not only covers the high-concurrency vehicle trajectory, power chassis and three-electric system status, but also includes extremely bandwidth-intensive visual images and radar point clouds.

[0028] When faced with scenarios involving a massive number of vehicles accessing the network concurrently, the traditional "end-to-cloud" direct connection upload or extensive aggregation architecture has revealed serious limitations. With the leapfrog growth in management scale, the existing architecture, lacking effective data compression mechanisms and intelligent transmission scheduling strategies, will directly lead to a sharp overdraft of network bandwidth and an exponential increase in cloud storage costs.

[0029] The current common data transmission method for vehicle-to-everything (V2X) networks is the general technical path of "fixed-period full reporting - general compression - message queue access". The core implementation logic is as follows: the vehicle terminal packages all sensor parameters into a complete independent frame at fixed intervals, sends it to the cloud via the MQTT protocol, transmits messages via Kafka, and performs intermediate processing by microservices. Finally, the original data or simply compressed data is persistently stored. This approach is built around the data reporting access and basic message flow, without fully combining the time-series correlation and high-frequency, low-volume characteristics of massive V2X data for in-depth and specialized optimization.

[0030] For example, the common shortcomings of vehicle-to-everything (V2X) data transmission methods are as follows: 1. High data redundancy and huge bandwidth consumption.

[0031] When using the full data reporting mode, vehicles treat each data packet as an isolated entity, failing to utilize the continuity of time series and sending all status data to the cloud. Taking telemetry data (SOC, temperature, some system status bits, etc.) as an example, its changes are minimal in a short period of time. Continuously sending full data results in a large amount of bandwidth being wasted on repetitive data. If millions of vehicles send full data concurrently, it can easily cause I / O blockage at the cloud access gateway. Bandwidth costs increase linearly with the number of vehicles, making it difficult to achieve large-scale cost control.

[0032] 2. The compression efficiency is limited and it is not optimized for the characteristics of vehicle network data.

[0033] Currently, most compression algorithms (such as GZIP, LZ4 and ZSTD standard compression) are used. Although these compression algorithms are highly versatile, they are not well optimized for the specific characteristics of vehicle network data (temporal correlation, field repetition), resulting in limited compression effects. They also lack inter-frame predictive differential compression mechanisms, making it impossible to achieve efficient transmission of residual frames.

[0034] 3. Lack of effective data sharding and load balancing mechanisms.

[0035] Currently, most systems employ a single storage strategy or fixed-range partitioning, without a refined global sharding design for the unique vehicle identifier. In high-concurrency scenarios, such as those involving millions of vehicles, data is concentrated on a few nodes, failing to leverage the horizontal scalability of distributed clusters, resulting in a prominent single-point bottleneck at the consumer end.

[0036] 4. Poor timeliness of operation and maintenance response, and difficulty in fully reproducing the fault scene.

[0037] In the current operation and maintenance process, when a vehicle malfunctions, it usually relies on manual monitoring and alarms. Once an anomaly is detected, manual commands need to be issued to capture logs, resulting in a large response delay. At the same time, critical logs may be lost due to cyclic overwriting, making it impossible to reproduce the fault scene. Secondly, the current detection method uses a single threshold alarm, which is prone to missing hidden degradation or generating a large number of false alarms, failing to meet the needs of real-time and accurate operation and maintenance.

[0038] To address the transmission and maintenance bottlenecks caused by uploading massive amounts of high-frequency vehicle-to-everything (V2X) data to the cloud, this application aims to resolve the following core business and technical pain points: 1. Transmission bandwidth and storage cost: In the traditional full-data reporting mode, millions of vehicles send full data concurrently, resulting in huge cloud bandwidth consumption and storage costs that increase linearly with the number of vehicles, making it difficult to achieve large-scale management and control. 2. Reliability of transmission in weak networks: When the network fluctuates, the risk of full data accumulation or packet loss is extremely high, so it is necessary to improve the transmission arrival rate of high-frequency data in weak network environments; 3. Operation and maintenance response timeliness: Traditional fault handling that relies on manual alarms and manual issuance of commands to capture logs has a large delay.

[0039] Based on this, embodiments of this application provide a data processing method that can reduce the amount of data transmitted when uploading vehicle network data to the cloud.

[0040] Reference Figure 1 The diagram illustrates a flowchart of a data processing method provided in an embodiment of this application. The method may specifically include the following steps: S101: Lightweight the target string key name in the target data frame collected on the target device.

[0041] In actual implementation, the target device can be any terminal device that needs to upload data to the cloud. For example, the target device can be a vehicle, robot, mobile terminal or other multi-sensor device, etc. It can be set according to the needs of actual application. This embodiment does not impose any restrictions.

[0042] In actual implementation, the target data is the data that needs to be uploaded to the cloud or cloud service platform. For example, when the target device is a vehicle, the target data includes, but is not limited to, the vehicle's CAN data, IMU data, GNSS trajectory, positioning fusion results, radar target list, ultrasonic data, and perception results. The data that needs to be uploaded to the cloud or cloud service platform can be defined according to the needs of the actual application. This application embodiment does not impose any restrictions.

[0043] The frame rate of the target data frame is related to the preset target data acquisition period. For example, one acquisition period of the target data corresponds to one target data frame. The acquisition period and upload period of different data can be set according to the actual application requirements. This application embodiment does not impose any restrictions.

[0044] After acquiring the target data frame on the target device, the target data frame is first subjected to protocol lightweighting to compress the size of the data frame from the source.

[0045] In one feasible implementation, during the process of co-lightweighting the target string key name in the target data frame, a lightweight identifier corresponding to the target string key name in the target data frame can be obtained, wherein any target string key name has a corresponding lightweight identifier preset, and then the target string key name in the target data frame is replaced with the lightweight identifier corresponding to the target string key name.

[0046] Specifically, a lightweight identifier corresponding to any target string key name can be pre-set and stored on the target device.

[0047] For example, if the target device is a vehicle, a globally unified field mapping dictionary can be loaded on the vehicle side, and then the lightweight identifier corresponding to the target string key name of the target data frame can be determined by querying the field mapping dictionary.

[0048] The field mapping dictionary includes a lightweight identifier corresponding to any string key name. The lightweight identifier can be a short hexadecimal ID identifier, which can map the long string key names in the target data frame to a fixed-length short hexadecimal ID identifier, thereby reducing the size of the data frame.

[0049] For example, "vehicle_speed" is mapped to "0x01", "engine_rpm" is mapped to "0x02", "battery_level" is mapped to "0x03", "soc_temperature" is mapped to "0x04", and so on.

[0050] In practice, the field mapping dictionary can also be versioned for dynamic updates. For example, when it is necessary to add or modify field mappings, the field mapping relationships in the field mapping dictionary can be added or modified in the cloud or cloud service platform, the version of the field mapping dictionary can be updated, and the latest version of the field mapping dictionary can be distributed to each vehicle. Any vehicle can automatically download and apply the latest version of the field mapping dictionary.

[0051] In practice, when uploading any data frame to the cloud or cloud service platform, the version number of the field mapping dictionary can be included in the frame header of the data frame to ensure that the cloud or cloud service platform can parse and recover the string key name based on the same version of the field mapping dictionary.

[0052] S102: Generate a compressed target data frame corresponding to the target data frame based on the inter-frame difference information of adjacent target data frames.

[0053] Considering that target data, especially frequently uploaded target data, changes relatively little in a short period of time, if each data frame is fully compressed and uploaded, the uploaded data redundancy will be extremely high, resulting in huge upload bandwidth and cloud storage costs, and data accumulation or packet loss is very likely to occur when the network fluctuates.

[0054] Therefore, in the method provided in this application embodiment, after lightweighting the target string key name in the target data frame collected on the target device, further inter-frame prediction is performed, and compression is performed based on the inter-frame difference information of adjacent target data frames, which can significantly reduce the amount of data uploaded by the device such as a bicycle.

[0055] Reference Figure 2 This document illustrates a flowchart of the steps for generating a compressed target data frame according to an embodiment of this application. In one feasible implementation, generating the compressed target data frame corresponding to the target data frame based on the inter-frame difference information of adjacent target data frames may include the following sub-steps: A1: Group the target data frames according to their frame sequence numbers.

[0056] In actual implementation, the frame number can be configured to increment according to the acquisition order of each target data frame. The range of the frame number can be set according to the actual application requirements. For example, the range of the frame number can be 0-239. Then, the target data frame is serialized into a hexadecimal data stream. Each target data frame is grouped according to the frame number. Each group of target data frames includes one key target data frame and at least one residual target data frame.

[0057] Among them, the key target data frame is the frame containing complete data content in the subsequent compression process, and can be used as a reference frame for the subsequent residual target data frame; the residual target data frame refers to the frame that is compressed in the subsequent compression process to compress the difference information between it and the previous target data frame, and needs to be combined with the previous target data frame to recover the complete data content.

[0058] Specifically, based on the increasing order of the frame sequence numbers, it is determined whether the frame sequence number of any target data frame satisfies the condition that its frame sequence number modulo zero with respect to the preset key frame interval parameter.

[0059] That is, determine whether the frame sequence number of any target data frame satisfies N mod K = 0; Where N is the frame number; K is the keyframe interval parameter; and mod is the modulo operation, which represents the remainder after dividing two integers.

[0060] If the modulus is zero, the target data frame is a critical target data frame; If the modulus is not zero, the target data frame is a residual target data frame.

[0061] Any critical target data frame is grouped together with each residual target data frame that follows it.

[0062] In actual implementation, the keyframe interval parameter K can be set according to the needs of the actual application, or it can be distributed to the target device from the cloud or cloud service platform.

[0063] For example, the value of the key frame interval parameter K can be dynamically adjusted according to data characteristics and network conditions. When the variation between a certain type of data frames is small, the K value corresponding to that type of data can be increased to reduce the number of key target data frames and improve the compression ratio. When the variation between a certain type of data frames is large, the K value corresponding to that type of data can be decreased to increase the number of key target data frames, thereby ensuring the accuracy of data recovery. In scenarios where the K value changes dynamically, the K value corresponding to any target data frame must be consistent between the target device and the cloud. For example, when uploading any data frame to the cloud or cloud service platform, the K value can also be included in the frame header of that data frame.

[0064] For example, the value of the keyframe interval parameter K can be between 20 and 50, and can be optimized according to the actual application scenario; taking a keyframe interval parameter K of 30 as an example, that is, the first frame in every 30 target data frames is the key target data frame, and the remaining 29 frames are residual target data frames.

[0065] A2: Compress the full data of the key target data frame to generate a compressed target data frame corresponding to the key target data frame.

[0066] The full data of a critical target data frame refers to the complete data payload portion of the critical target data frame.

[0067] In actual implementation, the full data of the key target data frame can be compressed based on a preset compression algorithm. The preset compression algorithm includes the ZSTD standard compression algorithm, which has the advantages of high compression ratio, fast compression speed and fast decompression speed, and can achieve a balance between compression ratio and compression speed.

[0068] A3: Perform differential encoding on any residual target data frame to determine the differential data of the residual target data frame, and compress the differential data to generate the compressed target data frame corresponding to the residual target data frame.

[0069] Specifically, during the differential encoding of the residual target data frame, a bitwise XOR operation is performed between the residual target data frame and its previous target data frame to determine the differential data of the residual target data frame. The differential data is used to characterize the inter-frame difference information between the residual target data frame and its previous target data frame.

[0070] Since the target data, especially the frequently uploaded target data, changes little in a short period of time, the differential data contains a large number of 0 values ​​(indicating that the data is the same), resulting in better compression. Therefore, compression speed can be prioritized to meet real-time requirements.

[0071] For example, the formula for differential coding is shown below:

[0072] in, For the difference data bytes, For the residual target data frame of the first bytes, The first frame of the previous frame of the residual target data frame One byte, ⊕ represents the XOR operation.

[0073] For example, assuming the 60 target data frames are sequentially named: target data frame 0, target data frame 1, target data frame 2... target data frame 59, and the keyframe interval parameter K is set to 30, then: Among them, target data frame 0 is the key target data frame in a group, and target data frames 1-29 are the residual target data frames in the same group; For target data frame 1, it is XORed with the previous frame, i.e., key target data frame 0, to determine its own differential data; For target data frame 2, it is XORed with the previous frame, i.e. target data frame 1, to determine its own differential data; ... For target data frame 29, it is XORed with the previous frame, i.e. target data frame 28, to determine its own differential data. Target data frame 30 is a key target data frame in another group, and target data frames 31-59 are residual target data frames in the same group; For target data frame 31, it is XORed with the previous frame, i.e. key target data frame 30, to determine its own differential data; For target data frame 32, it is XORed with the previous frame, i.e. target data frame 31, to determine its own differential data. ... For target data frame 59, a bitwise XOR operation is performed with the previous frame, i.e., target data frame 58, to determine its own differential data.

[0074] For any residual target data frame, instead of compressing the full data of the frame, the differential data of the frame is compressed, which can significantly reduce the amount of data uploaded.

[0075] When compressing the differential data of any residual target data frame, compression is performed based on a preset compression algorithm; the preset compression algorithm includes the ZSTD standard compression algorithm.

[0076] In this embodiment of the application, the data frame is a structured state data packet that is periodically uploaded by the target device, such as the vehicle. The data structure of each frame consists of a frame header and a data payload. The frame header includes the frame sequence number and the protocol version.

[0077] For example, the data structure of a data frame is as follows in sequence: A 1-byte frame sequence number, the value of which can be 0x00~0xEF, i.e. decimal 0~239; The protocol version is 2 bytes. The protocol version may include the version number of the field mapping dictionary, or it may include the version number of the field mapping dictionary and the keyframe interval parameter K value corresponding to the data frame. The data payload consists of N bytes, which includes the actual data content. The field lengths of the data payload are variable.

[0078] In practice, differential coding and compression can be performed only on the data payload. The starting field of the data payload in the compressed data frame can also include the identifier of the preset compression algorithm, so that the cloud can parse the identifier of the compression algorithm and call the decompression operator corresponding to the compression algorithm.

[0079] For example, after compression using the ZSTD standard compression algorithm, the starting field of the data payload can be a 4-byte ZSTD file magic number: 0x28b52ffd. The cloud can determine which ZSTD decompression operator to call for decompression by parsing the starting field of the data payload.

[0080] When the frame header of a data frame is not compressed, the cloud can quickly locate the frame sequence number and protocol version without decompression, which facilitates frame routing and version compatibility. At the same time, the protocol version can be parsed by the cloud independently of the data content.

[0081] For example, for two target data frames continuously collected by the vehicle: target data frame 180 (frame number 0xb4) and target data frame 181 (frame number 0xb5), target data frame 180 is the critical target data frame 180, and target data frame 181 is the residual target data frame 181. The two frames of data are serialized and encoded into hexadecimal data after being lightweighted by the protocol.

[0082] The data payload of the key target data frame 180 is as follows: “12 5e 6b 59 7e 06 37 1f 3d 11 00 00 01 96 84 83 6d b6 02 80 01 ff ffff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffff ff ff ff ff..."; The data payload portion of residual target data frame 181 is as follows: “12 5e 6b 59 7e 06 37 1f 3d 11 00 00 01 96 84 83 71 9f 02 80 01 ff ffff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffff ff ff ff ff...".

[0083] The data payload portion of both target data frames is 342 bytes in length.

[0084] After compressing the data payload portion of the critical target data frame 180 using the ZSTD standard compression algorithm, and combining it with the frame header of the critical target data frame 180, the partial byte stream of the compressed target data frame 180 corresponding to the critical target data frame 180 is shown below: "b4 10 00 28 b5 2f fd 60 56 00 85 03 00 e4 04 12 5e 6b 59 7e 06 37 1f3d ..." The total length of the compressed target data frame 180 after compression is 125 bytes, of which: The first three bytes, “b4 10 00”, are the frame header, where b4 is the frame sequence number and “10 00” is the protocol version number. The fourth byte onwards is the compressed data payload, and the first four bytes of the data payload, “28 b5 2f fd”, are the ZSTD file magic number. The subsequent bytes are the compressed result of the full data of the data payload of the critical target data frame 180.

[0085] When the cloud parses the compressed target data frame 180, it first extracts the first 3 bytes of the frame header "b4 10 00". Then, starting from the 4th byte, it identifies the ZSTD file magic number and calls the ZSTD decompression operator to decompress the compressed result, obtaining a total of 342 bytes of data payload. Then, it adds back the 3 bytes of frame header, thus obtaining the complete 345 bytes of hexadecimal data of the target data frame 180. Then, based on the field mapping dictionary, the target string key name can be restored to obtain the target data frame 180.

[0086] For residual target data frame 181, perform a bitwise XOR operation between residual target data frame 181 and the data payload portion of key target data frame 180 to determine the differential data of residual target data frame 181: "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 1c 29 00 00 00 00 0000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000 00 00 00 00...".

[0087] In this 342-byte differential data, most bytes are 0x00, with a non-zero value "1c 29" appearing only in bytes 17-18. Since most fields did not change in the two consecutive acquisitions, the differential result is 0.

[0088] After compressing the differential data of residual target data frame 181 using the ZSTD standard compression algorithm, and combining it with the frame header of residual target data frame 181, the partial byte stream of the compressed target data frame 181 corresponding to residual target data frame 181 is shown below: "b5 10 00 28 b5 2f fd 60 56 00 6d 00 00 20 00 1c..." The total length of the byte stream of the compressed target data frame 181 corresponding to the residual target data frame 181 is only 27 bytes, of which: The first three bytes, “b5 10 00”, are the frame header, where b5 is the frame sequence number and “10 00” is the protocol version number. The fourth byte onwards is the compressed data payload, and the first four bytes of the data payload, “28 b5 2f fd”, are the ZSTD file magic number. The subsequent bytes are the compression result of the differential data of the residual target data frame 181.

[0089] When the cloud parses the compressed target data frame 181 after the residual target data frame 181 is compressed, it first extracts the first 3 bytes of the frame header "b4 10 00", and then starts from the 4th byte to identify the ZSTD file magic number and call the ZSTD decompression operator to decompress the compression result, obtaining 342 bytes of differential data. After performing a bitwise XOR operation on this differential data and the data payload part of the key target data frame 180, the data payload part of the residual target data frame 181 is obtained. Then, the 3 bytes of frame header are added back, and the complete 345 bytes of hexadecimal data of the residual target data frame 181 can be obtained. Then, based on the field mapping dictionary, the target string key name can be restored to obtain the target data frame 181.

[0090] The method provided in this application embodiment, based on protocol lightweighting, namely string key name lightweighting and inter-frame prediction compression, can significantly reduce the amount of uploaded data.

[0091] For example, if 30 target data frames are continuously collected on a real vehicle, the original data length of each target data frame is about 690 bytes. After the key target data frames are compressed using hexadecimal encoding and the ZSTD standard compression algorithm, the total length of the compressed target data frame byte stream is 125 bytes, and the compression ratio reaches 5.5:1.

[0092] The first residual target data frame after the key target data frame has a very high proportion of 0 bytes in the inter-frame prediction differential data. After compression by the ZSTD standard compression algorithm, the total length of the byte stream of its compressed target data frame is only 26 bytes, with a compression ratio of 26.5:1. The statistical average compression ratio of the entire group of 30 frames can reach 20:1, which significantly reduces the amount of data uploaded by the vehicle.

[0093] S103: Upload the compressed target data frame to the cloud service platform.

[0094] In practice, multiple MQTT shards can be pre-created on the cloud service platform. MQTT is a lightweight message transmission protocol based on the publish / subscribe paradigm, which is suitable for low-bandwidth, high-latency network environments.

[0095] In one feasible implementation, the process of uploading the compressed target data frame to the cloud service platform may include: First, the fragment topic corresponding to the target device is determined based on the unique identifier of the target device.

[0096] The unique identifier of a target device is a unique identifier for that device. For example, when the target device is a vehicle, the unique identifier of the vehicle includes, but is not limited to, the vehicle ID or vehicle identification number.

[0097] The hash value is obtained by hashing the unique identifier of the target device. For example, the CRC32 hash algorithm can be used for hash calculation. In actual implementation, other hash algorithms can also be used. This application does not limit the implementation.

[0098] Then, the hash value is modulo the total number of MQTT shards to determine the shard topic corresponding to the target device.

[0099] For example, taking a vehicle as the target device, the formula for determining the fragmented topics is as follows:

[0100] in, This is the vehicle's unique identifier; P is the fragment topic number; CRC32 is a 32-bit cyclic redundancy check hash function; mod is the modulo operation. This represents the total number of fragments.

[0101] Finally, based on the Message Queuing Telemetry Transport (MQTT) protocol, the compressed target data frame is published to the shard topic, so that the cloud service platform can obtain the compressed target data frame by subscribing to the shard topic.

[0102] The fragment topic determines the routing method of compressed target data frames for the target device. Moreover, the fragment topic is determined based on the unique identifier of the target device, which ensures that all data frames of the same target device, such as the same vehicle, can be published to the same fragment topic, thus guaranteeing the order and consistency of data processing.

[0103] Meanwhile, by determining the sharding topic corresponding to each target device based on the unique identifier of the target device, load balancing can also be achieved when transmitting data based on the MQTT protocol.

[0104] In actual implementation, as the number of target devices increases, the number of sharded topics can be expanded according to the needs of actual applications, and this application embodiment does not impose any restrictions.

[0105] Reference Figure 3 This document illustrates a flowchart of the data processing method provided in an embodiment of this application. The method is applied to a cloud service platform and may specifically include the following steps: S201: Obtain the compressed target data frame of the target device.

[0106] Specifically, in scenarios where data transmission is based on the MQTT protocol, the cloud service platform can use the Flink distributed stream processing engine to achieve high-throughput real-time processing of data from the target device. Flink is a computing framework specifically designed for processing real-time data streams, featuring low latency and high throughput. Flink supports Exactly-Once semantics to ensure the accuracy of data processing and avoid data loss or duplication.

[0107] Reference Figure 4 The diagram illustrates the architecture of Flink distributed stream processing provided in this application embodiment. Flink distributed stream processing includes multiple Flink tasks, each of which subscribes to an MQTT shard topic. This ensures that all data frames from the same target device, such as the same vehicle, can be processed by the same Flink task, further guaranteeing the order and consistency of data processing.

[0108] Any Flink task subscribes to an MQTT shard topic, and based on the Flink task's MQTT Source adapter, sequentially obtains each group of compressed target data frames from the target device. Based on the shard topic subscribed to by the Flink task, the unique identifier of the target device can be parsed through the ID resolver. For example, if the target device is a vehicle, the unique identifier of the vehicle can be parsed based on the vehicle ID resolver.

[0109] Then, the acquired compressed target data frames are buffered. Specifically, a corresponding buffer is established for each target device based on the unique identifier of the target device, and each set of compressed target data frames is stored in the buffer corresponding to the target device.

[0110] In actual implementation, the buffer corresponding to each target device can be a circular buffer or other buffers, such as a queue buffer. This application embodiment does not impose any restrictions.

[0111] For example, based on the frame number of the compressed target data frame and the key frame interval parameter K, it can be determined whether any compressed target data frame is the compressed target data frame corresponding to the key target data frame or the compressed target data frame corresponding to the residual target data frame.

[0112] In actual implementation, taking a circular buffer as an example, a circular buffer can be a first-in-first-out data structure with a fixed capacity that will not grow indefinitely. When the buffer is full, new data will overwrite the oldest data. In this embodiment, the capacity of the circular buffer can be 240 frames, that is, each circular buffer can store 240 data frames.

[0113] The circular buffer is implemented using an array. To avoid memory leaks, the access time complexity is O(1), which is fast. Data frames can be quickly found and updated by frame sequence number. At the same time, it can ensure that the same set of compressed target data frames of a target device are processed sequentially in the same task to restore the target data frames.

[0114] S202: Decompress the compressed target data frame to restore the target data frame of the target device.

[0115] In one feasible implementation, decompressing the compressed target data frame to recover the target data frame of the target device includes: In the buffer corresponding to the target device, such as a circular buffer, when the compressed target data frame is identified as the compressed target data frame corresponding to the key target data frame, the compressed target data frame is decompressed to obtain the key target data frame.

[0116] For example, when decompressing the compressed target data frame corresponding to the key target data frame, the first 3 bytes of the frame header are extracted first. Then, starting from the 4th byte, the identifier of the compression algorithm is identified to call the decompression operator corresponding to the compression algorithm. For example, when the magic number of the ZSTD file is identified, the ZSTD decompression operator is called to decompress the compression result to obtain the decompressed data payload. Then, the 3-byte frame header is added back to obtain the complete hexadecimal data of the key target data frame. Then, based on the field mapping dictionary, the target string key name can be restored to obtain the corresponding target data frame.

[0117] When the compressed target data frame is identified as the compressed target data frame corresponding to the residual target data frame, the compressed target data frame is decompressed to obtain the differential data of the residual target data frame. Based on the differential data of the residual target data frame and its previous target data frame, the residual target data frame is obtained.

[0118] For example, when parsing the compressed target data frame corresponding to the residual target data frame, the first 3 bytes of the frame header are extracted first. Then, starting from the 4th byte, the identifier of the compression algorithm is identified to call the decompression operator corresponding to the compression algorithm. For example, when the magic number of the ZSTD file is identified, the ZSTD decompression operator is called to decompress the compression result, and the decompressed data payload part, i.e., the differential data, is obtained. Then, the differential data is XORed with the data payload part of the previous frame to obtain the data payload part of the residual target data frame. Then, the 3-byte frame header is added back to obtain the complete hexadecimal data of the residual target data frame. Then, the target string key name can be restored based on the field mapping dictionary to obtain the corresponding target data frame.

[0119] For example, the bitwise XOR operation in the decompression process is as follows:

[0120] in, For the residual target data frame of the first bytes, For the difference data bytes, The first frame of the previous frame of the residual target data frame One byte, ⊕ represents the XOR operation.

[0121] In one feasible implementation, a Kafka message queue can also be established through an asynchronous Kafka producer. Kafka is a high-performance distributed messaging system that manages messages by topic classification, supports producers to publish messages to specified partitions and consumers to subscribe and consume messages by offset. It is a distributed stream processing engine used for state calculation of bounded and unbounded data streams.

[0122] The Kafka message queue includes a parallel real-time parsing stream and a compressed archiving stream. Each set of target data frames from the target device, after decompression, can be sent to the real-time parsing stream of the Kafka message queue for temporary storage. Each set of compressed target data frames from the target device can be packaged and sent to the compressed archiving stream of the Kafka message queue for persistent storage, thus balancing data real-time performance and storage efficiency.

[0123] When a set of compressed target data frames in the circular buffer has been decompressed and sent to the compressed archive stream of the Kafka message queue for persistent storage, the space of the circular buffer is released, and the next set of compressed target data frames is obtained for decompression and dual storage.

[0124] In actual implementation, the data storage format of the real-time parsing stream is JSON or Protobuf. Subsequently, the data can be processed or cleaned by other services according to the data type and then stored in a database, including but not limited to MySQL, ClickHouse, and MongoDB. The target data frame in the real-time parsing stream is temporarily stored, and its storage time can be set according to the needs of the actual application. This application embodiment does not impose any restrictions.

[0125] In practice, compressed archive streams can use OBS buckets for persistent data storage. Under the premise that the target data frame can be recovered based on the compressed target data frame, and historical data can be traced back and fully analyzed, the storage cost can also be reduced because the compressed target data frame is persistently stored.

[0126] In one feasible implementation, the method further includes: In the real-time parsing stream, based on each set of target data frames temporarily stored, anomaly detection is performed on the target monitoring data preset by the target device. When it is determined that there is an anomaly in the target monitoring data, an anomaly control command is sent to the target device.

[0127] The anomaly detection includes one or more of the following: threshold static detection, statistical analysis detection, and large model inference detection.

[0128] Taking a vehicle as the target device as an example, in the actual implementation process, any target monitoring data that needs to be monitored for any vehicle can be preset. This application embodiment does not impose any restrictions. For example, data such as SoC temperature and CPU usage can be monitored, and each target monitoring data is set with a corresponding threshold and confidence interval. Then, multi-level anomaly detection can be performed on the target detection data of the vehicle.

[0129] Specifically, performing multi-level anomaly detection on preset target monitoring data of target devices such as vehicles in a real-time parsing stream may include the following steps: B1: Perform threshold static detection in the real-time parsing stream.

[0130] Specifically, during the static threshold detection process, it is determined whether the target monitoring data in each set of target data frames temporarily stored is greater than the threshold corresponding to the target monitoring data.

[0131] When the target monitoring data is less than or equal to the threshold corresponding to the target monitoring data, it indicates that there is no abnormality, and monitoring continues.

[0132] When the target monitoring data is greater than the threshold corresponding to the target monitoring data, statistical analysis and detection are performed.

[0133] For example, if the threshold for SoC temperature is set to 105℃, when the SoC temperature is >105℃, it is determined to be a preliminary anomaly, and statistical analysis is performed to further verify whether it is an anomaly.

[0134] In actual implementation, if the target monitoring data seriously exceeds the limit (such as exceeding twice the threshold), the large model inference detection will be directly triggered. This application embodiment does not impose any restrictions.

[0135] B2: Perform statistical analysis and testing.

[0136] Specifically, during the statistical analysis and detection process, anomaly scores for the target monitoring data in the sliding window are calculated.

[0137] The anomaly score is determined based on the standard deviation and / or slope variation of the target monitoring data within the sliding window.

[0138] For example, the sliding window can be the size of n data values, and then the standard deviation and slope change of the target monitoring data within the sliding window are calculated.

[0139] The formula for calculating the standard deviation is as follows:

[0140] in, The standard deviation of the target monitoring data within the sliding window; For the first One data value, This represents the average value of the index within the sliding window. The data values ​​within the sliding window; The formula for calculating the slope change is as follows:

[0141] in, For slope changes; The last data value in the window; The first data value in the window; The data values ​​within the sliding window; Finally, the anomaly score of the sliding window is determined based on the standard deviation and / or slope change of the target monitoring data within the sliding window. The formula for calculating the anomaly score is as follows:

[0142] in: For abnormal ratings; Normalized weighting coefficients ( ); The standard difference constant threshold; The absolute value of the slope is an abnormal threshold; This is for absolute value operations.

[0143] When it is determined based on the sliding window that none of the anomaly scores exceed the preset confidence interval of the target monitoring data, it indicates that there is no anomaly, and monitoring continues.

[0144] When the anomaly score of the target monitoring data exceeds the preset confidence interval of the target monitoring data, large model inference detection is performed.

[0145] B3: Perform large model inference detection.

[0146] Specifically, during the large model inference detection process, the target monitoring data and the preset associated data of the target monitoring data are sent to the pre-trained large model for multi-dimensional time-series correlation analysis to determine whether there are any anomalies in the target monitoring data.

[0147] For example, an LSTM model can be activated. The LSTM model can be pre-trained to adapt to anomaly detection scenarios. Based on the LSTM model, time-series feature extraction and multi-dimensional data correlation analysis are performed on the target monitoring data and the preset associated data of the target monitoring data to determine whether anomalies exist.

[0148] When an anomaly is detected, an anomaly control command can be promptly sent to the target device, such as a vehicle, through a reverse MQTT channel based on the Flink architecture, thus achieving a closed-loop control system.

[0149] The anomaly control commands include one or more of the following: basic snapshot capture command, fine log return command, full raw log capture command, terminal system restart command, and parameter adjustment command.

[0150] The full set of raw logs includes, but is not limited to, sensor data, system logs, and application logs. By acquiring and analyzing the full set of raw logs, a complete reconstruction of the fault scene can be achieved.

[0151] Leveraging the real-time capabilities of the Flink engine, abnormal control commands can be issued in milliseconds and processed in seconds. Upon receiving the abnormal control command, the target device immediately executes the corresponding operation. If the control command involves capturing data, the captured data can be quickly transmitted back to the cloud. Once the cloud receives the data, it immediately parses and analyzes it, enabling instantaneous location of the fault.

[0152] In the anomaly detection provided in this application embodiment, complex anomaly scenarios can be handled through multi-level anomaly detection, anomaly identification through deep learning model is suitable for anomaly detection in high-order scenarios, and after an anomaly occurs, the control closed-loop management of fault recovery is automatically triggered without manual intervention.

[0153] In actual implementation, the recovery status of vehicle anomalies or malfunctions can be monitored in real time. If the vehicle recovers automatically as expected, it is recorded as a successful case; otherwise, a high-level alarm is triggered, and manual intervention is required. For example, the alarm module can be triggered to remind relevant personnel. Alarm reminders can be sent via SMS, telephone, WeChat, email, and Lark, among other methods.

[0154] Based on the multi-level anomaly detection provided in the embodiments of this application, a closed loop from anomaly detection to precise capture can be achieved, significantly improving response timeliness. Automatic triggering and second-level feedback enable instantaneous locking of the fault site and automatic issuance of anomaly control commands, realizing true closed-loop control, improving the system's self-healing capability and stability, and reducing manual intervention.

[0155] The method provided in this application has at least the following beneficial effects: 1. When the target device, such as the vehicle, uploads data, the target string key name in the target data frame is protocol lightweighted and inter-frame prediction is performed. Based on the inter-frame difference information of adjacent target data frames, compression is performed to remove a large amount of duplicate data. Combined with ZSTD compression, the amount of data per unit can be greatly reduced.

[0156] 2. Using the unique identifier of the target device for hash sharding can not only distribute the data of the target device while ensuring the data continuity of a single device, but also solve the single-point bottleneck at the consumer end and avoid data from converging on a few nodes, thus preventing the horizontal scaling capability of the distributed cluster from being compromised.

[0157] 3. Employing multi-level anomaly detection, in addition to detecting obvious static threshold out-of-bounds errors, it also provides early warnings for slowly deteriorating faults. Combined with AI reasoning for judgment, it significantly improves the anomaly recognition rate.

[0158] 4. Upon detecting an anomaly, it automatically generates control commands and supports the issuance of high-density commands, thereby enabling automatic fault recovery of the target equipment and instantaneous locking of the fault site through data capture.

[0159] Reference Figure 5 This diagram illustrates a functional block diagram of a data processing apparatus provided in an embodiment of this application. The apparatus can be applied to a vehicle and includes: The lightweight processing module 101 is used to lightweight the target string key name in the target data frame collected on the target device, including obtaining the lightweight identifier corresponding to the target string key name in the target data frame, and replacing the target string key name in the target data frame with the lightweight identifier corresponding to the target string key name, wherein any target string key name has a corresponding lightweight identifier preset. Compression module 102 is used to generate a compressed target data frame corresponding to the target data frame based on the inter-frame difference information of adjacent target data frames; The upload module 103 is used to upload the compressed target data frame to the cloud service platform.

[0160] Optionally, the compression module includes: The partitioning unit is used to group the target data frames according to the frame sequence number of the target data frames. Each group of target data frames includes a key target data frame and at least one residual target data frame. The first compression unit is used to compress the full data of the key target data frame to generate a compressed target data frame corresponding to the key target data frame. The second compression unit is used to perform differential encoding on any residual target data frame, determine the differential data of the residual target data frame, and compress the differential data to generate a compressed target data frame corresponding to the residual target data frame. The differential data is used to characterize the inter-frame difference information between the residual target data frame and its previous target data frame.

[0161] Optionally, the partitioning unit is used for: Based on the increasing order of the frame sequence number, determine in turn whether the frame sequence number of any target data frame satisfies the condition that its frame sequence number modulo zero with respect to the preset key frame interval parameter. If the modulus is zero, the target data frame is a critical target data frame; If the modulus is not zero, the target data frame is a residual target data frame; Any critical target data frame is grouped together with each residual target data frame that follows it.

[0162] Optionally, when compressing the full data of the key target data frame or the differential data of any residual target data frame, compression is performed based on a preset compression algorithm. The preset compression algorithm includes the ZSTD standard compression algorithm.

[0163] Optionally, the upload module is used for: The fragment topic corresponding to the target device is determined based on the unique identifier of the target device; Based on the Message Queue Telemetry Transport (MQTT) protocol, the compressed target data frame is published to the shard topic, so that the cloud service platform can obtain the compressed target data frame by subscribing to the shard topic.

[0164] Reference Figure 6 This diagram illustrates a functional block diagram of a data processing apparatus provided in an embodiment of this application. The apparatus can be applied to a cloud service platform and includes: Acquisition module 201 is used to acquire compressed target data frames from the target device; The decompression module 202 is used to decompress the compressed target data frame and restore the target data frame of the target device.

[0165] Optionally, the acquisition module is used for: Any Flink task subscribes to an MQTT shard topic and sequentially retrieves each group of compressed target data frames from the target device, storing them in the buffer corresponding to that target device.

[0166] Optionally, the decompression module is used for: In the buffer corresponding to the target device, when the compressed target data frame is identified as a compressed target data frame corresponding to the key target data frame, the compressed target data frame is decompressed to obtain the key target data frame. When the compressed target data frame is identified as the compressed target data frame corresponding to the residual target data frame, the compressed target data frame is decompressed to obtain the differential data of the residual target data frame. Based on the differential data of the residual target data frame and its previous target data frame, the residual target data frame is obtained.

[0167] Optionally, the device further includes a storage module for: Each set of target data frames obtained from the decompression of the target device is sent to a Kafka message queue for temporary storage; Each set of compressed target data frames from the target device is sent to a Kafka message queue for persistent storage.

[0168] Optionally, the device further includes an anomaly detection module, used for: Based on each set of target data frames temporarily stored, anomaly detection is performed on the target monitoring data preset by the target device. When it is determined that there is an anomaly in the target monitoring data, an anomaly control command is sent to the target device. The anomaly detection includes one or more of the following: threshold static detection, statistical analysis detection, and large model inference detection. The anomaly control commands include one or more of the following: basic snapshot capture command, fine log return command, full raw log capture command, vehicle terminal system restart command, and parameter adjustment command.

[0169] Optionally, the anomaly detection module is used for: Determine whether the target monitoring data in each set of target data frames temporarily stored is greater than the threshold corresponding to the target monitoring data; When the target monitoring data is greater than the threshold corresponding to the target monitoring data, an anomaly score of the target monitoring data in the sliding window is calculated, wherein the anomaly score is determined based on the standard deviation and / or slope change of the target monitoring data in the sliding window; When the anomaly score of the target monitoring data exceeds the preset confidence interval of the target monitoring data, a multidimensional time series correlation analysis is performed on the target monitoring data and the preset associated data of the target monitoring data to determine whether there is anomaly in the target monitoring data.

[0170] Reference Figure 7 The diagram illustrates an electronic device provided in an embodiment of this application. The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the data processing method embodiments described above and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0171] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0172] Reference Figure 8 The diagram illustrates a readable storage medium provided in an embodiment of this application. The readable storage medium stores a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the data processing method embodiments described above and achieve the same technical effects. To avoid repetition, further details are omitted here.

[0173] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0174] Reference Figure 9 The diagram illustrates a computer program product provided in an embodiment of this application, including a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the various processes of the data processing method embodiment described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0175] This application also provides a vehicle for executing the various processes of the data processing method embodiments described above, or includes the electronic equipment described in this application, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0176] This application also provides a cloud service platform, which is used in various processes of the data processing method embodiments described above and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] This application also provides a data processing system, which includes the vehicle and the cloud service platform described in this application.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0180] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. The description of the embodiments above is only for the purpose of helping to understand the method and core idea of ​​this application. Those skilled in the art can make many forms under the guidance of this application without departing from the spirit and scope of protection of the claims, and all of these are within the protection scope of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: Lightweighting the target string key name in the target data frame collected on the target device includes obtaining the lightweight identifier corresponding to the target string key name in the target data frame, and replacing the target string key name in the target data frame with the lightweight identifier corresponding to the target string key name, wherein any target string key name has a corresponding lightweight identifier preset. Based on the inter-frame difference information of adjacent target data frames, a compressed target data frame corresponding to the target data frame is generated; Upload the compressed target data frame to the cloud service platform.

2. The method according to claim 1, characterized in that, The step of generating a compressed target data frame corresponding to the target data frame based on the inter-frame difference information of adjacent target data frames includes: Based on the frame sequence number of the target data frame, the target data frames are grouped together, and each group of target data frames includes one key target data frame and at least one residual target data frame. The full data of the key target data frame is compressed to generate a compressed target data frame corresponding to the key target data frame; Differential coding is performed on any residual target data frame to determine the differential data of the residual target data frame, and the differential data is compressed to generate the compressed target data frame corresponding to the residual target data frame. The differential data is used to characterize the inter-frame difference information between the residual target data frame and its previous target data frame.

3. The method according to claim 2, characterized in that, The step of grouping the target data frames according to their frame sequence numbers includes: Based on the increasing order of the frame sequence number, determine in turn whether the frame sequence number of any target data frame satisfies the condition that its frame sequence number modulo zero with respect to the preset key frame interval parameter. If the modulus is zero, the target data frame is a critical target data frame; If the modulus is not zero, the target data frame is a residual target data frame; Any critical target data frame is grouped together with each residual target data frame that follows it.

4. The method according to claim 2, characterized in that, When compressing the full data of the key target data frame, or compressing the differential data of any residual target data frame, compression is performed based on a preset compression algorithm. The preset compression algorithm includes the ZSTD standard compression algorithm.

5. The method according to claim 1, characterized in that, Uploading the compressed target data frame to the cloud service platform includes: The fragment topic corresponding to the target device is determined based on the unique identifier of the target device; Based on the Message Queue Telemetry Transport (MQTT) protocol, the compressed target data frame is published to the shard topic, so that the cloud service platform can obtain the compressed target data frame by subscribing to the shard topic.

6. A data processing method, characterized in that, Applied to a cloud service platform, the method includes: Acquire compressed target data frames from the target device; The compressed target data frame is decompressed to restore the target data frame of the target device.

7. The method according to claim 6, characterized in that, The acquisition of compressed target data frames from the target device includes: Any Flink task subscribes to an MQTT shard topic and sequentially retrieves each group of compressed target data frames from the target device, storing them in the buffer corresponding to that target device.

8. The method according to claim 7, characterized in that, The step of decompressing the compressed target data frame to recover the target data frame of the target device includes: In the buffer corresponding to the target device, when the compressed target data frame is identified as a compressed target data frame corresponding to the key target data frame, the compressed target data frame is decompressed to obtain the key target data frame. When the compressed target data frame is identified as the compressed target data frame corresponding to the residual target data frame, the compressed target data frame is decompressed to obtain the differential data of the residual target data frame. Based on the differential data of the residual target data frame and its previous target data frame, the residual target data frame is obtained.

9. The method according to claim 6, characterized in that, The method further includes: Each set of target data frames obtained from the decompression of the target device is sent to a Kafka message queue for temporary storage; Each set of compressed target data frames from the target device is sent to a Kafka message queue for persistent storage.

10. The method according to claim 9, characterized in that, The method further includes: Based on each set of target data frames temporarily stored, anomaly detection is performed on the target monitoring data preset by the target device. When it is determined that there is an anomaly in the target monitoring data, an anomaly control command is sent to the target device. The anomaly detection includes one or more of the following: threshold static detection, statistical analysis detection, and large model inference detection. The anomaly control commands include one or more of the following: basic snapshot capture command, fine log return command, full raw log capture command, terminal system restart command, and parameter adjustment command.

11. The method according to claim 10, characterized in that, The step of performing anomaly detection on the target monitoring data preset by the target device based on each set of target data frames temporarily stored includes: Determine whether the target monitoring data in each set of target data frames temporarily stored is greater than the threshold corresponding to the target monitoring data; When the target monitoring data is greater than the threshold corresponding to the target monitoring data, an anomaly score of the target monitoring data in the sliding window is calculated, wherein the anomaly score is determined based on the standard deviation and / or slope change of the target monitoring data in the sliding window; When the anomaly score of the target monitoring data exceeds the preset confidence interval of the target monitoring data, a multidimensional time series correlation analysis is performed on the target monitoring data and the preset associated data of the target monitoring data to determine whether there is anomaly in the target monitoring data.

12. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the data processing method as described in any one of claims 1-5.

13. A vehicle, characterized in that, The vehicle is used to perform the data processing method according to any one of claims 1-5, or includes the electronic device according to claim 12.

14. A cloud service platform, characterized in that, The cloud service platform is used to execute the data processing method according to any one of claims 6-11.

15. A data processing system, characterized in that, The data processing system includes the vehicle of claim 13 and the cloud service platform of claim 14.

Images