Vehicle data processing platform

Abstract

The present invention relates to the field of data processing technology, and specifically provides a vehicle data processing platform, which aims to solve the technical problem of how to store and analyze massive levels of vehicle data. To this end, the first distributed message queue system in the vehicle data processing platform provided by the present invention includes multiple message queues, and the distributed data processing system includes multiple data processing nodes. The message queue can receive vehicle data sent by the first external system, and the data processing node can obtain the vehicle data received by the message queue pre-bound thereto, perform data analysis on the vehicle data and send the data analysis results to the distributed storage system. Based on the above structure, the present invention can expand the number of message queues and data processing nodes according to actual needs, and expand the storage space of the distributed storage system, which not only realizes the storage and processing of massive levels of vehicle data, but also avoids the existence of performance hotspot systems in the vehicle data processing platform.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a vehicle data processing platform. Background Art

[0002] As the core component of electric vehicles, the power battery needs to be monitored in a timely manner to ensure the safe operation of the electric vehicle. To this end, it is necessary to regularly send the power battery status data and the status data of power battery-related equipment in the vehicle (such as the battery management system, electric drive system and on-board equipment powered by the power battery, etc.) to the remote server, so as to accurately analyze the working status of the power battery based on this data.

[0003] However, due to the large number of power batteries and the diverse sources of status data for each battery (such as vehicles, charging equipment, and battery storage stations), and the fact that these data sources often send data to remote servers in very short intervals (e.g., a few seconds), massive amounts of vehicle data need to be stored and analyzed. Currently, however, this data is typically stored in relational databases based on single-machine storage, which cannot rapidly read and write massive amounts of data, and thus cannot achieve real-time data analysis and processing. Summary of the Invention

[0004] In order to solve the above-mentioned problems in the prior art, that is, to solve the technical problem of how to store and analyze massive amounts of vehicle data. To this end, the present invention provides a vehicle data processing platform, which includes a first distributed message queue system, a distributed data processing system, and a distributed storage system. The first distributed message queue system includes multiple message queues, and the distributed data processing system includes multiple data processing nodes. The message queue is configured to receive vehicle data sent by a first external system. The data processing node is configured to obtain vehicle data received by a message queue pre-bound thereto, perform data analysis on the vehicle data, and send the data analysis results to the distributed storage system for storage. The vehicle data processing platform can store and analyze massive amounts of vehicle data.

[0005] Furthermore, an optional technical solution provided by the present invention is:

[0006] The data processing node includes a data parsing module, a data caching module and a data aggregation analysis module, and the data aggregation analysis module includes a data statistics submodule and a data alarm submodule;

[0007] The data parsing module is configured to parse the vehicle data and send the parsing result to the data cache module;

[0008] The data statistics submodule is configured to obtain cached data in the data cache module, perform data statistics on the cached data, and send the data statistics results to the distributed storage system for storage;

[0009] The data alarm submodule is configured to obtain cached data in the data cache module, perform alarm analysis on the cached data, and send the alarm analysis results to the distributed storage system for storage.

[0010] Furthermore, an optional technical solution provided by the present invention is: the data cache module includes multiple data pools and a cache queue and a read-write thread corresponding to each data pool, and the data pool is configured to cache the parsing results of the vehicle data through a hash table.

[0011] Furthermore, an optional technical solution provided by the present invention is: when the vehicle data is power battery-related data, the data cache module is configured to allocate a data pool for the parsing results of the vehicle data in the following manner:

[0012] Data_pool ID =Hash(Battery ID )%M

[0013] Among them, the Data_pool ID is the number of the data pool, the Battery ID is the number of the power battery, M is the total number of data pools in the data cache module, Hash represents a hash function, and % represents a remainder operation.

[0014] Furthermore, an optional technical solution provided by the present invention is: the distributed storage system includes a distributed key-value storage subsystem, and the data processing node further includes a first data asynchronous sending module;

[0015] The first data asynchronous sending module is configured to receive the analysis results output by the data analysis module, receive the data statistics results and alarm analysis results output by the data fusion analysis module, and send the analysis results, data statistics results and alarm analysis results to the distributed key-value storage subsystem for storage through asynchronous transmission;

[0016] The distributed key-value storage subsystem is a storage subsystem built based on a Key-Value key-value pair storage structure.

[0017] Furthermore, an optional technical solution provided by the present invention is: the distributed storage system further includes a distributed object storage subsystem, and the vehicle data processing platform further includes a second distributed message queue system and a periodic persistence service system;

[0018] The data processing node further includes a second data asynchronous sending module, which is configured to receive the analysis results output by the data analysis module, receive the data statistics results and the alarm analysis results output by the data fusion analysis module, and send the analysis results, data statistics results and alarm analysis results to the second distributed message queue system via asynchronous transmission;

[0019] The periodic persistence service system is configured to periodically send the parsing results, data statistics results, and alarm analysis results received by the second distributed message queue system to the distributed object storage subsystem for storage;

[0020] The distributed object storage subsystem is a storage subsystem built based on an object storage structure.

[0021] Furthermore, an optional technical solution provided by the present invention is that: the vehicle data processing platform further includes a data warehouse, and the distributed object storage subsystem is further configured to add corresponding metadata to the data stored therein, and send the metadata-added data to the data warehouse for storage;

[0022] The metadata includes the name of the data table corresponding to the data in the distributed object storage subsystem, the name of the database to which the data table belongs, the partition information of the data table, and the storage path of the data.

[0023] Furthermore, an optional technical solution provided by the present invention is: the second distributed message queue system is further configured to receive a data acquisition instruction sent by a second external system, and send data corresponding to the data acquisition instruction to the second external system;

[0024] The data corresponding to the data acquisition instruction includes parsing results, and / or data statistics results, and / or alarm analysis results.

[0025] Furthermore, an optional technical solution provided by the present invention is: the vehicle data processing platform further includes a metadata management system, wherein the metadata management system is configured to store metadata corresponding to the vehicle data;

[0026] Among them, when the vehicle data is power battery-related data, the metadata corresponding to the power battery-related data includes the number of the power battery, the number of the vehicle / charging facility where the power battery is located, the location of the vehicle / charging facility where the power battery is located, and the data name, data format and data source of the power battery-related data.

[0027] Furthermore, an optional technical solution provided by the present invention is that: the data processing node also includes a metadata reading and writing module,

[0028] The metadata reading and writing module is configured to obtain the metadata of the vehicle data stored in the metadata management system and send the metadata to the data parsing module, and obtain new metadata generated by the data parsing module after parsing the vehicle data according to the metadata and send the new metadata to the metadata management system for storage.

[0029] Furthermore, an optional technical solution provided by the present invention is that the metadata reading and writing module includes a data caching submodule and a data reading submodule, and the data reading submodule is configured to obtain the metadata stored in the metadata management system in the following manner:

[0030] Determine whether the effective duration of the metadata is less than or equal to a preset first duration threshold and whether the predicted number of reads of the metadata is less than or equal to a preset first number threshold,

[0031] If the effective duration is less than or equal to the first duration threshold and the predicted number of reads is less than or equal to the first number threshold, the metadata is acquired in real time; otherwise, the metadata is acquired periodically and cached in the data cache submodule.

[0032] Furthermore, an optional technical solution provided by the present invention is that the metadata reading and writing module further includes a data writing submodule, and the data writing submodule is configured to send the new metadata to the metadata management system for storage in the following manner:

[0033] Determine whether the effective duration of the new metadata is less than or equal to a preset second duration threshold and whether the predicted number of write times of the new metadata is less than or equal to a preset second number threshold,

[0034] If the effective duration is less than or equal to the second duration threshold and the predicted number of writes is less than or equal to the second number threshold, the new metadata is sent to the metadata management system in real time for storage; otherwise, the new metadata is sent to the data cache sub-module and then the new metadata in the data cache sub-module is periodically sent to the metadata management system for storage.

[0035] Furthermore, an optional technical solution provided by the present invention is that the vehicle data processing platform also includes a data retrieval system,

[0036] The data retrieval system is configured to acquire and display the stored data corresponding to the first retrieval instruction in the distributed key-value storage subsystem according to the received first retrieval instruction, and acquire and display the metadata corresponding to the second retrieval instruction in the metadata management system according to the received second retrieval instruction;

[0037] When the stored data and the metadata are power battery related data, the first retrieval instruction includes the power battery number and / or data storage time, and the second retrieval instruction includes the power battery number and / or data storage time.

[0038] Furthermore, an optional technical solution provided by the present invention is that the first distributed message queue system is further configured to perform the following operations:

[0039] When a data processing node failure is detected, the binding relationship between the data processing node and its corresponding message queue is released, and a binding relationship is established between the message queue and a data processing node that has not failed, so that the data processing node that has not failed can obtain the vehicle data received by the message queue.

[0040] Furthermore, an optional technical solution provided by the present invention is: the data processing node further includes a monitoring module, and the monitoring module is configured to monitor the working status of the data processing node and send the monitoring result to an external monitoring system.

[0041] Furthermore, an optional technical solution provided by the present invention is: the data processing node also includes a parameter configuration module, and the parameter configuration module is configured to periodically obtain parameter values ​​of the parameters to be configured in the data processing node, and configure the parameters to be configured according to the parameter values.

[0042] Solution 1: A vehicle data processing platform, characterized in that the vehicle data processing platform includes a first distributed message queue system, a distributed data processing system, and a distributed storage system, wherein the first distributed message queue system includes multiple message queues, and the distributed data processing system includes multiple data processing nodes;

[0043] The message queue is configured to receive vehicle data sent by the first external system;

[0044] The data processing node is configured to obtain vehicle data received by a message queue pre-bound thereto, perform data analysis on the vehicle data, and send the data analysis results to the distributed storage system for storage.

[0045] Solution 2: The vehicle data processing platform according to Solution 1, characterized in that the data processing node includes a data parsing module, a data caching module, and a data aggregation analysis module, and the data aggregation analysis module includes a data statistics submodule and a data alarm submodule;

[0046] The data parsing module is configured to parse the vehicle data and send the parsing result to the data cache module;

[0047] The data statistics submodule is configured to obtain cached data in the data cache module, perform data statistics on the cached data, and send the data statistics results to the distributed storage system for storage;

[0048] The data alarm submodule is configured to obtain cached data in the data cache module, perform alarm analysis on the cached data, and send the alarm analysis results to the distributed storage system for storage.

[0049] Solution 3: The vehicle data processing platform according to Solution 2 is characterized in that the data cache module includes multiple data pools and a cache queue and read-write threads corresponding to each data pool, and the data pool is configured to cache the parsing results of the vehicle data through a hash table.

[0050] Solution 4: The vehicle data processing platform according to Solution 3, characterized in that when the vehicle data is power battery-related data, the data cache module is configured to allocate a data pool for the parsing results of the vehicle data in the following manner:

[0051] Data_pool ID =Hash(Battery ID )%M

[0052] Among them, the Data_pool ID is the number of the data pool, the Battery ID is the number of the power battery, M is the total number of data pools in the data cache module, Hash represents a hash function, and % represents a remainder operation.

[0053] Solution 5: The vehicle data processing platform according to Solution 2, characterized in that the distributed storage system includes a distributed key-value storage subsystem, and the data processing node further includes a first data asynchronous sending module;

[0054] The first data asynchronous sending module is configured to receive the analysis results output by the data analysis module, receive the data statistics results and alarm analysis results output by the data fusion analysis module, and send the analysis results, data statistics results and alarm analysis results to the distributed key-value storage subsystem for storage through asynchronous transmission;

[0055] The distributed key-value storage subsystem is a storage subsystem built based on a Key-Value key-value pair storage structure.

[0056] Solution 6: The vehicle data processing platform according to Solution 2, characterized in that the distributed storage system further includes a distributed object storage subsystem, and the vehicle data processing platform further includes a second distributed message queue system and a periodic persistence service system;

[0057] The data processing node further includes a second data asynchronous sending module, which is configured to receive the analysis results output by the data analysis module, receive the data statistics results and the alarm analysis results output by the data fusion analysis module, and send the analysis results, data statistics results and alarm analysis results to the second distributed message queue system via asynchronous transmission;

[0058] The periodic persistence service system is configured to periodically send the parsing results, data statistics results, and alarm analysis results received by the second distributed message queue system to the distributed object storage subsystem for storage;

[0059] The distributed object storage subsystem is a storage subsystem built based on an object storage structure.

[0060] Solution 7: The vehicle data processing platform according to Solution 6, characterized in that the vehicle data processing platform further includes a data warehouse, and the distributed object storage subsystem is further configured to add corresponding metadata to the data stored therein and send the metadata-added data to the data warehouse for storage;

[0061] The metadata includes the name of the data table corresponding to the data in the distributed object storage subsystem, the name of the database to which the data table belongs, the partition information of the data table, and the storage path of the data.

[0062] Solution 8: The vehicle data processing platform according to Solution 6, characterized in that the second distributed message queue system is further configured to receive a data acquisition instruction sent by a second external system, and send data corresponding to the data acquisition instruction to the second external system;

[0063] The data corresponding to the data acquisition instruction includes parsing results, and / or data statistics results, and / or alarm analysis results.

[0064] Solution 9: The vehicle data processing platform according to Solution 5, characterized in that the vehicle data processing platform further includes a metadata management system, wherein the metadata management system is configured to store metadata corresponding to the vehicle data;

[0065] Among them, when the vehicle data is power battery-related data, the metadata corresponding to the power battery-related data includes the number of the power battery, the number of the vehicle / charging facility where the power battery is located, the location of the vehicle / charging facility where the power battery is located, and the data name, data format and data source of the power battery-related data.

[0066] Solution 10: The vehicle data processing platform according to Solution 9, characterized in that the data processing node further includes a metadata reading and writing module.

[0067] The metadata reading and writing module is configured to obtain the metadata of the vehicle data stored in the metadata management system and send the metadata to the data parsing module, and obtain new metadata generated by the data parsing module after parsing the vehicle data according to the metadata and send the new metadata to the metadata management system for storage.

[0068] Solution 11: The vehicle data processing platform according to Solution 10, characterized in that the metadata reading and writing module includes a data caching submodule and a data reading submodule, and the data reading submodule is configured to obtain the metadata stored in the metadata management system by:

[0069] Determine whether the effective duration of the metadata is less than or equal to a preset first duration threshold and whether the predicted number of reads of the metadata is less than or equal to a preset first number threshold,

[0070] If the effective duration is less than or equal to the first duration threshold and the predicted number of reads is less than or equal to the first number threshold, the metadata is acquired in real time; otherwise, the metadata is acquired periodically and cached in the data cache submodule.

[0071] Solution 12: The vehicle data processing platform according to Solution 10, characterized in that the metadata reading and writing module further includes a data writing submodule, and the data writing submodule is configured to send the new metadata to the metadata management system for storage in the following manner:

[0072] Determine whether the effective duration of the new metadata is less than or equal to a preset second duration threshold and whether the predicted number of write times of the new metadata is less than or equal to a preset second number threshold,

[0073] If the effective duration is less than or equal to the second duration threshold and the predicted number of writes is less than or equal to the second number threshold, the new metadata is sent to the metadata management system in real time for storage; otherwise, the new metadata is sent to the data cache sub-module and then the new metadata in the data cache sub-module is periodically sent to the metadata management system for storage.

[0074] Solution 13: The vehicle data processing platform according to Solution 9 is characterized in that the vehicle data processing platform also includes a data retrieval system.

[0075] The data retrieval system is configured to acquire and display the stored data corresponding to the first retrieval instruction in the distributed key-value storage subsystem according to the received first retrieval instruction, and acquire and display the metadata corresponding to the second retrieval instruction in the metadata management system according to the received second retrieval instruction;

[0076] When the stored data and the metadata are power battery related data, the first retrieval instruction includes the power battery number and / or data storage time, and the second retrieval instruction includes the power battery number and / or data storage time.

[0077] Solution 14: The vehicle data processing platform according to any one of Solutions 1 to 13, wherein the first distributed message queue system is further configured to perform the following operations:

[0078] When a data processing node failure is detected, the binding relationship between the data processing node and its corresponding message queue is released, and a binding relationship is established between the message queue and a data processing node that has not failed, so that the data processing node that has not failed can obtain the vehicle data received by the message queue.

[0079] Solution 15: A vehicle data processing platform according to any one of Solutions 1 to 13, characterized in that the data processing node also includes a monitoring module, which is configured to monitor the working status of the data processing node and send the monitoring results to an external monitoring system.

[0080] Option 16, a vehicle data processing platform according to any one of Options 1 to 13, is characterized in that the data processing node also includes a parameter configuration module, and the parameter configuration module is configured to periodically obtain parameter values ​​of the parameters to be configured in the data processing node, and configure the parameters to be configured according to the parameter values.

[0081] Compared with the closest existing technology, the above technical solution has at least the following beneficial effects:

[0082] 1. The vehicle data processing platform provided by the present invention mainly includes a first distributed message queue system, a distributed data processing system and a distributed storage system. The first distributed message queue system includes multiple message queues, and the distributed data processing system includes multiple data processing nodes. The message queue can be configured to receive vehicle data sent by the first external system, and the data processing node can be configured to obtain vehicle data received by the message queue pre-bound thereto, perform data analysis on the vehicle data and send the data analysis results to the distributed storage system for storage. Based on the above structure, the present invention can expand the number of message queues in the first distributed message queue system according to actual needs, and can also expand the number of data processing nodes in the distributed data processing system according to actual needs, and can also expand the storage space of the distributed storage system according to actual needs. It not only realizes the storage and processing of massive levels of vehicle data, but also avoids the existence of performance hotspot systems in the vehicle data processing platform (i.e., systems that cannot improve processing capabilities through equipment expansion).

[0083] 2. The data processing nodes in the vehicle data processing platform provided by the present invention can obtain vehicle data generated by each data source through a first distributed message queue system and perform data statistics and alarm analysis on vehicle data within a certain period of time, facilitating rapid access to all relevant data for a particular vehicle by vehicle maintenance personnel. When the vehicle data is related to power batteries, the vehicle data processing platform provided by the present invention can rapidly obtain all relevant data for each power battery throughout its entire lifecycle. For example, relevant data on power batteries during driving, stopped, charging, battery replacement, and recycling.

[0084] 3. The data warehouse in the vehicle data processing platform provided by the present invention stores metadata of vehicle data. The metadata includes the name of the data table corresponding to the vehicle data in the distributed object storage subsystem, the name of the database to which the data table belongs, the partition information of the data table, and the storage path of the data. This facilitates relevant vehicle maintenance personnel to obtain vehicle data and corresponding metadata from the data warehouse, and then quickly obtain vehicle data within a specific time range, and / or specific battery, and / or specific data format based on the metadata of the vehicle data.

[0085] 4. The data retrieval system in the vehicle data processing platform provided by the present invention can quickly obtain the corresponding stored data and metadata from the distributed key-value storage subsystem and metadata management system according to the retrieval instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 is a schematic diagram of the main structure of a vehicle data processing platform according to one embodiment of the present invention;

[0087] Figure 2 It is a schematic diagram of the main structure of a vehicle data processing platform according to another embodiment of the present invention. DETAILED DESCRIPTION

[0088] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0089] The vehicle data processing platform provided by the present invention is described below with reference to the accompanying drawings.

[0090] See attached Figure 1 , Figure 1 The main structure of the vehicle data processing platform in this embodiment is shown as an example. Figure 1 As shown, in this embodiment, the vehicle data processing platform 1 primarily includes a first distributed message queue system 11, a distributed data processing system 12, and a distributed storage system 13. The first distributed message queue system 11 may include multiple message queues, and the distributed data processing system 12 may include multiple data processing nodes. Specifically, the message queues may be configured to receive vehicle data sent by the first external system 2, and the data processing nodes may be configured to retrieve vehicle data received by the message queues pre-assigned to them, perform data analysis on the vehicle data, and send the analysis results to the distributed storage system 13 for storage.

[0091] In this embodiment, the first external system 2 is a system that can obtain vehicle data and communicate with the first distributed message queue system 11, and send the obtained vehicle data to the first distributed message queue system 11. Optionally, when the vehicle data is power battery-related data, the first external system 2 can be a preset on-board system on the vehicle, or a preset intelligent communication system on the charging and swapping facilities, or a preset intelligent communication system in the energy storage power station. The distributed storage system 13 can be a storage system built based on distributed storage technology, which can store hundreds of millions of data. The first distributed message queue system 11 can be a distributed message queue system built based on the Apache Kafka message system.

[0092] In this embodiment, each data processing node can be bound to one or more message queues, and each message queue is bound to a data processing node. This ensures that all vehicle data received by the message queues is sent to the same data processing node for data analysis. For example, when the vehicle data is related to power batteries, each message queue receives data related to a power battery. By binding each message queue to a data processing node, it is ensured that each data processing node can perform data analysis on all relevant data for a power battery. This avoids data transmission loss and computing resource waste caused by distributing data related to the same power battery to different data processing nodes for data analysis, enabling the distributed data processing system 12 to quickly analyze data related to the same battery.

[0093] In an optional implementation of this embodiment, the first distributed message queue system 11 may also be configured to switch a failed data processing node in the following manner:

[0094] When a data processing node is detected to have failed, the binding relationship between the data processing node and its corresponding message queue is released, and a binding relationship is established between the message queue and a data processing node that has not failed, so that the data processing node that has not failed can obtain the vehicle data received by this message queue, thereby improving the stability and reliability of the vehicle data processing platform.

[0095] Furthermore, in another optional implementation of this embodiment, Figure 1 Each data processing node in the distributed data processing system 12 shown in the figure can include a data parsing module, a data caching module, a data aggregation analysis module, a first data asynchronous transmission module, a second data asynchronous transmission module, a metadata reading and writing module, and a message queue receiving module, etc. The distributed storage system 13 can include a distributed key-value storage subsystem and a distributed object storage subsystem; at the same time, in this embodiment Figure 1 The vehicle data processing platform shown may also include a second distributed message queue system and a periodic persistence service system, etc. The vehicle data processing platform in this optional embodiment will be described in detail below with reference to the accompanying drawings.

[0096] See attached Figure 2 , Figure 2 The main structure of the vehicle data processing platform in this embodiment is shown as an example. Figure 2As shown, the vehicle data processing platform in this embodiment mainly includes a first distributed message queue system (not shown), a distributed data processing system, a distributed storage system, a second distributed message queue system 14, a periodic persistence service system 15, a data warehouse 16, a metadata management system 17, a data retrieval system 18, a first external system (not shown) and a second external system 3. The distributed data processing system includes multiple data processing nodes, each of which can include a data parsing module 121, a data caching module 122, a data aggregation analysis module 123, a first data asynchronous sending module 124, a second data asynchronous sending module 125, a metadata reading and writing module 126 and a message queue receiving module 127. The data aggregation analysis module 123 can include a data statistics submodule 1231 and a data alarm submodule 1232. The distributed storage system can include a distributed key-value storage subsystem 131 and a distributed object storage subsystem 132.

[0097] In this embodiment, the message queue receiving module 127 communicates with a message queue pre-bound to a data processing node in a first distributed message queue system (not shown) to obtain vehicle data received by the message queue.

[0098] In this embodiment, the data parsing module 121 can be configured to parse vehicle data and send the parsing results to the data cache module 122. At the same time, in this embodiment, when the vehicle data is in a compressed format, the data parsing module 121 can also decompress the vehicle data and perform data parsing on the decompressed data. Optionally, in this embodiment, the data parsing module 121 can include a variety of parsers that can parse data in different data formats, such as a JSON data format parser, a Protobuf data format parser, and a CAN+DBC data format parser. The JSON data format parser is used to parse string data in the JSON data format, the Protobuf data format parser is used to parse binary numbers in the Protobuf data format, and the CAN+DBC data format parser stores a DBC file (Database CAN). Based on the DBC file and according to the ID and binary value of the CAN signal (Controller Area Network), the CAN signal is parsed to obtain the name and value of each signal contained in the CAN signal.

[0099] In this embodiment, the data statistics submodule 1231 can be configured to obtain the cached data in the data cache module 122, perform data statistics on the cached data and send the data statistics results to the distributed key-value storage subsystem 131 for storage. For example, when the vehicle data is power battery-related data, the data statistics submodule 1231 can perform statistics on the total number of power batteries within a period of time and the average / minimum / maximum value of each battery data. The data alarm submodule 1232 can be configured to obtain the cached data in the data cache module 122, perform alarm analysis on the cached data and send the alarm analysis results to the distributed key-value storage subsystem 131 for storage. For example, when the vehicle data is power battery-related data, the data alarm submodule 1232 can perform temperature alarms based on the battery temperature signal within a period of time. Optionally, in this embodiment, the distributed key-value storage subsystem 131 can also set the data storage period (such as 6 months) according to actual needs, which is conducive to the analysis and processing of historical vehicle data.

[0100] In this embodiment, the data cache module 122 has read-only read and write properties for downstream modules such as the data statistics submodule 1231 and the data alarm submodule 1232, that is, the data statistics submodule 1231 and the data alarm submodule 1232 can only obtain the data stored in the data cache module 122, but cannot modify the data stored in the data cache module 122. In this way, the data statistics submodule 1231 and the data alarm submodule 1232 can flexibly adjust the statistical algorithm used for data statistics and the alarm algorithm used for alarm processing according to actual needs without affecting the data stored in the data cache module 122. At the same time, in this embodiment, the data cache module 122 can be configured to store battery data within a certain period of time (such as 5 minutes). The data cache module 122 can include multiple data pools and a cache queue and a read-write thread corresponding to each data pool. The data pool can be configured to cache the parsing results of the vehicle data through a hash table. By configuring a separate cache queue and a separate read-write thread for each data pool, interaction between data pools can be avoided, which affects their respective stored data. Optionally, when the vehicle data is power battery-related data, the hash table is a doubly linked list consisting of a hash key and its key value. The hash key of the hash table is the power battery serial number, and the key value of the hash key is the battery data of the power battery. A hash table can store a collection of all battery data of a power battery at a sampling time. The elements in the hash table are sorted according to the arrival time of the battery data, that is, the tail element of the hash table is the newest and the head element is the oldest. At the same time, the maximum number of elements stored in the hash table can be flexibly adjusted according to actual needs. Whenever a battery data is stored at the tail of the table, a battery data is deleted from the head. When the vehicle data is power battery-related data, the data cache module 122 can be configured to allocate a data pool for the parsed results of the vehicle data in the following manner:

[0101] Data_pool ID =Hash(Battery ID )%M (1)

[0102] The meaning of each parameter in formula (1) is:

[0103] Data_pool ID Is the number of the data pool, Battery ID is the number of the power battery, M is the total number of data pools in the data cache module 122, Hash represents a hash function, and % represents a remainder operation.

[0104] In this embodiment, the first data asynchronous sending module 124 can be configured to receive the analysis results output by the data analysis module 121, the data statistics results and the alarm analysis results output by the data fusion analysis module 123, and send the analysis results, data statistics results and alarm analysis results to the distributed key-value storage subsystem 131 for storage in a regular manner (such as regularly sending at a time interval of 100ms) through asynchronous transmission. At the same time, the first data asynchronous sending module 124 can flexibly adjust the time interval of regular sending and the number of data sending threads according to actual needs. When multiple data sending threads are configured, the data sending speed of the first data asynchronous sending module 124 can be increased. Optionally, in this embodiment, the distributed key-value storage subsystem 131 can be a storage subsystem built based on a Key-Value key-value pair storage structure. For example, the distributed key-value storage subsystem 131 is a storage subsystem built based on an Apache Cassandra storage structure.

[0105] In this embodiment, the second data asynchronous sending module 125 can be configured to receive the analysis results output by the data analysis module 121, the data statistics results and the alarm analysis results output by the data fusion analysis module 123, and send the analysis results, data statistics results and alarm analysis results to the second distributed message queue system 14 in a regular manner (such as sending them regularly at a time interval of 100ms) through asynchronous transmission. At the same time, the second data asynchronous sending module 125 can flexibly adjust the time interval of regular sending and the number of data sending threads according to actual needs. When multiple data sending threads are configured, the data sending speed of the second data asynchronous sending module 125 can be increased. The periodic persistence service system 15 in this embodiment can be configured to send the analysis results, data statistics results and alarm analysis results received by the second distributed message queue system 14 in a regular manner (such as sending them regularly at a time interval of 5 minutes) to the distributed object storage subsystem 132 for storage. Optionally, the periodic persistence service system 15 in this embodiment can be a system built based on the Apache Kafka message system. In this embodiment, the second distributed message queue system 14 can also be configured to receive a data acquisition instruction sent by the second external system 3 and send data corresponding to the data acquisition instruction to the second external system 3. The data corresponding to the data acquisition instruction may include parsing results, data statistics, and / or alarm analysis results. For example, when the second external system 3 is an early warning system, the data acquisition instruction may be a battery alarm data acquisition instruction. The second distributed message queue system 14 can send the alarm analysis results to the early warning system based on the battery alarm data acquisition instruction.

[0106] In this embodiment, the distributed object storage subsystem 132 may also be configured to add metadata to the data it stores and send the metadata-added data to the data warehouse 16 for storage. The added metadata may include the name of the data table corresponding to the data in the distributed object storage subsystem 132, the name of the database to which the data table belongs, partition information of the data table (e.g., partition information based on time information: hour, day, month, year, etc.), and the data storage path. Optionally, in this embodiment, the distributed object storage subsystem 132 may be a storage subsystem built based on an object storage structure. For example, the distributed object storage subsystem 132 may be a storage subsystem built based on Amazon S3 storage technology.

[0107] In this embodiment, the metadata reading and writing module 126 can be configured to obtain metadata for vehicle data stored in the metadata management system 17 and send the obtained metadata to the data parsing module 121. It can also be configured to obtain new metadata generated by the data parsing module 121 after parsing the vehicle data based on the metadata and send the generated new metadata to the metadata management system 17 for storage. When the vehicle data is power battery-related data, the metadata stored in the metadata management system 17 can include the power battery number, the number of the vehicle / charging facility where the power battery is located, the location of the power battery in the vehicle / charging facility, and the data name, data format, and data source of the power battery-related data. The data source refers to whether the data originates from the vehicle or the charging facility, and which system / device in the vehicle it originates from, such as the vehicle's battery system and air conditioning system. Furthermore, in this embodiment, the metadata reading and writing module 126 includes a data caching submodule, a data reading submodule, and a data writing submodule.

[0108] Specifically, the data reading submodule can be configured to obtain the metadata stored in the metadata management system 17 in the following manner:

[0109] Determine whether the effective duration of the metadata is less than or equal to a preset first duration threshold and whether the predicted number of metadata reads is less than or equal to a preset first number threshold: if the effective duration is less than or equal to the first duration threshold and the predicted number of metadata reads is less than or equal to the first number threshold, obtain the metadata in real time; otherwise, obtain the metadata periodically (e.g., at a time interval of 1 minute) and cache the metadata in the data cache submodule.

[0110] The data writing submodule is configured to send the metadata generated by the data parsing module 121 to the metadata management system 17 for storage in the following manner:

[0111] Determine whether the effective duration of the new metadata is less than or equal to a preset second duration threshold and whether the predicted number of writes of the new metadata is less than or equal to the preset second number threshold: if the effective duration is less than or equal to the second duration threshold and the predicted number of writes is less than or equal to the second number threshold, the new metadata is sent to the metadata management system 17 in real time for storage; otherwise, the new metadata is sent to the data cache submodule and then the new metadata in the data cache submodule is periodically sent to the metadata management system 17 for storage (for example, it is periodically sent to the metadata management system 17 for storage at a time interval of 1 minute).

[0112] In this embodiment, the data retrieval system 18 can be configured to obtain and display the stored data corresponding to the first retrieval instruction in the distributed key-value storage subsystem 131 according to the received first retrieval instruction, and to obtain and display the metadata corresponding to the second retrieval instruction in the metadata management system 17 according to the received second retrieval instruction; at the same time, in this embodiment, the data retrieval system 18 can flexibly adjust the maximum amount of data that can be displayed for each search according to actual needs (for example, a maximum of 1,000 data items can be displayed for each search). Optionally, when the stored data and metadata are power battery-related data, the first retrieval instruction may include the number and / or data storage time of the power battery, and the second retrieval instruction may include the number and / or data storage time of the power battery. The data retrieval system 18 can search for the data of the corresponding power battery according to the number of the power battery, and can also search for the data of all power batteries within the corresponding time range according to the data storage time, and can also search for the data of the power battery with a specified number within the corresponding time range according to the number and data storage time of the power battery.

[0113] Furthermore, in this embodiment Figure 2 Each data processing node in the vehicle data processing platform shown may also include a monitoring module and a parameter configuration module. The monitoring module may be configured to monitor the working status of the data processing node and send the monitoring results to an external monitoring system, that is, the monitoring module may monitor the working status of all modules in the data processing node, including the data parsing module, the data caching module, the data aggregation analysis module, the first data asynchronous transmission module, the second data asynchronous transmission module, the metadata reading and writing module, and the message queue receiving module. For example, in this embodiment, the monitoring module may monitor the data processing volume of each of the above modules within a certain time period (such as 1s or 1min), may monitor the time taken by each of the above modules to process a certain data, may monitor the number of successful or failed data processing times of each of the above modules within a certain time period (such as 1s or 1min), etc.

[0114] The parameter configuration module can be configured to periodically obtain (e.g., obtain at intervals of 1 minute) the parameter values ​​of the parameters to be configured in the data processing node, and configure the parameters to be configured according to the parameter values. For example, the parameter configuration module can also periodically obtain the parameter values ​​of all configuration parameters in the data processing node, and configure all configuration parameters according to the obtained parameter values, thereby achieving instant parameter configuration of these modules without stopping the operation of other modules in the data processing node. Optionally, the parameter configuration module in this embodiment can communicate with an external system to obtain the corresponding parameter values, or can communicate with a system (e.g., a metadata management system) or module preset in the vehicle data processing platform for storing parameter values ​​to obtain the corresponding parameter values.

[0115] Continue to refer to the attached Figure 1 and 2 In this embodiment, the number of message queues in the first distributed message queue system 11 and the second distributed message queue system 14 can be expanded based on actual needs. The number of data processing nodes in the distributed data processing system 12 can also be expanded based on actual needs. The storage space of the distributed key-value storage subsystem 131 and the distributed object storage subsystem 132 can also be expanded based on actual needs. This achieves distributed expansion of the first distributed message queue system 11, the second distributed message queue system 14, the distributed data processing system 12, and the distributed storage system 13. This eliminates performance hotspots in the vehicle data processing platform, meaning systems whose processing capacity cannot be increased through device expansion. For example, the distributed data processing system 12 can include three data processing nodes, each of which includes a server with 8 CPUs and 15GB of memory; the distributed key-value storage subsystem 131 can include three servers for data storage, each of which has 16 CPUs, 64GB of memory, and a 3TB hard drive.

[0116] Those skilled in the art will appreciate that the systems in the platforms in the embodiments can be adaptively changed and set in one or more platforms different from the embodiments. The systems or modules or submodules in the embodiments can be combined into one system or module or submodule, and in addition they can be divided into multiple subsystems or submodules. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by alternative features that provide the same, equivalent or similar purposes.

[0117] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is intended to be within the scope of the present invention and to form different embodiments. For example, in the claims of the present invention, any one of the claimed embodiments may be used in any combination.

[0118] It should be noted that the above-described embodiments illustrate rather than limit the present invention, and that those skilled in the art will be able to devise alternative embodiments without departing from the scope of the appended claims. In the claims, the word "comprising" does not exclude the presence of elements or steps not listed in the claim. The use of the words first and second does not indicate any order; these words may be interpreted as names.

[0119] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A vehicle data processing platform, characterized in that: The vehicle data processing platform includes a first distributed message queue system, a distributed data processing system and a distributed storage system, the first distributed message queue system includes a plurality of message queues, and the distributed data processing system includes a plurality of data processing nodes; The message queue is configured to receive vehicle data sent by a first external system; The data processing node is configured to obtain vehicle data received by a message queue pre-bound thereto, perform data analysis on the vehicle data and send the data analysis results to the distributed storage system for storage; The vehicle data processing platform further includes a metadata management system configured to store metadata corresponding to the vehicle data; The data processing node includes a metadata reading and writing module, the metadata reading and writing module includes a data caching submodule and a data reading submodule, and the data reading submodule is configured to obtain the metadata stored in the metadata management system in the following manner: Determine whether the effective duration of the metadata is less than or equal to a preset first duration threshold and whether the predicted number of reads of the metadata is less than or equal to a preset first number threshold, If the effective duration is less than or equal to the first duration threshold and the predicted number of reads is less than or equal to the first number threshold, the metadata is acquired in real time; otherwise, the metadata is acquired periodically and cached to the data cache submodule.

2. The vehicle data processing platform according to claim 1, characterized in that: The data processing node includes a data parsing module, a data caching module and a data aggregation analysis module, and the data aggregation analysis module includes a data statistics submodule and a data alarm submodule; The data parsing module is configured to parse the vehicle data and send the parsing result to the data cache module; The data statistics submodule is configured to obtain cached data in the data cache module, perform data statistics on the cached data and send the data statistics results to the distributed storage system for storage; The data alarm submodule is configured to obtain cached data in the data cache module, perform alarm analysis on the cached data and send the alarm analysis result to the distributed storage system for storage.

3. The vehicle data processing platform according to claim 2, characterized in that: The data cache module includes multiple data pools and a cache queue and a read-write thread corresponding to each data pool. The data pool is configured to cache the parsing results of the vehicle data through a hash table.

4. The vehicle data processing platform according to claim 3, characterized in that: When the vehicle data is power battery related data, the data cache module is configured to allocate a data pool for the parsing result of the vehicle data in the following manner: Data_pool ID =Hash(Battery ID )%M Among them, the Data_pool ID is the number of the data pool, the Battery ID is the serial number of the power battery, M is the total number of data pools in the data cache module, Hash represents a hash function, and % represents a remainder operation.

5. The vehicle data processing platform according to claim 2, characterized in that: The distributed storage system includes a distributed key-value storage subsystem, and the data processing node also includes a first data asynchronous sending module; The first data asynchronous sending module is configured to receive the analysis results output by the data analysis module, receive the data statistics results and the alarm analysis results output by the data aggregation analysis module, and send the analysis results, data statistics results and alarm analysis results to the distributed key-value storage subsystem for storage through asynchronous transmission; The distributed key-value storage subsystem is a storage subsystem built based on a Key-Value pair storage structure.

6. The vehicle data processing platform according to claim 2, characterized in that: The distributed storage system further includes a distributed object storage subsystem, and the vehicle data processing platform further includes a second distributed message queue system and a periodic persistence service system; The data processing node also includes a second data asynchronous sending module, which is configured to receive the analysis results output by the data analysis module, receive the data statistics results and the alarm analysis results output by the data aggregation analysis module, and send the analysis results, data statistics results and alarm analysis results to the second distributed message queue system through asynchronous transmission; The periodic persistence service system is configured to periodically send the parsing results, data statistics results and alarm analysis results received by the second distributed message queue system to the distributed object storage subsystem for storage; The distributed object storage subsystem is a storage subsystem built based on an object storage structure.

7. The vehicle data processing platform according to claim 6, characterized in that: The vehicle data processing platform also includes a data warehouse, and the distributed object storage subsystem is further configured to add corresponding metadata to the data stored therein, and send the data with the metadata added to the data warehouse for storage; The metadata includes the name of the data table corresponding to the data in the distributed object storage subsystem, the name of the database to which the data table belongs, the partition information of the data table and the storage path of the data.

8. The vehicle data processing platform according to claim 6, characterized in that: The second distributed message queue system is further configured to receive a data acquisition instruction sent by a second external system, and send data corresponding to the data acquisition instruction to the second external system; The data corresponding to the data acquisition instruction include analysis results, and / or data statistics results, and / or alarm analysis results.

9. The vehicle data processing platform according to claim 5, characterized in that: When the vehicle data is power battery related data, the metadata corresponding to the power battery related data includes the number of the power battery, the number of the vehicle / charging facility where the power battery is located, the location of the vehicle / charging facility where the power battery is located, and the data name, data format and data source of the power battery related data.

10. The vehicle data processing platform according to claim 9, characterized in that: The metadata reading and writing module is configured to obtain the metadata of the vehicle data stored in the metadata management system and send the metadata to the data parsing module, and to obtain new metadata generated by the data parsing module after parsing the vehicle data according to the metadata and send the new metadata to the metadata management system for storage.

11. The vehicle data processing platform according to claim 10, characterized in that: The metadata reading and writing module further includes a data caching submodule and a data writing submodule, wherein the data writing submodule is configured to send new metadata to the metadata management system for storage in the following manner: Determine whether the effective duration of the new metadata is less than or equal to a preset second duration threshold and whether the predicted number of write times of the new metadata is less than or equal to a preset second number threshold, If the effective duration is less than or equal to the second duration threshold and the predicted number of writes is less than or equal to the second number threshold, the new metadata is sent to the metadata management system in real time for storage; Otherwise, after sending the new metadata to the data cache submodule, the new metadata in the data cache submodule is periodically sent to the metadata management system for storage.

12. The vehicle data processing platform according to claim 9, characterized in that: The vehicle data processing platform also includes a data retrieval system, The data retrieval system is configured to acquire and display stored data corresponding to the first retrieval instruction in the distributed key-value storage subsystem according to the received first retrieval instruction, and acquire and display metadata corresponding to the second retrieval instruction in the metadata management system according to the received second retrieval instruction; Wherein, when the stored data and the metadata are power battery related data, the first retrieval instruction includes the power battery number and / or data storage time, and the second retrieval instruction includes the power battery number and / or data storage time.

13. The vehicle data processing platform according to any one of claims 1 to 12, characterized in that: The first distributed message queue system is also configured to perform the following operations: When a data processing node failure is detected, the binding relationship between the data processing node and its corresponding message queue is released, and a binding relationship is established between the message queue and a data processing node that has not failed, so that the data processing node that has not failed can obtain the vehicle data received by the message queue.

14. The vehicle data processing platform according to any one of claims 1 to 12, characterized in that: The data processing node further includes a monitoring module, which is configured to monitor the working status of the data processing node and send the monitoring result to an external monitoring system.

15. The vehicle data processing platform according to any one of claims 1 to 12, characterized in that: The data processing node further includes a parameter configuration module, and the parameter configuration module is configured to periodically obtain parameter values ​​of parameters to be configured in the data processing node, and perform parameter configuration on the parameters to be configured according to the parameter values.

Images