A vehicle data integration method
By adopting long connection method and multi-thread pooling technology in the vehicle data integration system, asynchronous, non-blocking and classification processing is achieved, concurrency and real-time problems in the existing technology are solved, and efficient and reliable vehicle data integration is achieved, satisfying the Real-time processing requirements for vehicle data of ten thousand levels.
Patent Information
- Application Number
- CN202410561120.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-05-08
AI Technical Summary
The existing vehicle data integration system is prone to paralysis under high concurrency conditions and cannot meet the real-time transmission requirements of seconds. The integrated method of synchronous blocking leads to an increase in delay, resulting in the loss of real-time data in the vehicle.
The long connection method is used to realize the asynchronous, non-blocking, decoding and classification integration of vehicle data. Through a multiplexer selector, the main thread pool and slave thread pool are combined, and the callback listening mechanism and header verification are used to ensure the concurrency of data integration. , real-time and accuracy.
It realizes processing of data integration tasks of 10,000-level vehicles, reduces latency, improves processing speed and overall robustness, avoids data loss and decoding failure, and ensures data integrity and fault tolerance.
Smart Images

Figure CN118426915B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of databases, and in particular to a vehicle data integration method. Background Art
[0002] In the field of transportation, vehicle data has the characteristics of high update frequency, large data volume, and high real-time requirements. Currently, most vehicle data integration systems integrate data based on interfaces. This synchronous blocking integration method limits the amount of data that can be received from on-board devices at the same time per unit time, resulting in increased latency and failure to meet the requirements of real-time transmission in seconds. In addition, the interface is prone to paralysis under high concurrency conditions, resulting in the loss of real-time vehicle data.
[0003] In order to better perceive the operation status of urban vehicles, monitor the operation trajectory of important vehicles, and provide assistance to the management unit for command and decision-making, the present invention proposes a vehicle data integration method. Summary of the invention
[0004] In order to make up for the defects of the prior art, the present invention provides a simple and efficient vehicle data integration method.
[0005] The present invention is achieved through the following technical solutions:
[0006] A vehicle data integration method, characterized in that: asynchronous, non-blocking, decoding and classification integration of vehicle data is realized based on a long connection method; specifically as follows:
[0007] Adopt multiplexing selector, combine main thread pool and slave thread pool, realize non-blocking capability of vehicle data integration, and ensure concurrency of data integration;
[0008] In case the business logic processing process takes a long time, a callback monitoring mechanism is adopted. A callback method is returned first. After the calling component completes the processing, the task status is obtained by monitoring the callback object. The caller is notified through the status, notification and callback to realize the asynchronous capability of vehicle data integration and ensure the real-time nature of data integration.
[0009] Use the method of identification header inspection, special character escape replacement and recursive parsing of packet length and integrity to achieve data packet boundary division, thereby realizing vehicle data integration decoding capability and ensuring the accuracy and effectiveness of data integration;
[0010] Combined with the classification of vehicle data business attributes, the classified data identifier is configured, the identifier value in the vehicle data packet is determined, and the processor in the corresponding factory class method is matched to realize the classification processing capability of vehicle data integration, thereby avoiding the confusion caused by high-frequency data crossover and improving the processing speed.
[0011] When implementing the non-blocking capability of vehicle data integration, first abstract the main thread pool and the slave thread pool, the main thread pool is responsible for receiving client connections, and the slave thread pool is responsible for reading vehicle data;
[0012] The main thread pool and the slave thread pool are event loop groups, each event loop group includes several event loop components, and each event loop component corresponds to a thread that continuously executes processing tasks in a loop;
[0013] Each event loop component includes a selector and a task queue, wherein the selector is used to listen to the network communication bound to it, and the task queue is used to process event tasks;
[0014] In the main thread pool, each event loop component performs the following three steps:
[0015] 1) Poll and receive events in a load-balanced manner;
[0016] 2) Process the received event, establish a connection with the client, generate a network channel, and register it with the selector of the event loop component from the thread pool;
[0017] 3) Run the tasks in the task queue.
[0018] From the thread pool, each event loop component performs the following three steps:
[0019] 1) Poll and read vehicle data in a load-balanced manner;
[0020] 2) The network channel is associated with the corresponding pipeline to process the read vehicle data;
[0021] 3) Run the tasks in the task queue.
[0022] When implementing the asynchronous capability of vehicle data integration, pipelines are used for business processing after vehicle data is read. Each pipeline corresponds to a network channel, and several processors are maintained in the pipeline.
[0023] After the vehicle data is processed by the processor, the caller cannot get the result immediately. In this case, the callback monitoring mechanism is used to return a callback object immediately. The listener is registered to the callback object, and the callback notifies the caller when the vehicle data processing operation succeeds, fails or is canceled.
[0024] In the callback object, it is determined whether the current operation is completed, whether the completed current operation is successful, and whether the completed current operation is cancelled. If not successful, the failure reason of the completed current operation is obtained.
[0025] The data structure of the data packet includes five parts: header identifier, data header, data body, CRC checksum and tail identifier;
[0026] When implementing the vehicle data integration decoding capability, first read the data packet in the buffer and convert it into a hexadecimal string;
[0027] A fixed string identifier verification method is used to verify whether the header identifier in the data packet is valid. To prevent extreme situations, if the header identifier verification fails, the corresponding data packet is discarded, and the remaining data in the previous packet cache is discarded at the same time to prevent the remaining data in the previous packet cache from not containing the data start character, resulting in the spliced data packet always failing the verification;
[0028] Based on the head and tail character rules, special character strings in the data content are replaced by escapes;
[0029] Verify the length of the data packet. Since the header, data header, CRC checksum and tail identifier are fixed lengths, there is a minimum length for the data packet. When the length of the data packet is less than the minimum length of the entire data packet, the data packet is stored in a temporary cache queue; after reading the next data packet, it is first merged with the data packet stored in the temporary cache queue, and then the decoding process is repeated; when the length of the data packet is greater than the minimum length of the entire data packet, a data packet integrity check is performed;
[0030] The integrity of the data packet is verified according to the data packet integrity verification algorithm, and the incomplete data packets that do not meet the integrity rules are stored in a temporary cache queue, waiting to be merged with the next data packet and re-decoding the process; data parsing operations are performed on complete data packets that meet the integrity rules.
[0031] When realizing the classification processing capability of vehicle data integration, vehicle data is divided into three categories: vehicle static data, vehicle dynamic data and vehicle alarm data according to data frequency and business logic;
[0032] The vehicle static data refers to administrative and operational management data related to the vehicle's road transport business activities that remain unchanged for a certain period of time, including but not limited to data on operating vehicles, operators, employees, transport management agencies and operating routes;
[0033] The vehicle dynamic data refers to the relevant data generated by the vehicle during operation, including but not limited to vehicle positioning data, operating status, personnel, cargo and route data;
[0034] The vehicle alarm data is vehicle alarm status data, and the alarm status includes but is not limited to overspeed alarm, fatigue driving alarm, emergency alarm, entering designated area alarm, leaving designated area alarm, road section congestion alarm, dangerous road section alarm, crossing the boundary alarm, theft alarm, hijacking alarm and driver behavior alarm;
[0035] The service type of different types of vehicle data is identified. The integrated vehicle data uniformly starts with up, the static data service identifier is base, the dynamic data service identifier is exg, and the warning data service identifier is warn. For specific data under different vehicle types, the original identification header of the belonging service data type is continued, and the sub-service type identifier is supplemented at the end, such as the vehicle driver data type identifier is up_base_driver_info;
[0036] When parsing data, the factory class calls the matching processor to complete different types of data processing according to the hexadecimal values corresponding to different business identifiers, and then associates the parsed vehicle data with the business to continue the operation.
[0037] A vehicle data integration device, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.
[0038] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method when executed by a processor.
[0039] The beneficial effects of the present invention are as follows: the vehicle data integration method can not only make full use of CPU resources and avoid multi-thread switching overhead, but also reduce delays, ensure the utilization efficiency of business processing threads, the integrity of data after decoding, and the overall robustness and fault tolerance, and can realize the processing of tens of thousands of vehicle data integration tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Attached Figure 1 Schematic diagram of the vehicle data integration method of the present invention.
[0042] Attached Figure 2 It is a schematic diagram of the vehicle data integration decoding method of the present invention. DETAILED DESCRIPTION
[0043] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0044] The vehicle data integration method realizes asynchronous, non-blocking, decoding and classification integration of vehicle data based on a long connection method; the details are as follows:
[0045] Adopt multiplexing selector, combine main thread pool and slave thread pool, realize non-blocking capability of vehicle data integration, and ensure concurrency of data integration;
[0046] In case the business logic processing process takes a long time, a callback monitoring mechanism is adopted. A callback method is returned first. After the calling component completes the processing, the task status is obtained by monitoring the callback object. The caller is notified through the status, notification and callback to realize the asynchronous capability of vehicle data integration and ensure the real-time nature of data integration.
[0047] Use the method of identification header inspection, special character escape replacement and recursive parsing of packet length and integrity to achieve data packet boundary division, thereby realizing vehicle data integration decoding capability and ensuring the accuracy and effectiveness of data integration;
[0048] Combined with the classification of vehicle data business attributes, the classified data identifier is configured, the identifier value in the vehicle data packet is determined, and the processor in the corresponding factory class method is matched to realize the classification processing capability of vehicle data integration, thereby avoiding the confusion caused by high-frequency data crossover and improving the processing speed.
[0049] When implementing the non-blocking capability of vehicle data integration, first abstract the main thread pool and the slave thread pool, the main thread pool is responsible for receiving client connections, and the slave thread pool is responsible for reading vehicle data;
[0050] The main thread pool and the slave thread pool are event loop groups, each event loop group includes several event loop components, and each event loop component corresponds to a thread that continuously executes processing tasks in a loop;
[0051] Each event loop component includes a selector and a task queue, wherein the selector is used to listen to the network communication bound to it, and the task queue is used to process event tasks;
[0052] In the main thread pool, each event loop component performs the following three steps:
[0053] 1) Poll and receive events in a load-balanced manner;
[0054] 2) Process the received event, establish a connection with the client, generate a network channel, and register it with the selector of the event loop component from the thread pool;
[0055] 3) Run the tasks in the task queue.
[0056] From the thread pool, each event loop component performs the following three steps:
[0057] 1) Poll and read vehicle data in a load-balanced manner;
[0058] 2) The network channel is associated with the corresponding pipeline to process the read vehicle data;
[0059] 3) Run the tasks in the task queue.
[0060] When implementing the asynchronous capability of vehicle data integration, pipelines are used for business processing after vehicle data is read. Each pipeline corresponds to a network channel, and several processors are maintained in the pipeline.
[0061] After the vehicle data is processed by the processor, the caller cannot get the result immediately. In this case, the callback monitoring mechanism is used to return a callback object immediately. The listener is registered to the callback object, and the callback notifies the caller when the vehicle data processing operation succeeds, fails or is canceled.
[0062] In the callback object, it is determined whether the current operation is completed, whether the completed current operation is successful, and whether the completed current operation is cancelled. If not successful, the failure reason of the completed current operation is obtained.
[0063] The data structure of the data packet includes five parts: header identifier, data header, data body, CRC checksum and tail identifier;
[0064] When implementing vehicle data integration decoding capabilities, the implementation steps are as follows:
[0065] Step 1), after reading the data packet in the buffer, first determine whether there is data in the temporary cache queue, merge the existing data, and then convert it into a hexadecimal string;
[0066] Step 2) Use a fixed string identifier verification method to verify whether the header identifier in the data packet is valid; to prevent extreme situations, if the header identifier verification fails, the corresponding data packet is discarded, and the remaining data in the previous packet cache is discarded at the same time to prevent the remaining data in the previous packet cache from not containing the data start character, resulting in the spliced data packet always failing the verification;
[0067] Step 3), based on the head identifier and tail identifier character rules, escape and replace the special character string in the data content;
[0068] Step 4) Verify the length of the data packet. Since the header, data header, CRC check code and tail identifier are fixed lengths, there is a minimum length for the data packet. When the length of the data packet is less than the minimum length of the entire data packet, the data packet is stored in a temporary cache queue and waits to be merged with the next data packet to complete the decoding operation; when the length of the data packet is greater than the minimum length of the entire data packet, a data packet integrity check is performed;
[0069] Step 5) Verify the integrity of the data packet according to the data packet integrity verification algorithm, store the incomplete data packet that does not meet the integrity rules in a temporary cache queue, and wait to be merged with the next data packet to complete the decoding operation; perform data parsing operations on the complete data packet that meets the integrity rules.
[0070] When realizing the classification processing capability of vehicle data integration, vehicle data is divided into three categories: vehicle static data, vehicle dynamic data and vehicle alarm data according to data frequency and business logic;
[0071] The vehicle static data refers to administrative and operational management data related to the vehicle's road transport business activities that remain unchanged for a certain period of time, including but not limited to data on operating vehicles, operators, employees, transport management agencies and operating routes;
[0072] The vehicle dynamic data refers to the relevant data generated by the vehicle during operation, including but not limited to vehicle positioning data, operating status, personnel, cargo and route data;
[0073] The vehicle alarm data is vehicle alarm status data, and the alarm status includes but is not limited to overspeed alarm, fatigue driving alarm, emergency alarm, entering designated area alarm, leaving designated area alarm, road section congestion alarm, dangerous road section alarm, crossing the boundary alarm, theft alarm, hijacking alarm and driver behavior alarm;
[0074] The service type of different types of vehicle data is identified. The integrated vehicle data uniformly starts with up, the static data service identifier is base, the dynamic data service identifier is exg, and the warning data service identifier is warn. For specific data under different vehicle types, the original identification header of the belonging service data type is continued, and the sub-service type identifier is supplemented at the end, such as the vehicle driver data type identifier is up_base_driver_info;
[0075] When parsing data, the factory class calls the matching processor to complete different types of data processing according to the hexadecimal values corresponding to different business identifiers, and then associates the parsed vehicle data with the business to continue the operation.
[0076] The vehicle data integration device comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.
[0077] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0078] Compared with the existing technology, the vehicle data integration method has the following characteristics:
[0079] First, the data interaction between the main thread and the slave thread is simple and the responsibilities are clear. The main thread only needs to receive new connections, and the slave thread completes the subsequent vehicle data reading and processing. It can easily increase the number of selector instances to fully utilize CPU resources, while avoiding multi-thread switching overhead, and can realize the processing of tens of thousands of vehicle data integration tasks.
[0080] Second, asynchronous operations reduce latency and ensure efficient use of business processing threads. They also ensure overall robustness and fault tolerance for timeouts and exceptions, and are the basis for achieving real-time vehicle data integration at the 10,000-level.
[0081] Third, the fixed string identification verification method designed based on business standards ensures the validity of the data packet data and avoids the decoding failure of the entire data due to a single data packet anomaly.
[0082] Fourth, a special string escape replacement method based on the head identifier and tail identifier character rules converts special characters to avoid conflicts during data transmission.
[0083] Fifth, data packet length check and integrity check solve the problems of missing data boundaries, too large or too small data volume in a single packet during data packet transmission, ensuring the integrity of the data after decoding.
[0084] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.
Claims
1. A vehicle data integration method, characterized in that: Asynchronous, non-blocking, decoding and classification integration of vehicle data is achieved based on long connection mode; the details are as follows: Adopt multiplexing selector, combine main thread pool and slave thread pool, realize non-blocking capability of vehicle data integration, and ensure concurrency of data integration; When implementing the non-blocking capability of vehicle data integration, first abstract the main thread pool and the slave thread pool, the main thread pool is responsible for receiving client connections, and the slave thread pool is responsible for reading vehicle data; The main thread pool and the slave thread pool are event loop groups, each event loop group includes several event loop components, and each event loop component corresponds to a thread that continuously executes processing tasks in a loop; Each event loop component includes a selector and a task queue, wherein the selector is used to listen to the network communication bound to it, and the task queue is used to process event tasks; In the main thread pool, each event loop component performs the following three steps: 1) Poll and receive events in a load-balanced manner; 2) Process the received event, establish a connection with the client, generate a network channel, and register it with the selector of the event loop component from the thread pool; 3) Run tasks in the task queue; From the thread pool, each event loop component performs the following three steps: 1) Poll and read vehicle data in a load-balanced manner; 2) The network channel is associated with the corresponding pipeline to process the read vehicle data; 3) Run tasks in the task queue; In case the business logic processing process is time-consuming, a callback monitoring mechanism is adopted. A callback method is returned first. After the calling component completes the processing, the task status is obtained by monitoring the callback object. The caller is notified through the status, notification and callback to realize the asynchronous capability of vehicle data integration and ensure the real-time performance of data integration. Use the method of identification header inspection, special character escape replacement and recursive parsing of packet length and integrity to achieve data packet boundary division, thereby realizing vehicle data integration decoding capability and ensuring the accuracy and effectiveness of data integration; The data structure of the data packet includes five parts: header identifier, data header, data body, CRC checksum and tail identifier; When implementing the vehicle data integration decoding capability, first read the data packet in the buffer and convert it into a hexadecimal string; A fixed string identifier verification method is used to verify whether the header identifier in the data packet is valid. If the header identifier verification fails, the corresponding data packet is discarded, and the remaining data in the previous packet cache is discarded at the same time to prevent the remaining data in the previous packet cache from not containing the data start character, causing the spliced data packet to always fail the verification; Based on the custom header and footer character rules, special strings in the data content are replaced by escape characters; Check the length of the data packet. When the length of the data packet is less than the minimum length of the whole data packet, store the data packet in a temporary cache queue. After reading the next data packet, merge it with the data packet stored in the temporary cache queue and then re-decode it. When the length of the data packet is greater than the minimum length of the whole data packet, perform a data packet integrity check. Verify the integrity of the data packet according to the data packet integrity verification algorithm, store the incomplete data packets that do not meet the integrity rules in a temporary cache queue, wait to be merged with the next data packet, and re-decode the process; perform data parsing operations on the complete data packets that meet the integrity rules; Combined with the classification of vehicle data business attributes, the classified data identifier is configured, the identifier value in the vehicle data packet is determined, and the processor in the corresponding factory class method is matched to realize the classification processing capability of vehicle data integration, thereby avoiding the confusion caused by high-frequency data crossover and improving the processing speed.
2. The vehicle data integration method according to claim 1, characterized in that: When implementing the asynchronous capability of vehicle data integration, pipelines are used for business processing after vehicle data is read. Each pipeline corresponds to a network channel, and several processors are maintained in the pipeline. After the vehicle data is processed by the processor, a callback object is immediately returned using the callback monitoring mechanism; the listener is registered to the callback object, and the callback notifies the caller when the vehicle data processing operation succeeds, fails, or is canceled.
3. The vehicle data integration method according to claim 2, characterized in that: In the callback object, it is determined whether the current operation is completed, whether the completed current operation is successful, and whether the completed current operation is cancelled. If not successful, the failure reason of the completed current operation is obtained.
4. The vehicle data integration method according to claim 1, characterized in that: The minimum length of the entire packet of data is the sum of the fixed lengths of the header, data header, CRC checksum, and tail.
5. The vehicle data integration method according to claim 1, characterized in that: When realizing the classification processing capability of vehicle data integration, vehicle data is divided into three categories: vehicle static data, vehicle dynamic data and vehicle alarm data according to data frequency and business logic; Different types of vehicle data are identified by their service types. The integrated vehicle data uniformly starts with "up". The static data service identifier is "base", the dynamic data service identifier is "exg", and the warning data service identifier is "warn". When parsing data, the factory class calls the matching processor to complete different types of data processing according to the hexadecimal values corresponding to different business identifiers, and then associates the parsed vehicle data with the business to continue the operation.
6. The vehicle data integration method according to claim 5, characterized in that: The vehicle static data refers to the fixed administrative and operational management data related to the vehicle's road transport business activities, including but not limited to data on operating vehicles, operators, employees, transport management agencies and operating routes; The vehicle dynamic data refers to the relevant data generated by the vehicle during operation, including but not limited to vehicle positioning data, operating status, personnel, cargo and route data; The vehicle alarm data is vehicle alarm status data, and the alarm status includes but is not limited to speeding alarm, fatigue driving alarm, emergency alarm, entering a designated area alarm, leaving a designated area alarm, road congestion alarm, dangerous road section alarm, crossing the boundary alarm, theft alarm, hijacking alarm and driver behavior alarm.
7. A vehicle data integration device, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.
8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Node type data acquisition system
CN115865724A
Multi-thread data processing method and device, electronic equipment and storage medium
CN116382857A
Perception data processing method and device, electronic equipment and storage medium
CN117216682A
