New energy vehicle operation data processing method, electronic equipment and storage medium

Abstract

The invention discloses a new energy vehicle operation data processing method, electronic equipment and a storage medium, and the method comprises the steps: initializing a server memory data processing window and window queue parameters, starting a collection program, and collecting vehicle operation data according to a configured rolling strategy; window matching is carried out on the collected data to obtain a preprocessing window queue, and numerical analysis and correction are carried out on the preprocessing window queue to ensure the quality and credibility of the data; performing connection aggregation processing on the preprocessed window queue data to ensure the timeliness and integrity of the data under a time sequence condition; and according to the requirement of an analysis task, the data is disassembled and distributed to the corresponding parallel processors to complete corresponding business calculation, and a calculation result is sent to an input queue to wait for scheduling and is written into a storage medium of a persistence layer in a batch form. Through dynamically constructing a vehicle data parallel processing window and a plurality of window matching strategies, the data throughput of a single node is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a data processing method for massive time series data reported by new energy vehicles under the condition of limited computing resources. Background Art

[0002] The processing method for the massive time series data reported by new energy vehicles generally adopts thread plus cache technology or stream technology to realize the task of real-time processing of continuous data with time series characteristics.

[0003] Both methods have great limitations when processing data with massive group timing requirements. For example, they cannot fully utilize the full performance of a single node; they are difficult to cope with high concurrency and high throughput write requirements; and the implementation of stream processing in the face of massive timing stream group processing requirements is also a great challenge.

[0004] In view of this, a data processing method for massive time series data reported by new energy vehicles under the condition of limited computing resources is a topic worthy of study. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to improve the data throughput and data processing efficiency of a single data processing node to reduce the cost investment of data processing.

[0006] In order to solve the above technical problems, the present invention provides a method for processing new energy vehicle operation data, the method comprising the following steps:

[0007] S1. Initialize the server memory data processing window and window queue parameters, start the collection program and collect vehicle operation data according to the configured rolling strategy;

[0008] S2. Window matching is performed on the collected data according to a window matching algorithm to obtain a preprocessing window queue, and numerical analysis and correction are performed on the preprocessing window queue to ensure the quality and credibility of the data;

[0009] S3, performing connection aggregation processing on the pre-processed window queue data to ensure the timeliness and integrity of the data under time series conditions;

[0010] S4. According to the needs of the analysis task, the data is disassembled and distributed to the corresponding parallel processors to complete the corresponding business calculations, the calculation results are sent to the input queue to wait for scheduling, and are written to the storage medium of the persistence layer in batches.

[0011] Furthermore, the S3 also includes:

[0012] Determine whether the data after the connection aggregation is invalid. If there is an invalid situation, send the data to the verification queue for manual verification; if there is no invalid situation, determine whether the data after the connection aggregation is complete. If the data is complete, proceed to the next step S4. If the data is incomplete, return to S3 and perform connection aggregation again.

[0013] Furthermore, the initialization of the server memory data processing window and window queue parameters in S1 also includes:

[0014] During initialization, the system creates a corresponding value queue with adjustable length for each window, and simultaneously starts a scheduling module to manage the scheduling relationship between the window and the value queue to ensure reasonable data distribution.

[0015] Furthermore, the scheduling module dynamically adjusts the numerical queue length of the current window within a certain range according to the queue length of each window and the consumption rate of the window.

[0016] Furthermore, the scheduling module can monitor the consumption status of the window queue. When the window queue is congested, it locks the vehicle data in the window queue that is congested, adjusts the consumption priority of the corresponding vehicle data, and adopts a diversion strategy to transfer the vehicle data that is not congested to other windows.

[0017] Furthermore, the step of collecting the vehicle operation data according to the configured rolling strategy in S1 further includes:

[0018] Establish grading and window marking for consumption of vehicle data;

[0019] The scheduling module dynamically maintains the classification and window marking information according to actual conditions;

[0020] The vehicle data is window mapped through a hash remainder strategy or a polling strategy to ensure that the data of the same vehicle can only be consumed by the same window and that the vehicle data distributed in each window is balanced.

[0021] Furthermore, before obtaining the preprocessing window queue, S2 also includes cleaning the collected data:

[0022] For outliers, the statistical characteristics of vehicle data are used to detect outliers and the detection results are corrected;

[0023] For missing values, the deletion method, filling method, interpolation method and model prediction method are combined to correct the missing values;

[0024] For outliers, the median absolute deviation is used as the basis, combined with sliding window linear interpolation and isolation forest analysis methods to eliminate discrete data.

[0025] An electronic device, comprising:

[0026] one or more processors;

[0027] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the method for processing the operating data of the new energy vehicle.

[0028] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of a computer, the computer executes the method for processing operating data of a new energy vehicle.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention greatly improves the data throughput of a single node by dynamically constructing a vehicle data parallel processing window and using a variety of window matching strategies and adaptive queues, so that the massive time series data reported by new energy vehicles can be efficiently processed under the condition of limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical scheme and technical effect of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0033] The present invention aims to provide a method for processing new energy vehicle operation data. Figure 1 , the method mainly comprises the following steps:

[0034] Step 1: Initialize the server memory data processing window and window queue parameters; start the collection program and collect vehicle operation data according to the configured rolling strategy. Specifically:

[0035] 1.1. Binding threads to CPUs: In this embodiment, a Java-based multi-threaded data processing system can utilize its Thread Affinity technology to initialize and process threads, and bind specific threads to specific CPUs. This has the following advantages: on the one hand, it can reduce the performance loss caused by CPU switching; on the other hand, since the thread is fixed to run on a single CPU, it can ensure that all computing resources of the CPU are used for the thread, thereby maximizing computing power.

[0036] 1.2. Data processing window and value queue: Each thread after initialization represents a data processing window. During initialization, the system will create a value queue with adjustable length for each window to temporarily store the data to be processed.

[0037] 1.3: Scheduling module: During initialization, the system creates a corresponding length-adjustable numerical queue for each window, and starts a scheduling module to manage the scheduling relationship between the window and the numerical queue to ensure reasonable data distribution. The scheduling module dynamically adjusts the length of the numerical queue of the current window within a certain range according to the length of each window queue and the consumption rate of the window. The scheduling module can also monitor the consumption status of the window queue. When the window queue is congested, it locks the vehicle data in the window queue that is congested, adjusts the consumption priority of the corresponding vehicle data, and adopts a diversion strategy to transfer the vehicle data that is not congested to other windows.

[0038] Due to the complex operating environment of the vehicle, the collected vehicle data may be abnormal, missing, or outlier.

[0039] In this embodiment, before obtaining the preprocessing window queue, the collected data is also cleaned, specifically including:

[0040] For outliers, the statistical characteristics of vehicle data are used to detect outliers and the detection results are corrected;

[0041] For missing values, the deletion method, filling method, interpolation method and model prediction method are combined to correct the missing values;

[0042] For outliers, the median absolute deviation (MAD) is used as the basis, combined with sliding window linear interpolation and isolation forest analysis methods to eliminate discrete data.

[0043] Step 2: Window matching is performed on the collected data according to the window matching algorithm to obtain a preprocessing window queue, and numerical analysis and correction are performed on the preprocessing window queue to ensure the quality and credibility of the data. Specifically:

[0044] 2.1. Establish grading and window marking for vehicle data consumption;

[0045] 2.2. The scheduling module dynamically maintains the classification and window marking information according to the actual situation;

[0046] 2.3. Use hash remainder strategy or polling strategy to map vehicle data to windows to ensure that data of the same vehicle can only be consumed by the same window and that vehicle data distributed in each window is balanced, to prevent vehicle data skew from causing window task backlog and avalanche effect, ultimately leading to a decline in data throughput of the entire node.

[0047] It should be noted that the hash remainder strategy and the polling strategy are two common load balancing algorithms, which will not be described in detail here.

[0048] Furthermore, when a window queue is backlogged, the window data will be dynamically rebalanced, and the vehicle data in the backlogged queue will be transferred to other windows. Specifically, the program uses a pre-screening strategy to pre-screen normal vehicle data from the scheduling buffer and assigns it to a non-backlogged window for processing. For excessively serious data backlogs, an alarm will be generated to warn the user, prompting external intervention.

[0049] Step 3: Perform connection aggregation processing on the preprocessed window queue data to ensure the timeliness and integrity of the data under time series conditions.

[0050] The connection aggregation here needs to solve the problem of data queue jumping caused by the segmented reception of a complete data and the shared window queue of multiple vehicles. Affected by the protocol restrictions and the actual situation of the vehicle, a complete protocol data packet may be split into multiple messages for transmission. In the case of multiple vehicles sharing a window queue, the complete data of a vehicle will not be continuous in the window queue, resulting in a deadlock in the window queue. Introducing queue jumping logic in the conventional queue allows the late-coming data to be connected and aggregated into a new data packet based on its offset and the data in front of the queue to wait for scheduling, or when it is found that the data packet in front of the queue is incomplete, it is allowed to lower its priority and move back, so that the complete vehicle data can get priority processing.

[0051] Determine whether the data after connection aggregation is invalid. If there is an invalid situation, send the data to the verification queue for manual verification; if there is no invalid situation, determine whether the data after connection aggregation is complete. If the data is complete, proceed to the next step. If the data is incomplete, return to step three and perform connection aggregation again.

[0052] Since the collected operation data of new energy vehicles is affected by the protocol, one piece of operation data may be disassembled into multiple blocks for transmission, that is, a large data set is split into smaller data blocks. The purpose of this is to enable the data to be effectively distributed to different processors and improve data processing efficiency. In view of this, step three is to check the validity and integrity of the vehicle data before processing the data, including:

[0053] Check the timeliness of the data: Since the processed data needs to ensure that the bicycle data must be consistent, it is necessary to perform time series recognition, prediction, and interpolation and completion on the collected data. Prediction and completion can improve the quality of collected data within a smaller range.

[0054] Check the integrity of the data: It is necessary to identify whether the data is a whole packet or a sub-packet. If it is a sub-packet, the packet data needs to be aligned. Under actual conditions, due to the influence of the environment, problems such as disordered order of multiple packets may occur. The system uses buffering technology and dislocation identification technology to rearrange the disordered data and realize the merging of data packets.

[0055] If the data is found to be invalid, that is, the data has timed out or is incoherent, the processing of this data will be terminated. Invalid data cannot be simply discarded, but must be sent to the verification queue for manual verification later.

[0056] If the confirmed data is not a complete data packet (usually the data to be sent is split into multiple packets when it is too large), then the data is temporarily stored and step three is repeated to aggregate the data again.

[0057] Step 4: Disassemble the data and distribute it to the corresponding parallel processors according to the needs of the analysis task to complete the corresponding business calculations. Send the calculation results to the input queue for scheduling and write them to the storage medium of the persistence layer in batches.

[0058] Calculation result queue: The calculation results completed by the parallel processor will not be written to the storage immediately, but will be sent to an input queue first. This allows for unified management of the results and effective control of access to the storage system to prevent system performance degradation due to excessive concurrent access. A scheduling strategy is then used to control the timing of dequeuing and writing the calculation results.

[0059] Batch processing: The calculation results will be written to persistent storage in batches at appropriate times, such as hard disks or distributed file systems. Batch processing can reduce the number of I / O operations and improve the overall performance of the system.

[0060] The present invention also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the method for processing new energy vehicle operating data as described above.

[0061] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the method for processing operating data of a new energy vehicle as described above.

[0062] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing new energy vehicle operation data, characterized in that: The method comprises the following steps: S1. Initialize the server memory data processing window and window queue parameters, start the collection program and collect vehicle operation data according to the configured rolling strategy; S2. Window matching is performed on the collected data according to a window matching algorithm to obtain a preprocessing window queue, and numerical analysis and correction are performed on the preprocessing window queue to ensure the quality and credibility of the data; S3, performing connection aggregation processing on the pre-processed window queue data to ensure the timeliness and integrity of the data under time series conditions; S4. According to the needs of the analysis task, the data is disassembled and distributed to the corresponding parallel processors to complete the corresponding business calculations, the calculation results are sent to the input queue to wait for scheduling, and are written to the storage medium of the persistence layer in batches.

2. The method for processing new energy vehicle operation data according to claim 1, characterized in that: The S3 further includes: Determine whether the data after the connection aggregation is invalid. If there is an invalid situation, send the data to the verification queue for manual verification; if there is no invalid situation, determine whether the data after the connection aggregation is complete. If the data is complete, proceed to the next step S4. If the data is incomplete, return to S3 and perform connection aggregation again.

3. The method for processing new energy vehicle operation data according to claim 1, characterized in that: Initializing the server memory data processing window and window queue parameters in S1 also includes: During initialization, the system creates a corresponding value queue with adjustable length for each window, and simultaneously starts a scheduling module to manage the scheduling relationship between the window and the value queue to ensure reasonable data distribution.

4. The method for processing new energy vehicle operation data according to claim 3, characterized in that: The scheduling module dynamically adjusts the numerical queue length of the current window within a certain range according to the queue length of each window and the consumption rate of the window.

5. The method for processing new energy vehicle operation data according to claim 3, characterized in that: The scheduling module can monitor the consumption status of the window queue. When the window queue is congested, it locks the vehicle data of the congested window queue, adjusts the consumption priority of the corresponding vehicle data, and adopts a diversion strategy to transfer the vehicle data that is not congested to other windows.

6. The method for processing new energy vehicle operation data according to claim 3, characterized in that: The step of collecting vehicle operation data according to the configured rolling strategy in S1 further includes: Establish grading and window marking for consumption of vehicle data; The scheduling module dynamically maintains the classification and window marking information according to actual conditions; The vehicle data is window mapped through a hash remainder strategy or a polling strategy to ensure that the data of the same vehicle can only be consumed by the same window and that the vehicle data distributed in each window is balanced.

7. The method for processing new energy vehicle operation data according to claim 1, characterized in that: The step S2 also includes cleaning the collected data before obtaining the preprocessing window queue: For outliers, the statistical characteristics of vehicle data are used to detect outliers and the detection results are corrected; For missing values, the deletion method, filling method, interpolation method and model prediction method are combined to correct the missing values; For outliers, the median absolute deviation is used as the basis, combined with sliding window linear interpolation and isolation forest analysis methods to eliminate discrete data.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the method for processing new energy vehicle operating data as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer executes the method for processing new energy vehicle operating data as described in any one of claims 1-7.

Images