Multi-sensor target data partitioning method, system and target data fusion method, system of spark distributed computing engine

Abstract

The multi-sensor target data partition method, system and target data fusion method and system of a spark distributed computing engine belong to the technical field of vehicle multi-sensor target data.The present application solves the problem of long running time and extremely low efficiency of the existing fusion mode.The application comprises the following steps: step S1, selecting the historical data of the vehicle speed greater than 0 detected by the multi-sensor, taking the historical data in the continuous time points as a partition, and merging the partitions with the difference between the termination time of each partition and the starting time of the next partition less than 1s; step S2, setting the partition end threshold i as 10, calculating the difference between the number of rows of each partition and the maximum partition row number, until the difference between the number of rows of each partition and the maximum partition row number satisfies less than i%, then ending the division of each partition; step S3, if the final partition end i value exceeds 30, the historical data needs to be re-partitioned to ensure that the difference between the data row number of each partition and the maximum partition row number is less than 30%.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle multi-sensor target data, in particular to a multi-sensor target data partitioning method and system of a Spark distributed computing engine and a target object data fusion method and system. BACKGROUND

[0002] With the rise of intelligent driving cars, the verification of the reliability and accuracy of the automatic driving function control software is becoming increasingly important, so it is necessary to build intelligent networked car intelligent driving scene libraries for various scenes such as highway scenes and urban expressway congestion scenes. In the construction process, the vehicle needs to be intelligently modified and equipped with intelligent cameras, laser radars, millimeter wave radars and other target object detection sensors, as well as other traditional sensors such as acceleration sensors, GPS positioning, and vehicle CAN bus collection. In the subsequent data processing and analysis process, multiple target object detection sensors are needed to perform target object level fusion and tracking on the same target object to make up for the deficiency of target object loss caused by a single sensor. However, the exponential explosive growth of vehicle data with ultra-high sampling frequency and ultra-long data collection mileage makes the amount of collected data reach T level. How to quickly and accurately fuse the target object data of multiple sensors from the collected historical data and output stable target object data becomes a major difficulty.

[0003] Currently, this problem can be solved in the following two ways: 1. Target object fusion at the vehicle end, according to the time sequence, sequentially obtaining target object data detected by multiple sensors such as laser radar, millimeter wave radar, and camera, performing pose fusion, and outputting the fused target object data. 2. Fusion of sensor raw data, such as camera image data, laser radar and millimeter wave radar point cloud data, which are superimposed to form superimposed fusion data with multiple sensor characteristics, and then target object detection is performed to output the fused target object data.

[0004] Both methods can achieve fusion of target object data, but there are defects in using this method:

[0005] 1) Both need to be output in real time during vehicle operation.

[0006] 2) When processing billions of rows of data at a time, the running time is long and the efficiency is extremely low.

[0007] Based on the above factors, the existing fusion method of target object data can only output in real time during vehicle operation, and when processing billions of rows of data at a time, the running time is long and the efficiency is extremely low.

[0008] Therefore, in the prior art, the fusion mode of target object data can only be output in real time during vehicle operation, and when processing hundreds of millions of rows or even larger amounts of data at a time, the running time is long and the efficiency is extremely low. For example, patent document CN111222568A discloses a "vehicle network data fusion method and device", which fuses sensor fusion data and time-space synchronous vehicle network data to obtain vehicle network fusion data, thereby realizing not only the fusion of the vehicle's own sensor data, but also the fusion of the sensor data and the network data, and accurately predicting the driving conditions of the vehicle and the vehicle and other objects around it. The technical solution described in the patent document can only fuse sensor data and network data to accurately predict the driving conditions of the vehicle and the vehicle and other objects around it, and does not provide a solution to the problem that the fusion mode of target object data can only be output in real time during vehicle operation, and when processing hundreds of millions of rows or even larger amounts of data at a time, the running time is long and the efficiency is extremely low. Patent document CN111913177A discloses a "target object detection method and device and storage medium", which uses sensors to fuse target object information to judge and identify the target object, thereby improving the reliability of target object detection. The patent document improves the reliability of target object detection, but the analysis results cannot provide a solution to the problem that the fusion mode of target object data can only be output in real time during vehicle operation, and when processing hundreds of millions of rows or even larger amounts of data at a time, the running time is long and the efficiency is extremely low. SUMMARY

[0009] The present application solves the problem that the existing fusion mode of target object data can only be output in real time during vehicle operation, and when processing hundreds of millions of rows or even larger amounts of data at a time, the running time is long and the efficiency is extremely low.

[0010] The multi-sensor target object data partitioning method of the Spark distributed computing engine comprises the following steps:

[0011] Step S1, selecting historical data of the vehicle speed greater than 0 detected by the multi-sensor, taking the historical data at consecutive time points as a partition, and merging the partitions with a difference between the end time of each partition and the start time of the next partition less than 1s;

[0012] Step S2, setting the partition end threshold i as 10, calculating the difference between the number of rows of each partition and the maximum partition row number, until the difference between the number of rows of each partition and the maximum partition row number satisfies i%, then ending the partitioning of each partition;

[0013] Step S3, if the final value i of the partitioning end is greater than 30, the historical data needs to be cleaned and filtered, and then partitioned again. ’

[0014] ​Further, in one embodiment of the present application, in the step S2, the difference between the number of rows of each partition and the maximum number of partition rows is calculated as:

[0015] If greater than i%, then the number of surrounding vehicles in each partition is divided to obtain new partitions under the condition that the number of surrounding vehicles in each partition is less than 1;

[0016] If less than i%, i value does not exceed 30, the partition ends.

[0017] Further, in one embodiment of the present application, after the number of surrounding vehicles in each partition is divided to obtain new partitions, the difference between the number of rows of each partition and the maximum number of rows is calculated as:

[0018] If still greater than i%, then the number of surrounding vehicles is divided to obtain new partitions under the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s;

[0019] If less than i%, i value does not exceed 30, the partition ends.

[0020] Further, in one embodiment of the present application, after the number of surrounding vehicles is divided to obtain new partitions under the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, the difference between the number of rows of each partition and the maximum number of partition rows is calculated as:

[0021] If still greater than i%, then the i value is updated to 10*cycle times, the difference between the number of rows of each partition and the maximum number of partition rows is calculated, if still greater than i%, the same method is used to divide to obtain new partitions, until the difference between the number of rows of each partition and the maximum number of partition rows meets less than i%;

[0022] If less than i%, i value does not exceed 30, the partition ends.

[0023] The multi-sensor target data extraction and fusion method of the Spark distributed computing engine disclosed in the present application comprises the following steps:

[0024] Select multi-sensor target historical data as data to be processed;

[0025] Step S4, performing target pose matching on the data to be processed, randomly selecting a channel obtained by target pose matching as a reference channel set S={reference id, matching id}, traversing the remaining channel set dn={reference id, matching id}, when the number of S∩dn is greater than 1, updating the S set as S=S∪dn;

[0026] Step S5, in all target channel, remove the dn channel meeting step S4, randomly select the obtained channel again, until all channels are used as reference channels, and the target id fusion is completed;

[0027] Step S6, each channel of the extracted is renumbered as the fusion id of the target object;

[0028] Step S7, extracting all original sensor corresponding time point detection data values contained in the fusion id of the target object;

[0029] Step S8, randomly selecting the extracted fusion id of the historical data to perform the target object historical data fusion.

[0030] Further, in an embodiment of the present application, in step S4, first, the selected multi-sensor target object historical data is partitioned to form the data to be processed by using the multi-sensor target object data partition method of the Spark distributed computing engine of claim 1, and then the target object historical data fusion processing of steps S4 to S8 is performed for the data in each partition.

[0031] The multi-sensor target data partition system of the Spark distributed computing engine of the present application, the system comprises the following modules:

[0032] Module S1, for selecting the historical data of the vehicle speed greater than 0 detected by the multi-sensor, and merging the historical data in the continuous time points as a partition, and the difference between the end time of each partition and the start time of the next partition is less than 1s;

[0033] Module S2, for setting the partition end threshold i as 10, calculating the difference between the number of rows of each partition and the maximum partition row number, until the difference between the number of rows of each partition and the maximum partition row number satisfies less than i%, then ending the cutting of each partition;

[0034] Module S3, for the final partition end final value i ’ more than 30, the historical data needs to be cleaned and filtered, and then partitioned again.

[0035] The multi-sensor target object data extraction fusion system of the Spark distributed computing engine of the present application, the system comprises the following modules:

[0036] Data selection module, for selecting multi-sensor target object historical data as data to be processed;

[0037] Module S4, performing target object pose matching on the data to be processed, randomly selecting the channel obtained by the target object pose matching as the reference channel set S={reference id, matching id}, traversing the remaining channel set dn={reference id, matching id}, when the number of S∩dn is greater than 1, then updating the S set as S=S∪dn;

[0038] Module S5, for removing the dn channel satisfying step S4 in all target channels, and randomly selecting the obtained channel again until all channels are used as reference channels, and the target id fusion is completed;

[0039] Module S6, for renumbering each channel extracted as the target fusion id;

[0040] Module S7, for extracting all original sensor corresponding time point detection data values contained in the target fusion id;

[0041] Module S8, for randomly selecting the extracted fusion id of historical data to perform target historical data fusion.

[0042] The computer readable storage medium provided by the application has a computer program stored thereon, and the computer program is executed by a processor to realize the steps of any one of the above methods.

[0043] The computer device provided by the application comprises a memory and a processor, and the memory stores a computer program.

[0044] The application solves the problem that the existing target data fusion method can only output in real time during vehicle operation, and the running time is long and the efficiency is extremely low when processing a large amount of data.

[0045] Specific beneficial effects include:

[0046] 1. The multi-sensor target data partitioning method of the Spark distributed computing engine provided by the application is to adopt the historical data of the target data detected by different sensors of the vehicle, and the target data is not output in real time during vehicle operation.

[0047] 2. The multi-sensor target data partitioning method of the Spark distributed computing engine provided by the application effectively divides the large amount of target historical data detected by multiple sensors, realizes data fusion and expansion of different sensors, maximizes the use of cluster resources for parallel computing, and the partitioning calculation method is based on actual data conditions for partitioning.

[0048] 3. The multi-sensor target data extraction and fusion method of the Spark distributed computing engine has the effects that the fusion id of the target is selected as an index through the benchmark channel fusion method, an index channel is established, and the target historical data is quickly and accurately extracted and fused.

[0049] 4. The combination of the multi-sensor target data partition method of the Spark distributed computing engine and the target data fusion method effectively avoids the time-consuming problem caused by fusion of a large amount of data, and can effectively shorten the running time of the fusion method of the prior art from more than 5 days to minutes.

[0050] The application is suitable for the intelligent networked vehicle testing technical field, and solves the technical problem that the vehicle data grows exponentially under the conditions of super-high sampling frequency and super-long data collection mileage, and the collected data reaches T level, and the multi-sensor data is quickly and accurately fused from the collected historical data, and stable target data is output. BRIEF DESCRIPTION OF DRAWINGS

[0051] The above and / or additional aspects and advantages of the application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0052] Figure 1 is a flow chart of the multi-sensor target data fusion method and system of the Spark distributed computing engine.

[0053] Figure 2 is a partition flow chart according to the first embodiment.

[0054] Figure 3 is a target id fusion diagram according to the fifth embodiment.

[0055] Figure 4 is a target id fusion flow chart according to the fifth embodiment.

[0056] Figure 5 is a high-gaussian kernel function filtering diagram according to the fifth embodiment; (a) initial scanning value diagram; (b) overlapping part is two different sensors diagram; (c) overlapping part is the same sensor diagram.

[0057] Figure 6 is a target channel diagram according to the sixth embodiment.

[0058] Figure 7 is a pose matching flow chart according to the sixth embodiment. DETAILED DESCRIPTION

[0059] The various embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. The embodiments described by reference to the accompanying drawings are exemplary and are intended to explain the present application, but cannot be understood as limiting the present application.

[0060] In one embodiment, the method for partitioning multi-sensor target object data of a Spark distributed computing engine comprises the following steps:

[0061] Step S1: Selecting historical data of a vehicle speed greater than 0 detected by multi-sensor detection, and taking historical data at consecutive time points as a partition, and merging partitions with a difference between the end time of each partition and the start time of the next partition less than 1s.

[0062] Step S2: Setting the partition end threshold i as 10, calculating the difference between the number of rows of each partition and the maximum number of rows of the partition, until the difference between the number of rows of each partition and the maximum number of rows of the partition satisfies i%, and then ending the partitioning of each partition.

[0063] Step S3: If the final value i of the partitioning end is greater than 30, the historical data needs to be cleaned and filtered, and then partitioned again. ’ ’ ’

[0064] In this embodiment, historical target object data detected by multi-sensor detection of a vehicle is obtained, and a large amount of historical target object data detected by different sensors is analyzed, and data fusion and expansion of different sensors are realized through target object data pose, speed, distance, etc. Target object data detected by different sensors includes vehicle speed, time column, target object relative distance, target object relative speed, target object relative distance, target object type, target object heading angle, target object length-width-height.

[0065] Specifically, the target object data detected by different sensors is built into a data partition sequence number column, so that the collected data is partitioned according to this column, and the number of data rows in each partition is approximately the same, and the partitioning rule is as shown in Figure 2 The user can set the partition end threshold and the final value i of i ’ , if the final value i ’ exceeds the set maximum value, the data needs to be cleaned and filtered, and then partitioned again, so that the difference between the number of rows of each partition and the maximum number of rows of the partition is less than the set maximum value.

[0066] The partitioning and parallel computing method described in this embodiment can effectively improve the efficiency of the computing speed and the accuracy of the computing.

[0067] Embodiment two, the embodiment is further limited to the multi-sensor target data partition method of the Spark distributed computing engine described in embodiment one, in the embodiment, in step S2, the difference between the number of rows of each partition and the maximum partition row number is:

[0068] If greater than i%, then according to the condition that the number of surrounding vehicles in each partition is less than 1, the number of surrounding vehicles in each partition is divided to obtain new partitions;

[0069] If less than i%, i ’ Not more than 30, the partition ends.

[0070] In the embodiment, according to the condition that the number of surrounding vehicles in each partition is less than 1, the first partition is performed, that is, in each speed interval that is not 0, further division is performed, because it is required to ensure that each partition should contain all time data of the stable appearance of the target, and to avoid the data of the same target being divided into different partitions, causing errors in the final result.

[0071] Embodiment three, the embodiment is further limited to the multi-sensor target data partition method of the Spark distributed computing engine described in embodiment two, in the embodiment, after the number of surrounding vehicles in each partition is divided to obtain new partitions, the difference between the number of rows of each partition and the maximum partition row number is:

[0072] If still greater than i%, then according to the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, the division is performed to obtain new partitions;

[0073] If less than i%, i ’ Not more than 30, the partition ends.

[0074] In the embodiment, the division is performed according to the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, in order to ensure that the boundary of each partition is in a relatively stable state with the target and the vehicle, and to divide the interaction process between the target and the vehicle into a partition as much as possible, and to ensure that the fused id number is stable when the target and the vehicle interact.

[0075] Embodiment four, the embodiment is further limited to the multi-sensor target data partition method of the Spark distributed computing engine described in embodiment three, in the embodiment, after the number of surrounding vehicles in each partition is divided to obtain new partitions according to the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, the difference between the number of rows of each partition and the maximum partition row number is:

[0076] If still greater than i%, update i value to 10*cycle number, calculate the difference between the number of rows of each partition and the maximum partition row number, if still greater than i%, use the same method to obtain new partitions, until the difference between the number of rows of each partition and the maximum partition row number meets less than i%, and i

[0077] If less than i%, i ’ Not more than 30, partition ends.

[0078] In this embodiment, the same method is used to obtain new partitions by using the method described in embodiment two and embodiment three, until the difference between the number of rows of each partition and the maximum partition row number meets less than i%, and i ’ Not more than 30, partition ends.

[0079] Embodiment five, a multi-sensor target data extraction and fusion method of a spark distributed computing engine, comprising the following steps:

[0080] Selecting multi-sensor target historical data as data to be processed;

[0081] Step S4, performing target pose matching on the data to be processed, randomly selecting a channel obtained by target pose matching as a reference channel set S={reference id, matching id}, traversing the remaining channel set d n ={reference id, matching id}, when the number of S∩d n is greater than 1, updating the S set to S=S∪d n ;

[0082] Step S5, removing the d n channel that meets step S4 in all target channels, and randomly selecting the obtained channel again until all channels are used as reference channels, and the target id fusion is completed;

[0083] Step S6, renumbering each channel extracted as a fusion id of the target;

[0084] Step S7, extracting all original sensor corresponding time point detection data values contained in the fusion id of the target;

[0085] Step S8, randomly selecting the fusion id of the historical data extracted to perform target historical data fusion.

[0086] In this embodiment, the process of performing target id fusion in each partition is as shown in Figure 4 As shown in Figure 3 , renumbering each channel extracted as a fusion id of the target, and from Figure 3It can be seen that each fusion id contains the id number corresponding to the same target object detected by other sensors, for example: the target object data with fusion id number 1 contains the data corresponding to id number 0 of sensor 1, the data corresponding to id number 14 of sensor 2 and the data of id number 20 of sensor n. Thus, the data of different ids detected by different sensors for a target object are fused to generate new fusion id numbers.

[0087] The target object historical data fusion is performed under the following conditions:

[0088] When only one sensor historical data is matched at the time point, the historical data is retained as the final fusion data;

[0089] When two different sensor historical data are matched at the time point, the historical data returned by the sensor with higher confidence is retained;

[0090] When two same sensor historical data are matched at the time point, the target object id historical data appearing first is retained.

[0091] The reference channel fusion method of the embodiment selects the fusion id of the target object as the index to establish an index channel, thereby achieving the effect of quickly and accurately extracting and fusing the target object historical data.

[0092] Each partition target object fusion id contains all the historical data values detected by the corresponding time points of all original sensors, including the relative position information, relative speed information, heading angle, target object confidence, target object type, sensor type, etc.

[0093] In each partition, a high Gaussian kernel function filter is performed on the continuous variables such as target object relative distance and relative speed to smooth the target object historical data, and the specific steps are as follows:

[0094] 1. Select the final historical data of any fusion id;

[0095] 2. Construct a Gaussian kernel function with standard deviation σ and mean μ, and truncate the Gaussian template with a length of μ±nσ standard deviation outside the mean value, which is:

[0096]

[0097] 3. Scan and filter each continuous variable using the Gaussian template to complete the final data fusion, and the filtering process is as follows:

[0098] 1) According to the time sequence, the template is scanned from the first point of the fusion id, when the template does not completely coincide with the data, the non-coincidence part of the data is filled by boundary extension, the value corresponding to the position where the template value coincides with the data is accumulated and multiplied, and the data filtering value of the point corresponding to the median value in the Gaussian template is obtained, as shown in (a). Figure 5

[0099] 2) When the template completely coincides with the data, and the coincident part is the data returned by different sensors, the template mean value is changed to the data value of the sensor with high confidence, and the template scanning is shown in (b). Figure 5

[0100] 3) When the coincident part is the data returned by the same sensor, the mean value is changed to 0, and the template scanning is shown in (c). Figure 5

[0101] 4) Taking (c) as an example, the original data corresponding to the updated Gaussian template mean value is: Figure 5

[0102] X5=G1xX3+G2xX4+G3xX5+G2xX6+G1xX7.

[0103] Embodiment six, the multi-sensor target data fusion method of the Spark distributed computing engine according to embodiment five, in step S4, first, the selected multi-sensor target historical data is partitioned by the multi-sensor target data partition method of the Spark distributed computing engine in embodiment one to form the data to be processed, then the target historical data fusion processing of steps S4 to S8 is performed for the data in each partition.

[0104] In this embodiment, the way of processing multiple partition data is parallel or serial processing, which can effectively improve the calculation speed of a large amount of historical data.

[0105] The specific target pose matching process is shown in (a), which includes the following steps: Figure 7

[0106] Step S601, randomly selecting a target id of a sensor in the partition as a fusion reference id;

[0107] Step S602, selecting all target ids detected by other target sensors in the time period of the id as target ids to be fused;

[0108] ​​​​​Step S603, according to the characteristics of different sensors, according to the specified conditions to determine the target id to be fused and the reference id as the matching target channel, continue to randomly match the target in the partition of one sensor, until all target ids of all sensors in the partition are matched as the reference id, and record all matched target channels.

[0109] The specific matching process can be better understood. The recorded results of all matched target channels Figure 7 As shown in the figure, Figure 6 It can be seen that in the channel with reference id 0, the matched ids of the same target of other sensors are as follows: after pose matching, the target with id number 15 in sensor 1, id number 10 in sensor 2, and id number 6 in sensor n are considered as the same target. Since the sensors selected for the reference id are different, the reference id values may be the same.

[0110] Among them, the Euclidean distance formula is:

[0111]

[0112] In the formula, p is the Euclidean distance between point (x1, x2) and point (y1, y2). The Euclidean distance commonly used distance definition refers to the real distance between two points in space.

[0113] The embodiment can effectively avoid the time-consuming problem caused by fusion of a large amount of data, and can effectively shorten the running time of the fusion method of the prior art from more than 5 days to minutes.

[0114] Embodiment seven, a multi-sensor target data partition system of a Spark distributed computing engine, the system comprises the following modules:

[0115] Module S1, for selecting multi-sensor detection historical data with vehicle speed greater than 0, and merging historical data in continuous time points as a partition, and the difference between the end time of each partition and the start time of the next partition is less than 1s;

[0116] Module S2, for setting the partition end threshold i as 10, calculating the difference between the number of rows of each partition and the maximum partition row number, until the difference between the number of rows of each partition and the maximum partition row number satisfies less than i%, then end the partitioning of each partition;

[0117] Module S3, for setting the final partition end value i ’ If it is greater than 30, the historical data needs to be cleaned and filtered, and then partitioned again.

[0118] Embodiment eight, a multi-sensor target object data extraction and fusion system of a Spark distributed computing engine, the system comprising the following modules:

[0119] a data selection module for selecting multi-sensor target object historical data as data to be processed;

[0120] module S4, performing target object pose matching on the data to be processed, randomly selecting a channel obtained by target object pose matching as a reference channel set S = {reference id, matching id}, and traversing the remaining channel set d n n When the number of S∩d n is greater than 1, updating the S set as S = S∪d n ;

[0121] module S5, for removing the d n channel satisfying step S4 in all target object channels, and randomly selecting the obtained channel again until all channels are reference channels, completing target object id fusion;

[0122] module S6, for renumbering each channel extracted as a target object fusion id;

[0123] module S7, for extracting all original sensor corresponding time point detection data values contained in the target object fusion id;

[0124] module S8, for randomly selecting a fusion id of historical data extracted to perform target object historical data fusion.

[0125] Embodiment nine, a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method of any one of embodiments one to six.

[0126] Embodiment ten, a computer device comprising a memory and a processor, the memory storing a computer program, and the processor executing the steps of the method of any one of embodiments one to six when running the computer program stored in the memory.

Claims

1. A method for multi-sensor target data partitioning for a Spark distributed computing engine, the method comprising: The method comprises the following steps: S1, selecting historical data of vehicle speed greater than 0 detected by a plurality of sensors, merging historical data at continuous time points as a partition, and merging each partition with a difference between the end time of the partition and the start time of the next partition less than 1s; Step S2, set the partition end threshold i to 10, the final value i ’ is 30, calculate the difference between the number of rows of each partition and the maximum partition row number, if greater than i%, then according to the condition that the number of surrounding vehicles in each partition is less than 1, the number of surrounding vehicles in each partition is further divided to obtain new partitions, if still greater than i%, then according to the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, the new partitions are obtained by cutting, if the conditions cannot be met, then update the i value to 10*cycle number, if the conditions are met, then recalculate the difference between the number of rows of each partition and the maximum partition row number, if still greater than i%, then re-execute step S2 until the difference between the number of rows of each partition and the maximum partition row number is less than i%, and the final value i ’ does not exceed 30, end the cutting of each partition; Step S3, if the cutting end value i ’ If it is more than 30, the historical data needs to be cleaned, filtered out, and partitioned again.

2. A multi-sensor target data partitioning system for a Spark distributed computing engine, characterized in that, The system comprises the following modules: S1, selecting historical data of vehicle speed greater than 0 detected by a plurality of sensors, merging historical data at continuous time points as a partition, and merging each partition with a difference between the end time of the partition and the start time of the next partition less than 1s; Module S2, set the partition end threshold i to 10, the final value i ’ is 30, calculate the difference between the number of rows of each partition and the maximum partition row number, if greater than i%, then according to the condition that the number of surrounding vehicles in each partition is less than 1, the number of surrounding vehicles in each partition is further divided to obtain new partitions, if still greater than i%, then according to the condition that the number of surrounding vehicles is greater than 1 and the relative distance and relative speed are stable within 2s, the new partitions are obtained by cutting, if the conditions cannot be met, update the value of i to 10*cycle number, if the conditions are met, recalculate the difference between the number of rows of each partition and the maximum partition row number, if still greater than i%, then re-execute step S2 until the difference between the number of rows of each partition and the maximum partition row number is less than i%, and the final value i ’ does not exceed 30, end the cutting of each partition; Module S3, for final partition end final value i ’ If more than 30, the historical data needs to be cleaned, filtered out, and partitioned again.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in claim 1.

4. A computer device comprising a memory and a processor, said memory having stored therein a computer program, characterized in that, The processor executes the steps of the method in claim 1 when running the computer program stored in the memory.

Images