A data processing method for multiple data sources

Abstract

This invention relates to a data processing method for multiple data sources. The method includes: acquiring first vehicle coordinates, a first road scene, and multiple data sources at any given time; performing position coordinate transformation on the multiple data sources based on the first vehicle coordinates and the first road scene; dividing the time period from the starting time point to the current time into a first time axis, using the earliest data time among the multiple data sources as the starting time point and according to the data frequency f; integrating the data from the multiple data sources based on the time scale of the first time axis; correcting the speed type data in the first integrated data sequence; identifying the masked data sequences of each type of data source; and outputting the first integrated data sequence and the multiple masked data sequences. This invention provides a solution for overall preprocessing of multiple data sources.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method for multiple data sources. Background Technology

[0002] The perception module of an autonomous driving system needs to preprocess data from multiple data sources and then feed the results into relevant prediction models for analysis. These data sources include perception data from LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data from filters (such as Kalman filters). Common data preprocessing methods involve integrating data from multiple data sources at discrete time points to obtain a corresponding time data sequence. In principle, the higher the integration level of the preprocessing, the richer the amount of data information received by the prediction model, resulting in higher prediction accuracy. Currently, common data preprocessing methods for perception modules include: preprocessing multiple perception data sources individually before feeding them into the prediction model; preprocessing multiple target detection data sources individually before feeding them into the prediction model; or preprocessing multiple state prediction data sources individually before feeding them into the prediction model. However, it is not possible to perform overall preprocessing of all perception data sources, target detection data sources, and state prediction data sources. This results in insufficient data variety for each prediction model, hindering further improvement in prediction accuracy. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a data processing method, electronic device, and computer-readable storage medium for multiple data sources. This method integrates multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters) into a single temporal space. To improve the accuracy of the integrated data, a mechanism for correcting the velocity information of the integrated data is provided. Furthermore, to enhance the model adaptability of the integrated data, a corresponding mask data sequence with temporal characteristics is generated for each data source. This invention provides a solution for the overall preprocessing of multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters), thereby helping to further improve the prediction accuracy of various prediction models in the sensing module.

[0004] To achieve the above objectives, a first aspect of the present invention provides a data processing method for multiple data sources, the method comprising:

[0005] At any given time, obtain the current vehicle coordinates and the current road scene as the corresponding first vehicle coordinates and first road scene; and obtain multiple data sources within the most recent first time period.

[0006] The location coordinate transformation process is performed on the multiple data sources based on the first vehicle coordinates and the first road scene;

[0007] If the location coordinate transformation is successful, the earliest data time in the multiple data sources is taken as the starting time point and the time period from the starting time point to the current time is divided into time axis segments according to the preset data frequency f to obtain the corresponding first time axis.

[0008] Based on the time scale of the first time axis, the data from the multiple data sources are integrated to generate a corresponding first integrated data sequence; and the speed type data in the first integrated data sequence is corrected.

[0009] Based on the first integrated data sequence, the mask data sequences of various data sources are identified to obtain corresponding multi-class mask data sequences;

[0010] The first integrated data sequence and the multi-type mask data sequence, after data correction, are output as the corresponding data processing results.

[0011] Preferably, the first road scenario includes a highway scenario, a street road scenario, a rural road scenario, and an urban road scenario;

[0012] The multiple data sources include multiple first data sources; each first data source includes a first source type and multiple first data objects; the first source type includes LiDAR, millimeter-wave radar, camera, target detection model, and filter; each first data object includes a first time and a first data group; each first data group includes one or more first data; the data types of the first data include position coordinates, orientation angle, target detection box size, and speed; the coordinate system of the first data with position coordinates and the coordinate system of the first vehicle coordinates are both preset unified coordinate systems, which are either geodetic coordinate systems or BEV coordinate systems.

[0013] The first time axis includes multiple first time scales t i 0 ≤ scale index i ≤ maximum scale index M, maximum scale index M ≥ 0; the first time scale t i The starting time point is the last first time scale t. i The end time point;

[0014] The first integrated data sequence includes multiple second data objects D i The second data object D i Including the first time scale t i The second source type includes LiDAR, millimeter-wave radar, camera, target detection model, and filter types; the second data group includes multiple second data sets. i,j , 1 ≤ data index j ≤ maximum data index N, maximum data index N ≥ 1; the second data d i,j The data types include position coordinates, orientation angle, target detection box size, and velocity; the second data d with the data type being position coordinates. i,j The coordinate system of the first vehicle coordinate system and the coordinate system of the first vehicle coordinate system are both the preset unified coordinate system;

[0015] The multiple types of mask data sequences include multiple first mask data sequences; the first mask data sequences include multiple first mask data Masks. i The first mask data Mask i The mask values ​​include 0 and 1.

[0016] Preferably, the step of performing position coordinate transformation processing on the multiple data sources based on the first vehicle coordinates and the first road scene specifically includes:

[0017] Each of the first data points in the multiple data sources whose data type is location coordinate is marked as a corresponding first target coordinate; the straight-line distance between each first target coordinate and the first vehicle coordinate is calculated to generate a corresponding first target distance; and the three first target distances with the shortest distance are selected as the corresponding first, second and third nearest coordinates.

[0018] Based on the first, second, and third nearest coordinates and the first road scene, the location transformation evaluation index is estimated to generate the corresponding first evaluation index P;

[0019] The system identifies whether the first evaluation index P is less than a preset first index threshold; if so, it sets the corresponding first conversion mode to the first mode; if not, it sets the corresponding first conversion mode to the second mode.

[0020] Based on the first conversion mode, the first road scene, and the first evaluation index P, the first location coordinates are converted to generate corresponding second location coordinates.

[0021] Based on each of the second position coordinates, the first data in the multiple data sources whose corresponding data type is position coordinate is reset.

[0022] Furthermore, the step of estimating the position transformation evaluation index and generating the corresponding first evaluation index P based on the first, second, and third nearest coordinates and the first road scene specifically includes:

[0023] The first vehicle coordinate is marked as the corresponding first vehicle coordinate (x0, y0, z0); and the first, second, and third nearest coordinates are marked as the corresponding first nearest coordinate (x1, y1, z1), second nearest coordinate (x2, y2, z2), and third nearest coordinate (x3, y3, z3).

[0024] The first road scene is identified; if the first road scene is a highway scene, the corresponding first weight sequence {w1, w2, w3} is set as a preset first weight constant sequence {w 1,1 ,w 1,2 ,w 1,3 If the first road scenario is a street scene, then the corresponding first weight sequence {w1, w2, w3} is set to a preset second weight constant sequence {w...}. 2,1 ,w 2,2 ,w 2,3 If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight sequence {w1, w2, w3} is set to a preset third weight constant sequence {w 3,1 ,w 3,2 ,w 3,3};

[0025] The first evaluation index P is calculated based on the first vehicle coordinates (x0, y0, z0), the first nearest coordinates (x1, y1, z1), the second nearest coordinates (x2, y2, z2), the third nearest coordinates (x3, y3, z3), and the first weight sequence {w1, w2, w3}.

[0026]

[0027]

[0028]

[0029]

[0030] Furthermore, the step of performing position coordinate transformation processing on each of the first position coordinates according to the first conversion mode, the first road scene, and the first evaluation index P to generate corresponding second position coordinates specifically includes:

[0031] The first coordinate vector S1 is obtained by performing coordinate vector transformation on the first position coordinates;

[0032] When the first transformation mode is the first mode, the rotation matrix and offset matrix from the unified coordinate system to the vehicle coordinate system are confirmed to obtain the corresponding first rotation matrix A and first offset matrix T; the first coordinate vector S1 is transformed according to the first rotation matrix A and the first offset matrix T to obtain the corresponding second coordinate vector S2, S2 = A·S1 + T;

[0033] When the first conversion mode is the second mode, the first road scene is identified; if the first road scene is a highway scene, the corresponding first weight parameter k is set to a preset first weight constant. If the first road scenario is a street scene, then the corresponding first weight parameter k is set to a preset second weight constant. If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight parameter k is set to a preset third weight constant. The first coordinate vector S1 is then transformed according to the first weight parameter k and the first evaluation index P to obtain the corresponding second coordinate vector S2.

[0034] Extract each coordinate component of the second coordinate vector S2 to form the corresponding second position coordinates.

[0035] Preferably, the step of using the earliest data time among the multiple data sources as the starting time point and dividing the time period from the starting time point to the current time according to a preset data frequency f to obtain the corresponding first time axis specifically includes:

[0036] The earliest first time among the multiple data sources is taken as the corresponding start time point; the time period from the start time point to the current time is taken as the corresponding first data time period; the corresponding first time interval Δt = 1 / f is calculated according to the data frequency f; the first data time period is divided into multiple equal time scales according to the first time interval Δt, the equal time scales including the start time point; the number of the obtained equal time scales is counted to obtain the corresponding first quantity; the value of the first quantity minus 1 is taken as the corresponding maximum scale index M; and the first equal time scale is taken as the corresponding first time scale t. i＝0 The second equally divided time scale is taken as the corresponding first time scale t. i＝1 This process continues until the last equally divided time scale is taken as the corresponding first time scale t. i＝MUp to; and by all the first time scales t obtained. i This forms the corresponding first time axis.

[0037] Furthermore, the step of integrating and processing the data from the multiple data sources based on the time scale of the first time axis to generate a corresponding first integrated data sequence specifically includes:

[0038] The total number of all data types in the multiple data sources is counted to obtain the corresponding total number of types, and the maximum data index N is set according to the total number of types.

[0039] Initialize the first integrated data sequence according to the maximum scale index M and the maximum data index N; and in the initialized first integrated data sequence, process all the second data objects D. i This is denoted as the corresponding first invalid data object; wherein, the first integrated data sequence obtained after initialization includes the second data object D of the maximum scale index M. i ; each of the second data objects D obtained during initialization i Compared with the first time scale t i One-to-one correspondence; the second data object D obtained after initialization i Including a corresponding first time scale t i A second source type initialized to empty and a second data group with a fixed data length of N-2; the initialized second data group includes N-2 initialized empty second data d. i,j ;

[0040] All the first data objects from the multiple data sources are sorted according to the chronological order of the first time to obtain the corresponding sequence of first data objects;

[0041] The first data objects in the first data object sequence are traversed sequentially. During traversal, the currently traversed first data object is taken as the corresponding current first data object, the first source type of the first data source corresponding to the current first data object is taken as the corresponding current type, the first time of the current first data object is taken as the corresponding current data time, and the first time scale t on the first time axis corresponding to the current data time is taken as the current data time. i As the corresponding current time scale, and using the second data object D in the first integrated data sequence that corresponds to the current time scale, iAs the corresponding current second data object; and to identify whether the current second data object is recorded as the first invalid data object; if the current second data object is recorded as the first invalid data object, then the second source type of the current second data object is set to the current type, and the second data d corresponding to the second data group of the current second data object is set according to one or more of the first data of the current first data object. i,j The settings are configured, and upon successful configuration, the current second data object is reclassified as the corresponding first valid data object. If the current second data object is not reclassified as the first invalid data object, the second source type of the current second data object is used as the corresponding first type. A type optimization process is performed on the first type and the current type according to a preset type optimization mode to obtain the corresponding first optimization result. When the first optimization result is the current type, the second source type and the second data group of the current second data object are cleared, and the second source type of the current second data object is set to the current type. The second data d in the second data group of the current second data object is then configured according to one or more of the first data from the current first data object. i,j Configure settings;

[0042] When the traversal is complete, the first integrated data sequence with the data filling completed will be output as the corresponding data integration processing result.

[0043] Furthermore, the step of performing type optimization processing on the first type and the current type according to a preset type optimization mode to obtain the corresponding first optimization result specifically includes:

[0044] The preferred type mode is identified; the preferred type mode includes a first preferred mode, a second preferred mode, and a third preferred mode.

[0045] When the type preference mode is the first preference mode, the current weather type is obtained as the corresponding first weather type; according to the first weather type, a preset data source level list is queried, and the first type priority sequence field of the first data source level record that matches the first weather type in the data source level list is extracted as the corresponding first type priority sequence; the sequence index of the third source type that matches the first type in the first type priority sequence is recorded as the corresponding first index, and the sequence index of the third source type that matches the current type is recorded as the corresponding current index; when the current index is less than the first index, the corresponding first preference result is set as the current type; when the current index is greater than the first index, the corresponding first preference result is set as the first type; the data source level list includes multiple first data source level records; the first data source level record includes the first weather type field and the first type priority sequence field; the first type priority sequence is composed of multiple third source types sorted; the third source types include lidar type, millimeter-wave radar type, camera type, target detection model type, and filter type;

[0046] When the type preference mode is the second preference mode, for the second data object D in the first data object sequence whose second source type is the first type and which is recorded as the first valid data object, i The quantity is counted to obtain the corresponding second quantity; and the second data object D in the first data object sequence whose second source type is the current type and is recorded as the first valid data object is... i The quantities are counted to obtain a corresponding third quantity; when the third quantity is greater than the second quantity, the corresponding first preferred result is set as the current type; and when the third quantity is less than or equal to the second quantity, the corresponding first preferred result is set as the first type.

[0047] When the type preference mode is the third preference mode, the second data object D, which is closest to the current second data object in the first data object sequence and is recorded as the first valid data object by a specified number H, is selected. i Form a corresponding second data object set; and for the second data object D in the second data object set whose second source type is the first type. i The quantity is statistically analyzed to obtain the corresponding fourth quantity; and the second data object D in the second data object set whose second source type is the current type is... iThe number of items is counted to obtain the corresponding fifth number; when the fifth number is greater than the fourth number, the corresponding first preferred result is set as the current type; and when the fifth number is less than or equal to the fourth number, the corresponding first preferred result is set as the first type.

[0048] Preferably, the data correction processing for the velocity type data in the first integrated data sequence specifically includes:

[0049] In the first integrated data sequence, the second data d, whose data type is position coordinates, is... i,j The second data object D corresponding to the second data group that is not empty i Let V be the corresponding valid point object. r The index of the valid point object is r≥1; and it is composed of all the valid point objects V. r Sorting generates a corresponding sequence of valid point objects;

[0050] The preset speed correction mode is identified; the speed correction mode includes a first correction mode and a second correction mode.

[0051] When the velocity correction mode is the first correction mode, a linear correction method is used to correct each effective point object V in the effective point object sequence. r Perform speed correction;

[0052] When the velocity correction mode is the second correction mode, a filtering correction method is used to correct each effective point object V in the effective point object sequence. r Perform speed correction.

[0053] Furthermore, the linear correction method is used to apply linear correction to each of the valid point objects V in the valid point object sequence. r Speed ​​correction includes:

[0054] The first valid point object V r As the corresponding current valid point object; and the three nearest valid point objects V following the current valid point object in the valid point object sequence. r Let these be the corresponding last one, last two, and last three valid point objects; and based on the last one, last two, and last three valid point objects, apply the second data d, whose data type is speed, to the current valid point object. i,j Perform data correction processing;

[0055] The second valid point object V r As the corresponding current valid point object; and the nearest valid point object V in the sequence of valid point objects preceding the current valid point object; rThe previous valid point object is denoted as V; and the two nearest valid point objects V following the current valid point object in the sequence of valid point objects are denoted as V. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous valid point object and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing;

[0056] The third valid point object V r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the sequence of valid point objects. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous one and previous two valid point objects and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing;

[0057] The fourth valid point object V r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the sequence of valid point objects. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous one and previous two valid point objects and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing;

[0058] And so on, until the second-to-last valid point object V is... r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the most recent valid point object V in the sequence of valid point objects after the current valid point object. r This is denoted as the corresponding next valid point object; and based on the previous and second previous valid point objects and the next valid point object, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing;

[0059] The last valid point object V r As the corresponding current valid point object; and the three nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let these be the corresponding first, second, and third valid point objects; and based on the first, second, and third valid point objects, assign the second data d, whose data type is speed, to the current valid point object. i,j Perform data correction processing.

[0060] More preferably, the second data d with data type speed in the current valid point object is obtained by applying the data to the last three valid point objects. i,j Data correction processing includes:

[0061] The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, which is of data type position coordinate, from the last one, the last two, and the last three valid point objects. i,j Extract them as the corresponding next coordinate vector p1, next second coordinate vector p2, and next third coordinate vector p3;

[0062] The time scale differences from the current valid point object to the next valid point object, from the next valid point object to the second valid point object, and from the second valid point object to the third valid point object are calculated to obtain the corresponding subsequent time difference Δt1, subsequent second time difference Δt2, and subsequent third time difference Δt3; and the longest time difference Δt is calculated from the time scale difference from the current valid point object to the third valid point object. max ;

[0063] For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0064] If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0065] If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, the current effective point velocity vector v is estimated based on the current coordinate vector p0, the next coordinate vector p1, the next second coordinate vector p2, the next third coordinate vector p3, the next time difference Δt1, the next second time difference Δt2, and the next third time difference Δt3 to obtain the corresponding estimated velocity vector v. * , The second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

[0066] More preferably, the second data d of data type speed in the current valid point object is processed based on the previous valid point object and the next and next two valid point objects. i,j Data correction processing includes:

[0067] The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the next valid point object, and the next two valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first coordinate vector p1 and the second coordinate vector p2;

[0068] The time scale differences from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the next two valid point objects are calculated to obtain the corresponding previous time difference Δt. -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the previous valid point object and the current valid point object and between the current valid point object and the next two valid point objects is taken as the corresponding longest time difference Δt. max ;

[0069] For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0070] If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0071] If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1The latter coordinate vector p1, the latter second coordinate vector p2, and the former time difference Δt -1 The subsequent time difference Δt1 and the subsequent second time difference Δt2 are used to estimate the velocity vector of the current effective point to obtain the corresponding estimated velocity vector v. * , The second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

[0072] More preferably, the second data d, whose data type is speed, in the current valid point object is determined based on the previous and second valid point objects and the next and second valid point objects. i,j Data correction processing includes:

[0073] The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, the next valid point object, and the next two valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 The first coordinate vector p1 and the second coordinate vector p2;

[0074] The time scale differences from the two previous valid point objects to the previous valid point object, from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the two subsequent valid point objects are calculated to obtain the corresponding first two time differences Δt. -2 Previous time difference △t -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the two preceding valid point objects and the current valid point object and the current valid point object and the two subsequent valid point objects is taken as the corresponding longest time difference Δt. max ;

[0075] For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0076] If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0077] If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The latter coordinate vector p1, the latter second coordinate vector p2, and the former two time differences Δt -2 The preceding time difference Δt -1 The subsequent time difference Δt1 and the subsequent second time difference Δt2 are used to estimate the velocity vector of the current effective point to obtain the corresponding estimated velocity vector v. * , The second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

[0078] More preferably, the second data d, whose data type is speed, in the current valid point object is determined based on the previous and second valid point objects and the next valid point object. i,j Data correction processing includes:

[0079] The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, and the next valid point object. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the next coordinate vector p1;

[0080] The time scale differences from the two previous valid point objects to the previous valid point object, from the previous valid point object to the current valid point object, and from the current valid point object to the next valid point object are calculated to obtain the corresponding first two time differences Δt. -2 Previous time difference △t-1 The maximum time difference Δt1 is taken as the corresponding longest time difference Δt, which is the time difference between the two previous valid point objects and the current valid point object and the time difference between the current valid point object and the next valid point object. max ;

[0081] For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0082] If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0083] If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The latter coordinate vector p1, and the former two time differences Δt -2 The preceding time difference Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector of the current effective point using the time difference Δt1. * , The second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

[0084] More preferably, the second data d with data type speed in the current valid point object is obtained by applying the data from the previous three valid point objects. i,j Data correction processing includes:

[0085] The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, and the previous three valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the first three coordinate vectors p-3 ;

[0086] The time scale differences from the first three valid point objects to the first two valid point objects, from the first two valid point objects to the previous valid point object, and from the previous valid point object to the current valid point object are calculated to obtain the corresponding first three time differences Δt. -3 The first two time differences △t -2 Time difference with the previous time Δt -1 The time scale difference between the first three valid point objects and the current valid point object is taken as the corresponding longest time difference Δt. max ;

[0087] For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0088] If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0089] If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The first three coordinate vectors p -3 The first three time differences △t -3 The first two time differences Δt -2 and the previous time difference Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector at the current effective point. * , The second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

[0090] Furthermore, the filtering correction method is used to apply the filtering correction method to each of the valid point objects V in the valid point object sequence. r Perform speed correction, specifically including

[0091] The corresponding Kalman filter observation-state equation set is created based on the preset motion model and is denoted as the first observation-state equation set; the first observation-state equation set uses position coordinates as the observation and velocity as the state variable;

[0092] From the second valid point object V in the sequence of valid point objects r The process begins by iterating through the valid point objects sequentially; during the iteration, the currently iterated valid point object V is... r As the corresponding current valid point object; and set the data type of the current valid point object to the second data d of the position coordinates. i,j As the observation at the current moment, the previous valid point object V of the current valid point object. r The data type is the second data d, which is the speed. i,j The observations at the current time and the state quantities at the previous time are used as the state quantities at the previous time; and the observations at the current time and the state quantities at the previous time are substituted into the first observation-state equation set to predict the state quantities at the current time to obtain the corresponding first predicted quantity v. p ; and set the data type of the current valid point object to the second data d of velocity. i,j Extracted as the corresponding first velocity vector v ’ ; and for the first velocity vector v ’ With the first prediction v p The system identifies whether the vector difference meets the preset vector difference range; if so, it proceeds to the next valid point object V. r Continue the traversal; otherwise, change the data type of the current valid point object to the second data d of velocity. i,j Reset to the first prediction value v p .

[0093] Preferably, the step of identifying the mask data sequences of various data sources based on the first integrated data sequence to obtain corresponding multi-class mask data sequences specifically includes:

[0094] The number of the first data sources of the multiple types of data sources is counted to obtain the corresponding number of the first data sources;

[0095] The length of the first mask data sequence is initialized to the maximum scale index M; the initialized first mask data sequence includes the first mask data Mask of the maximum scale index M. i Each of the first mask data Mask i The mask value is initialized to 0; the first mask data sequence corresponds one-to-one with the first data source;

[0096] Each of the first data sources is traversed; during traversal, the current first data source is taken as the corresponding current data source, the first source type of the current data source is taken as the corresponding current source type, and the first mask data sequence corresponding to the current data source is taken as the corresponding current mask data sequence; and the second data object D in the first integrated data sequence whose second source type matches the current source type is taken as the current data sequence. i All are marked as the corresponding current second data object; and the first time scale t of each of the current second data objects is set. i The scale index i is extracted as the corresponding first scale index; and the first mask data Mask corresponding to each of the first scale indices in the current mask data sequence is used. i The mask value is reset to 1;

[0097] At the end of the traversal, the corresponding multi-class mask data sequence is output, which is composed of all the first mask data sequences.

[0098] A second aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0099] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0100] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0101] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0102] This invention provides a data processing method, electronic device, and computer-readable storage medium for multiple data sources. It integrates multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters) into a single temporal space. To improve the accuracy of the integrated data, a mechanism for correcting the velocity information of the integrated data is provided. Furthermore, to improve the model adaptability of the integrated data, a corresponding mask data sequence with temporal features is generated for each data source. This invention provides a solution for the overall preprocessing of multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters). Based on this solution, the prediction accuracy of various prediction models in the sensing module can be further improved. Attached Figure Description

[0103] Figure 1 This is a schematic diagram of a data processing method for multiple data sources provided in Embodiment 1 of the present invention;

[0104] Figure 2 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0106] The perception module of the autonomous driving system can perform overall preprocessing on multiple data sources (perception data source output by LiDAR, perception data source output by millimeter-wave radar, perception data source output by camera, target detection data source output by target detection model, and state quantity prediction data source output by filter) through a data processing method for multiple data sources provided in Embodiment 1 of the present invention. Figure 1 This is a schematic diagram of a data processing method for multiple data sources provided in Embodiment 1 of the present invention, as shown below. Figure 1 As shown, this method mainly includes the following steps:

[0107] Step 1: At any given time, obtain the current vehicle coordinates and the current road scene as the corresponding first vehicle coordinates and first road scene; and obtain multiple data sources within the most recent first time period.

[0108] The first road scenario includes highway scenarios, street road scenarios, rural road scenarios, and urban road scenarios; the multiple data sources include multiple first data sources; the first data sources include first source types and multiple first data objects; the first source types include LiDAR, millimeter-wave radar, camera, target detection model, and filter types; the first data objects include first time and first data groups; the first data groups include one or more first data; the data types of the first data include position coordinates, orientation angle, target detection box size, and velocity; the coordinate system of the first data with position coordinates as the data type and the coordinate system of the first vehicle coordinates are both preset unified coordinate systems, which are either geodetic coordinate systems or BEV coordinate systems.

[0109] Here, if the first source type is LiDAR, millimeter-wave radar, or camera, it means the first data source is the perception data source output by LiDAR, millimeter-wave radar, or camera, which is commonly referred to as the basic feature data of LiDAR point cloud, millimeter-wave radar point cloud, and camera-captured image. These basic feature data refer to the basic feature data obtained after feature extraction from the original LiDAR point cloud, original millimeter-wave radar point cloud, and original camera-captured image, respectively. In this case, the first data group of each first data object typically includes data with data types of position coordinates and orientation angle. If the first source type is target detection model, it means the first data source is the target detection data output by the target detection model, which is commonly referred to as the target detection box. In this case, the first data group of each first data object typically includes data with data types of position coordinates, orientation angle, and target detection box size. If the first source type is filter, it means the first data source is the state quantity prediction data output by the filter, which is commonly referred to as the motion state prediction. In this case, the first data group of each first data object typically includes data with data types of position coordinates, orientation angle, and velocity. It is evident that different source types result in different data compositions for the first data group. Furthermore, due to the different sampling frequencies of each source, there is a time synchronization issue. Subsequent steps are needed to integrate the aforementioned source data into a unified time domain space.

[0110] Step 2: Perform position coordinate transformation on multiple data sources based on the first vehicle coordinates and the first road scene;

[0111] Here, in this embodiment of the invention, the relative distances between the position coordinates of the three closest first data objects to the vehicle (i.e., the first, second, and third closest coordinates) and the vehicle are used to calculate the position transformation evaluation index. The larger the position transformation evaluation index, the larger the distance between the first data objects in the multiple data sources obtained this time, and the more necessary it is to reduce the coordinates to improve data sensitivity; the smaller the position transformation evaluation index, the smaller the distance between the first data objects in the multiple data sources obtained this time, and the more necessary it is to perform a vehicle coordinate system transformation to improve data sensitivity.

[0112] Specifically, this includes: Step 21, marking each first data point in the multiple data sources whose data type is position coordinate as the corresponding first target coordinate; calculating the straight-line distance between each first target coordinate and the first vehicle coordinate to generate the corresponding first target distance; and selecting the three shortest first target distances as the corresponding first, second, and third nearest coordinates;

[0113] Step 22: Based on the first, second, and third nearest coordinates and the first road scene, estimate the position transformation evaluation index to generate the corresponding first evaluation index P;

[0114] Specifically, this includes: step 221, assigning the first vehicle coordinate to the corresponding first vehicle coordinate (x0, y0, z0); and assigning the first, second, and third nearest coordinates to the corresponding first nearest coordinate (x1, y1, z1), second nearest coordinate (x2, y2, z2), and third nearest coordinate (x3, y3, z3);

[0115] Step 222: Identify the first road scene; if the first road scene is a highway scene, then set the corresponding first weight sequence {w1, w2, w3} to the preset first weight constant sequence {w 1,1 ,w 1,2 ,w 1,3 If the first road scene is a street scene, then the corresponding first weight sequence {w1, w2, w3} is set to the preset second weight constant sequence {w 2,1 ,w 2,2 ,w 2,3 If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight sequence {w1, w2, w3} is set to the preset third weight constant sequence {w 3,1 ,w 3,2 ,w 3,3};

[0116] Here, the first, second, and third weight constant sequences are three pre-defined empirical constant sequences. Under normal circumstances: the first weight constant sequence {w 1,1 ,w 1,2 ,w 1,3} is {2.0, 0.1, 0.1}, the second weight constant sequence {w 2,1 ,w 2,2 ,w 2,3} is {0.5, 2.0, 2.0}, the third weight constant sequence {w 3,1 ,w 3,2 ,w 3,3 The expression is {1.0, 1.0, 1.0}.

[0117] Step 223: Calculate the corresponding first evaluation index P based on the first vehicle coordinates (x0, y0, z0), the first nearest coordinates (x1, y1, z1), the second nearest coordinates (x2, y2, z2), the third nearest coordinates (x3, y3, z3), and the first weight sequence {w1, w2, w3}.

[0118]

[0119]

[0120]

[0121]

[0122] Step 23: Identify whether the first evaluation index P is less than the preset first index threshold; if yes, set the corresponding first conversion mode to the first mode; if no, set the corresponding first conversion mode to the second mode.

[0123] Here, the first indicator threshold is a preset empirical constant threshold, which is normally 10;

[0124] Step 24: Based on the first conversion mode, the first road scene, and the first evaluation index P, perform position coordinate transformation processing on each first position coordinate to generate the corresponding second position coordinate;

[0125] Specifically, this includes: step 241, performing coordinate vector transformation on the first position coordinates to obtain the corresponding first coordinate vector S1;

[0126] Step 242: When the first transformation mode is the first mode, confirm the rotation matrix and offset matrix from the unified coordinate system to the vehicle coordinate system to obtain the corresponding first rotation matrix A and first offset matrix T; perform coordinate transformation on the first coordinate vector S1 according to the first rotation matrix A and the first offset matrix T to obtain the corresponding second coordinate vector S2.

[0127] S2 = A·S1 + T;

[0128] Step 243: When the first conversion mode is the second mode, the first road scene is identified; if the first road scene is a highway scene, the corresponding first weight parameter k is set to a preset first weight constant. If the first road scenario is a street scene, then the corresponding first weight parameter k is set to the preset second weight constant. If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight parameter k is set to a preset third weight constant. Then, based on the first weight parameter k and the first evaluation index P, the first coordinate vector S1 is transformed to obtain the corresponding second coordinate vector S2.

[0129]

[0130] Here, the first, second, and third weighting constants are three preset empirical weighting constants. Under normal circumstances: the first weighting constant... The second weighting constant is 2.0. The third weighting constant is 1.5. It is 1.0;

[0131] Step 244: Extract each coordinate component of the second coordinate vector S2 to form the corresponding second position coordinates;

[0132] Step 25: Reset the data of the first data with the corresponding data type as position coordinates in the multiple data sources based on each second position coordinate.

[0133] Step 3: If the location coordinate transformation is successful, the earliest data time in the multiple data sources is taken as the starting time point, and the time period from the starting time point to the current time is divided into time axis segments according to the preset data frequency f to obtain the corresponding first time axis.

[0134] The first time axis includes multiple first time scales t. i 0 ≤ tick index i ≤ maximum tick index M, maximum tick index M ≥ 0; the first time tick t i The starting time point is the last first time scale t. i The end time point;

[0135] Specifically, this includes: taking the earliest first time from multiple data sources as the corresponding starting time point; taking the time period from the starting time point to the current time as the corresponding first data period; calculating the corresponding first time interval Δt = 1 / f based on the data frequency f; dividing the first data period into multiple equal time scales based on the first time interval Δt, with each equal time scale including the starting time point; counting the number of equal time scales to obtain the corresponding first quantity; using the first quantity minus 1 as the corresponding maximum scale index M; and taking the first equal time scale as the corresponding first time scale t. i＝0 The second equally divided time scale is taken as the corresponding first time scale t. i＝1 This process continues until the last equally divided time scale is taken as the corresponding first time scale t. i＝M Up to; and by all the first time scales t obtained. i This forms the corresponding first time axis.

[0136] Step 4: Based on the time scale of the first time axis, perform data integration processing on multiple data sources to generate the corresponding first integrated data sequence; and perform data correction processing on the speed type data in the first integrated data sequence;

[0137] Specifically, this includes: Step 41, performing data integration processing on multiple data sources based on the time scale of the first time axis to generate the corresponding first integrated data sequence;

[0138] The first integrated data sequence includes multiple second data objects D. i Second data object D i Including the first time scale t i The second source type includes LiDAR, millimeter-wave radar, camera, target detection model, and filter types; the second data group includes multiple second data sets. i,j , 1 ≤ data index j ≤ maximum data index N, maximum data index N ≥ 1; second data d i,j The data types include position coordinates, orientation angle, target detection box size, and velocity; the second data d is of data type position coordinates. i,j The coordinate system of the first vehicle coordinate system and the coordinate system of the first vehicle coordinate system are both preset unified coordinate systems;

[0139] Specifically, this includes: Step 411, counting the number of all data types in multiple data sources to obtain the corresponding total number of all types, and setting the maximum data index N based on the total number of all types;

[0140] Step 412: Initialize the first integrated data sequence according to the maximum scale index M and the maximum data index N; and in the initialized first integrated data sequence, add all second data objects D.i This is denoted as the corresponding first invalid data object; wherein, the first integrated data sequence obtained after initialization includes the second data object D with the maximum scale index M. i The second data objects D obtained after initialization i Compared with the first time scale t i One-to-one correspondence; the second data object D obtained after initialization i Includes a corresponding first time scale t i A second source type initialized to empty and a second data group with a fixed data length of N-2; the initialized second data group includes N-2 initialized empty second data d. i,j ;

[0141] Here, it can be understood that a time domain space with M on the horizontal axis, N on the vertical axis, and M×N shape has been created, where the horizontal axis is the time scale and the vertical axis is the dimension of the data type; it should be noted that in this embodiment of the invention, setting the data to empty actually means setting the value of the data to 0;

[0142] Step 413: Sort all the first data objects from multiple data sources according to their chronological order to obtain the corresponding sequence of first data objects;

[0143] Step 414: Iterate through the first data objects in the first data object sequence sequentially; during iteration, take the currently iterated first data object as the corresponding current first data object, take the first source type of the first data source corresponding to the current first data object as the corresponding current type, take the first time of the current first data object as the corresponding current data time, and take the first time scale t on the first time axis corresponding to the current data time as the current data time. i As the corresponding current time scale, and using the second data object D in the first integrated data sequence that corresponds to the current time scale, i As the corresponding current second data object; and identify whether the current second data object is recorded as the first invalid data object; if the current second data object is recorded as the first invalid data object, then set the second source type of the current second data object to the current type, and according to one or more first data of the current first data object, set the corresponding second data d in the second data group of the current second data object. i,jThe settings are configured, and upon successful configuration, the current second data object is reclassified as the corresponding first valid data object. If the current second data object is not reclassified as the first invalid data object, the second source type of the current second data object is used as the corresponding first type. A type optimization process is then performed on the first type and the current type according to a preset type optimization mode to obtain the corresponding first optimization result. When the first optimization result is the current type, the second source type and the second data group of the current second data object are cleared, and the second source type of the current second data object is set to the current type. Furthermore, based on one or more first data points of the current first data object, the corresponding second data d in the second data group of the current second data object is configured. i,j Configure settings;

[0144] Here, the current step is actually to fill the data of each first data object in multiple data sources into the M×N time domain space in a way that proceeds from far to near in time.

[0145] It should be noted that during the filling process, multiple first data objects may correspond to the same event at the same time. Therefore, this embodiment of the invention provides a data optimization method based on the source type corresponding to the duplicate data object, that is, to perform type optimization processing on the first type and the current type according to a preset type optimization mode to solve this problem; specifically, the above-mentioned type optimization processing on the first type and the current type according to the preset type optimization mode to obtain the corresponding first optimization result includes:

[0146] Step A1: Identify the type selection pattern;

[0147] Among them, the type optimization mode includes the first optimization mode, the second optimization mode, and the third optimization mode;

[0148] Here, when the type selection mode is the first selection mode, it means that the priority of the data source needs to be selected based on the real-time weather conditions; when the type selection mode is the second selection mode, it means that the priority of the data source needs to be determined based on the number of valid data of the two source types in the first data object sequence; when the type selection mode is the third selection mode, it means that the priority of the data source needs to be determined based on the number of valid data of the two source types in the period before the current time point.

[0149] Step A2: When the type preference mode is the first preference mode, obtain the current weather type as the corresponding first weather type; query the preset data source level list according to the first weather type, extract the first type priority sequence field of the first data source level record that matches the first weather type in the data source level list as the corresponding first type priority sequence; record the sequence index of the third source type that matches the first type in the first type priority sequence as the corresponding first index, and record the sequence index of the third source type that matches the current type as the corresponding current index; when the current index is less than the first index, set the corresponding first preference result as the current type; when the current index is greater than the first index, set the corresponding first preference result as the first type; the data source level list includes multiple first data source level records; the first data source level record includes a first weather type field and a first type priority sequence field; the first type priority sequence is composed of multiple third source types sorted.

[0150] The third source types include lidar, millimeter-wave radar, camera, target detection model, and filter.

[0151] Step A3: When the type preference mode is the second preference mode, for the second data object D in the first data object sequence whose second source type is the first type and which is recorded as the first valid data object... i The quantity is counted to obtain the corresponding second quantity; and the second data object D in the first data object sequence whose second source type is the current type and is recorded as the first valid data object is... i The quantities are counted to obtain the corresponding third quantity; when the third quantity is greater than the second quantity, the corresponding first preferred result is set as the current type; when the third quantity is less than or equal to the second quantity, the corresponding first preferred result is set as the first type.

[0152] Step A4: When the type preference mode is the third preference mode, select the second data object D from the first data object sequence that is closest to the current second data object and is recorded as the first valid data object by a specified number H. i Form a corresponding second data object set; and for the second data object D in the second data object set whose second source type is the first type. i The quantity is statistically analyzed to obtain the corresponding fourth quantity; and the second data object D with the second source type of the current type in the second data object set is analyzed. i The number of items is counted to obtain the corresponding fifth number; when the fifth number is greater than the fourth number, the corresponding first preferred result is set as the current type; when the fifth number is less than or equal to the fourth number, the corresponding first preferred result is set as the first type.

[0153] Step 415: When the traversal ends, the first integrated data sequence that has been filled with data is output as the corresponding data integration processing result.

[0154] Step 42: Perform data correction processing on the velocity type data in the first integrated data sequence;

[0155] Here, in order to improve the accuracy of integrated data, this embodiment of the invention also provides a processing mechanism for correcting the speed information of integrated data;

[0156] Specifically, this includes: Step 421, taking the second data d from the first integrated data sequence, whose data type is position coordinates. i,j The second data object D corresponding to the non-empty second data group i Let V be the corresponding valid point object. r The index of the valid point object is r≥1; and it is determined by all valid point objects V. r Sorting generates a corresponding sequence of valid point objects;

[0157] Step 423: Identify the preset speed correction mode; the speed correction mode includes a first correction mode and a second correction mode.

[0158] Here, this embodiment of the invention supports two speed correction modes: a first correction mode and a second correction mode. When the speed correction mode is the first correction mode, it means that a linear correction method is used for correction. In fact, the speed at the current correction time point is estimated based on the reference point position and time difference before and after each correction time point, and the estimated speed is compared with the current filling speed. If the difference is reasonable, the current filling speed is maintained; if it is unreasonable, the estimated speed is used to replace the current filling speed. When the speed correction mode is the second correction mode, it means that a filtering correction method is used for correction. In fact, the speed at each correction time point is estimated based on conventional Kalman filtering technology, and the estimated speed is compared with the current filling speed. If the difference is reasonable, the current filling speed is maintained; if it is unreasonable, the estimated speed is used to replace the current filling speed.

[0159] Step 423: When the velocity correction mode is the first correction mode, a linear correction method is used to correct each valid point object V in the valid point object sequence. r Perform speed correction;

[0160] Specifically, this includes: step 4231, setting the first valid point object V... r As the corresponding current valid point object; and the three nearest valid point objects V following the current valid point object in the valid point object sequence. rLet these be the corresponding last one, second, and third valid point objects; and based on the last one, second, and third valid point objects, determine the second data d of data type speed in the current valid point object. i,j Perform data correction processing;

[0161] Among them, based on the last three valid point objects, the second data d of data type speed in the current valid point object is used. i,j Data correction processing includes:

[0162] Step B1: The second data d, whose data type is position coordinates, is retrieved from the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d of the data type position coordinate from the last one, the second one, and the third one valid point objects. i,j Extract them as the corresponding next coordinate vector p1, next second coordinate vector p2, and next third coordinate vector p3;

[0163] Step B2: Calculate the time scale differences from the current valid point object to the next valid point object, from the next valid point object to the second valid point object, and from the second valid point object to the third valid point object to obtain the corresponding subsequent time difference Δt1, subsequent second time difference Δt2, and subsequent third time difference Δt3; and calculate the longest time difference Δt from the time scale difference between the current valid point object and the third valid point object. max ;

[0164] Step B3, for the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0165] Step B4, if the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0166] Step B5, if the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, the velocity vector of the current valid point is estimated based on the current coordinate vector p0, the next coordinate vector p1, the next second coordinate vector p2, the next third coordinate vector p3, the next time difference Δt1, the next second time difference Δt2, and the next third time difference Δt3 to obtain the corresponding estimated velocity vector v. * , The second data d with speed as the data type in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it adds the second data d of velocity type to the current valid point object.i,j Reset to the corresponding estimated velocity vector v * ;

[0167] Step 4232, transfer the second valid point object V r As the corresponding current valid point object; and take the nearest valid point object V in the valid point object sequence that precedes the current valid point object. r Let V be the corresponding previous valid point object; and then denote the two nearest valid point objects V following the current valid point object in the valid point object sequence. r Let these be the corresponding first and second valid point objects; and based on the first and second valid point objects, determine the second data d of data type velocity in the current valid point object. i,j Perform data correction processing;

[0168] Among them, based on the previous valid point object and the next and second valid point objects, the second data d of data type speed in the current valid point object is calculated. i,j Data correction processing includes:

[0169] Step C1: The second data d, whose data type is position coordinates, is retrieved from the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, which is of data type position coordinate, from the previous valid point object, the next valid point object, and the next two valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first coordinate vector p1 and the second coordinate vector p2;

[0170] Step C2: Calculate the time scale differences from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the next two valid point objects to obtain the corresponding previous time difference Δt. -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the previous valid point object and the current valid point object and between the current valid point object and the two subsequent valid point objects is taken as the corresponding longest time difference Δt. max ;

[0171] Step C3, for the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0172] Step C4, if the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0173] Step C5, if the longest time difference Δt is confirmed maxIf the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first coordinate vector p1, the second coordinate vector p2, and the first time difference Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector at the current effective point using the first time difference Δt1 and the second time difference Δt2. * , The second data d with speed as the data type in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it adds the second data d of velocity type to the current valid point object. i,j Reset to the corresponding estimated velocity vector v * ;

[0174] Step 4233, transfer the third valid point object V r As the corresponding current valid point object; and take the two nearest valid point objects V in the valid point object sequence that precede the current valid point object. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the valid point object sequence. r Let these be the corresponding first and second valid point objects; and based on the first and second valid point objects and the first and second valid point objects, determine the second data d of data type speed in the current valid point object. i,j Perform data correction processing;

[0175] Among them, based on the previous two valid point objects and the next two valid point objects, the second data d of data type speed in the current valid point object is calculated. i,j Data correction processing includes:

[0176] Step D1: The second data d, whose data type is position coordinate, is retrieved from the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, which is of data type position coordinate, from the previous valid point object, the second valid point object, the next valid point object, and the second valid point object. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 The first coordinate vector p1 and the second coordinate vector p2;

[0177] Step D2: Calculate the time scale differences from the previous two valid point objects to the previous valid point object, from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the next two valid point objects to obtain the corresponding first two time differences Δt. -2 Previous time difference △t -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the two preceding valid point objects and the current valid point object and the two subsequent valid point objects is taken as the corresponding longest time difference Δt. max ;

[0178] Step D3, for the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0179] Step D4, if the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0180] Step D5, if the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The first coordinate vector p1, the second coordinate vector p2, and the time difference Δt between the first two coordinate vectors. -2 Previous time difference △t -1 The estimated velocity vector v is obtained by estimating the velocity vector at the current effective point using the first time difference Δt1 and the second time difference Δt2. * , The second data d with speed as the data type in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it adds the second data d of velocity type to the current valid point object. i,j Reset to the corresponding estimated velocity vector v * ;

[0181] Step 4234, the fourth valid point object V r As the corresponding current valid point object; and take the two nearest valid point objects V in the valid point object sequence that precede the current valid point object. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the valid point object sequence. rLet these be the corresponding first and second valid point objects; and based on the first and second valid point objects and the first and second valid point objects, determine the second data d of data type speed in the current valid point object. i,j Perform data correction processing;

[0182] In the current step, the second data d with speed data type in the current valid point object is processed based on the previous two valid point objects and the next two valid point objects. i,j The data correction process is the same as that in step 4233, and will not be repeated here.

[0183] Steps 4235, and so on, continue until the second-to-last valid point object V is found. r As the corresponding current valid point object; and take the two nearest valid point objects V in the valid point object sequence that precede the current valid point object. r Let V be the corresponding first and second valid point objects; and let V be the nearest valid point object after the current valid point object in the valid point object sequence. r This is denoted as the corresponding next valid point object; and based on the previous, the two previous valid point objects, and the next valid point object, the second data d of data type velocity in the current valid point object is... i,j Perform data correction processing;

[0184] Among them, based on the previous two valid point objects and the next valid point object, the second data d of data type speed in the current valid point object is calculated. i,j Data correction processing includes:

[0185] Step E1: The second data d, whose data type is position coordinates, is retrieved from the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, which is of data type position coordinate, from the previous valid point object, the two previous valid point objects, and the next valid point object. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the next coordinate vector p1;

[0186] Step E2: Calculate the time scale differences between the two previous valid point objects and the previous valid point object, between the previous valid point object and the current valid point object, and between the current valid point object and the next valid point object to obtain the corresponding time difference Δt. -2 Previous time difference △t -1 The maximum time difference between the two previous valid point objects and the current valid point object, and the maximum time difference between the current valid point object and the next valid point object, is taken as the corresponding longest time difference Δt. max ;

[0187] Step E3, for the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0188] Step E4, if the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0189] Step E5, if the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The second coordinate vector p1, and the first two time differences Δt -2 Previous time difference △t -1 The estimated velocity vector v is obtained by estimating the velocity vector of the current effective point using the time difference Δt1. * , The second data d with speed as the data type in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it adds the second data d of velocity type to the current valid point object. i,j Reset to the corresponding estimated velocity vector v * ;

[0190] Step 4236, transfer the last valid point object V r As the corresponding current valid point object; and take the three nearest valid point objects V in the valid point object sequence that precede the current valid point object. r Let these be the corresponding first, second, and third valid point objects; and based on the first, second, and third valid point objects, determine the second data d of data type speed in the current valid point object. i,j Perform data correction processing;

[0191] Among them, based on the first, second, and third valid point objects, the second data d of the current valid point object with data type of speed is used. i,j Data correction processing includes:

[0192] Step F1: The second data d, whose data type is position coordinates, is retrieved from the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d with the data type of position coordinate from the previous valid point object, the second valid point object, and the third valid point object. i,jExtract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the first three coordinate vectors p -3 ;

[0193] Step F2: Calculate the time scale differences from the first three valid point objects to the first two valid point objects, from the first two valid point objects to the previous valid point object, and from the previous valid point object to the current valid point object to obtain the corresponding first three time differences Δt. -3 The first two time differences △t -2 Time difference with the previous time Δt -1 The time scale difference between the first three valid point objects and the current valid point object is taken as the corresponding longest time difference Δt. max ;

[0194] Step F3, for the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded;

[0195] Step F4, if the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated.

[0196] Step F5, if the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The first three coordinate vectors p -3 The first three time differences △t -3 The first two time differences △t -2 Time difference with the previous time Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector at the current effective point. * , The second data d with speed as the data type in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it adds the second data d of velocity type to the current valid point object. i,j Reset to the corresponding estimated velocity vector v * ;

[0197] Step 424: When the velocity correction mode is the second correction mode, a filtering correction method is used to correct each valid point object V in the valid point object sequence. r Perform speed correction;

[0198] Specifically, it includes: Step 4241, creating a corresponding Kalman filter observation-state equation set based on the preset motion model, denoted as the first observation-state equation set;

[0199] Among them, the first observation-state equation set uses position coordinates as the observation and velocity as the state variable;

[0200] Here, the motion model is a pre-defined obstacle motion model, such as a uniform motion model or a uniformly accelerated motion model. The method of constructing a Kalman filter observation-state equation set based on a known motion model is also a publicly available technical implementation method, and will not be elaborated further here.

[0201] Step 4242, from the second valid point object V in the valid point object sequence r The process begins by iterating through the valid point objects one by one; during the iteration, the currently traversed valid point object V is... r As the corresponding current valid point object; and set the data type of the current valid point object to the second data d of the position coordinates. i,j As the observation at the current moment, the previous valid point object V of the current valid point object. r The data type is the second data d of velocity. i,j The observations at the current moment and the state variables at the previous moment are used as the state variables. These are then substituted into the first observation-state equation set to predict the state variables at the current moment, yielding the corresponding first predicted value v. p ; and set the data type of the current valid point object to the second data d of velocity. i,j Extracted as the corresponding first velocity vector v ’ ; and for the first velocity vector v ’ With the first predictor v p The system identifies whether the vector difference meets the preset vector difference range; if so, it proceeds to the next valid point object V. r Continue the traversal; otherwise, change the data type of the current valid point object to the second data d of the velocity. i,j Reset to the first predictor v p .

[0202] Here, the current step is actually based on the conventional Kalman filtering technique for each correction time point, that is, the effective point object V. r The speed is estimated, and the estimated speed is compared with the current valid point object V. r The filling speed is compared; if the difference is reasonable, the current filling speed is maintained; otherwise, the estimated speed is used for the current valid point object V. r The filling speed is replaced.

[0203] Step 5: Based on the first integrated data sequence, identify the mask data sequences of various data sources to obtain the corresponding multi-class mask data sequences;

[0204] Among them, the multi-class mask data sequence includes multiple first mask data sequences; the first mask data sequence includes multiple first mask data Masks. i First mask data i The mask values ​​include 0 and 1;

[0205] Specifically, this includes: Step 51, counting the number of first data sources for multiple types of data sources to obtain the corresponding number of first data sources;

[0206] Step 52: Initialize the first mask data sequence with a length of M equal to the maximum scale index of the first data source; the initialized first mask data sequence includes the first mask data Mask with the maximum scale index M. i Each first mask data Mask i The mask value is initialized to 0; the first mask data sequence corresponds one-to-one with the first data source;

[0207] Step 53: Traverse each first data source; during traversal, take the current first data source as the corresponding current data source, take the first source type of the current data source as the corresponding current source type, and take the first mask data sequence corresponding to the current data source as the corresponding current mask data sequence; and take the second data object D in the first integrated data sequence whose second source type matches the current source type. i All are marked as the corresponding current second data object; and the first time scale t of each current second data object is set. i Extract the tick index i as the corresponding first tick index; and extract the first mask data Mask corresponding to each first tick index in the current mask data sequence. i The mask value is reset to 1;

[0208] Step 54: When the traversal is complete, output the corresponding multi-class mask data sequence composed of all the first mask data sequences.

[0209] Here, when the perception module of the vehicle autonomous driving system uses some prediction models with self-attention or multi-head attention networks, it needs to prepare corresponding mask data sequences in order to reduce the number of calculations of the model and improve the prediction efficiency of the model. Therefore, in this embodiment of the invention, while performing overall data integration on multiple data sources, a corresponding first mask data sequence with temporal domain features is generated for each type of data source to form a corresponding multi-type mask data sequence.

[0210] Step 6: Output the first integrated data sequence and the multi-class mask data sequence after data correction as the corresponding data processing results.

[0211] Figure 2 This is a schematic diagram of an electronic device provided in Embodiment 2 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 2 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0212] exist Figure 2 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0213] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0214] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0215] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0216] This invention provides a data processing method, electronic device, and computer-readable storage medium for multiple data sources. It integrates multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters) into a single temporal space. To improve the accuracy of the integrated data, a mechanism for correcting the velocity information of the integrated data is provided. Furthermore, to improve the model adaptability of the integrated data, a corresponding mask data sequence with temporal features is generated for each data source. This invention provides a solution for the overall preprocessing of multiple data sources (sensing data sources output by LiDAR, millimeter-wave radar, cameras, target detection models, and state prediction data sources from filters). Based on this solution, the prediction accuracy of various prediction models in the sensing module can be further improved.

[0217] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0218] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0219] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method for multiple data sources, characterized in that, The method includes: At any given time, obtain the current vehicle coordinates and the current road scene as the corresponding first vehicle coordinates and first road scene; and obtain multiple data sources within the most recent first time period. The location coordinate transformation process is performed on the multiple data sources based on the first vehicle coordinates and the first road scene; If the location coordinate transformation is successful, the earliest data time in the multiple data sources is taken as the starting time point and the time period from the starting time point to the current time is divided into time axis segments according to the preset data frequency f to obtain the corresponding first time axis. Based on the time scale of the first time axis, the data from the multiple data sources are integrated to generate a corresponding first integrated data sequence; and the speed type data in the first integrated data sequence is corrected. Based on the first integrated data sequence, the mask data sequences of various data sources are identified to obtain corresponding multi-class mask data sequences; The first integrated data sequence and the multi-class mask data sequence after data correction are output as the corresponding data processing results. The first road scenario includes highway scenarios, street road scenarios, rural road scenarios, and urban road scenarios; the multiple data sources include multiple first data sources; the first data sources include first source types and multiple first data objects; the first source types include LiDAR, millimeter-wave radar, camera, target detection model, and filter types; the first data objects include first time and first data groups; the first data groups include one or more first data; the data types of the first data include position coordinates, orientation angle, target detection box size, and speed; the coordinate system of the first data with position coordinates and the coordinate system of the first vehicle coordinates are both preset unified coordinate systems, which are either geodetic coordinate systems or BEV coordinate systems; The step of performing position coordinate transformation processing on the multiple data sources based on the first vehicle coordinates and the first road scene specifically includes: Each of the first data points in the multiple data sources whose data type is location coordinate is marked as a corresponding first target coordinate; the straight-line distance between each first target coordinate and the first vehicle coordinate is calculated to generate a corresponding first target distance; and the three first target distances with the shortest distance are selected as the corresponding first, second and third nearest coordinates. Based on the first, second, and third nearest coordinates and the first road scene, the location transformation evaluation index is estimated to generate the corresponding first evaluation index P; The system identifies whether the first evaluation index P is less than a preset first index threshold; if so, it sets the corresponding first conversion mode to the first mode; if not, it sets the corresponding first conversion mode to the second mode. Based on the first conversion mode, the first road scene, and the first evaluation index P, the first location coordinates are converted to generate corresponding second location coordinates. Based on each of the second position coordinates, the first data in the multiple data sources whose corresponding data type is position coordinate is reset; The step of estimating the location transformation evaluation index and generating the corresponding first evaluation index P based on the first, second, and third nearest coordinates and the first road scene specifically includes: The first vehicle coordinate is marked as the corresponding first vehicle coordinate (x0, y0, z0); and the first, second, and third nearest coordinates are marked as the corresponding first nearest coordinate (x1, y1, z1), second nearest coordinate (x2, y2, z2), and third nearest coordinate (x3, y3, z3). The first road scene is identified; if the first road scene is a highway scene, the corresponding first weight sequence {w1, w2, w3} is set as a preset first weight constant sequence {w 1,1 ,w 1,2 ,w 1,3 If the first road scenario is a street scene, then the corresponding first weight sequence {w1, w2, w3} is set to a preset second weight constant sequence {w...}. 2,1 ,w 2,2 ,w 2,3 If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight sequence {w1, w2, w3} is set to a preset third weight constant sequence {w 3,1 ,w 3,2 ,w 3,3 }; where the first weight constant sequence {w 1,1 ,w 1,2 ,w 1,3 }, the second weight constant sequence, and the third weight constant sequence {w 3,1 ,w 3,2 ,w 3,3 The values ​​are {2.0, 0.1, 0.1}, {0.5, 2.0, 2.0}, and {1.0, 1.0, 1.0}, respectively. The first evaluation index P is calculated based on the first vehicle coordinates (x0, y0, z0), the first nearest coordinates (x1, y1, z1), the second nearest coordinates (x2, y2, z2), the third nearest coordinates (x3, y3, z3), and the first weight sequence {w1, w2, w3}. , , , ； The step of performing position coordinate transformation processing on each of the first position coordinates according to the first conversion mode, the first road scene, and the first evaluation index P to generate corresponding second position coordinates specifically includes: The first position coordinates are transformed into a coordinate vector to obtain the corresponding first coordinate vector S1; When the first transformation mode is the first mode, the rotation matrix and offset matrix from the unified coordinate system to the vehicle coordinate system are confirmed to obtain the corresponding first rotation matrix A and first offset matrix T; based on the first rotation matrix A and the first offset matrix T, the first coordinate vector S1 is transformed to obtain the corresponding second coordinate vector S2. ; When the first conversion mode is the second mode, the first road scene is identified; if the first road scene is a highway scene, the corresponding first weight parameter k is set to a preset first weight constant. If the first road scenario is a street scene, then the corresponding first weight parameter k is set to a preset second weight constant. If the first road scenario is a rural road scenario or an urban road scenario, then the corresponding first weight parameter k is set to a preset third weight constant. The first coordinate vector S1 is then transformed according to the first weight parameter k and the first evaluation index P to obtain the corresponding second coordinate vector S2. Wherein, the first weighting constant The second weighting constant The third weighting constant These correspond to versions 2.0, 1.5, and 1.0, respectively. Extract each coordinate component of the second coordinate vector S2 to form the corresponding second position coordinates.

2. The data processing method for multiple data sources according to claim 1, characterized in that, The first time axis includes multiple first time scales t i 0 ≤ scale index i ≤ maximum scale index M, maximum scale index M ≥ 0; the first time scale t i The starting time point is the last first time scale t. i The end time point; The first integrated data sequence includes multiple second data objects D i The second data object D i Including the first time scale t i The second source type includes LiDAR, millimeter-wave radar, camera, target detection model, and filter types; the second data group includes multiple second data sets. i,j , 1 ≤ data index j ≤ maximum data index N, maximum data index N ≥ 1; the second data d i,j The data types include position coordinates, orientation angle, target detection box size, and velocity; the second data d with the data type being position coordinates. i,j The coordinate system of the first vehicle coordinate system and the coordinate system of the first vehicle coordinate system are both the preset unified coordinate system; The multiple types of mask data sequences include multiple first mask data sequences; the first mask data sequences include multiple first mask data Masks. i The first mask data Mask i The mask values ​​include 0 and 1.

3. The data processing method for multiple data sources according to claim 2, characterized in that, The first time axis is obtained by dividing the time period from the earliest data time in the multiple data sources into time axis segments according to a preset data frequency f, using the earliest data time in the multiple data sources as the starting time point. Specifically, this includes: The earliest first time among the multiple data sources is taken as the corresponding start time point; the time period from the start time point to the current time is taken as the corresponding first data time period; the corresponding first time interval Δt = 1 / f is calculated according to the data frequency f; the first data time period is divided into multiple equal time scales according to the first time interval Δt, the equal time scales including the start time point; the number of the obtained equal time scales is counted to obtain the corresponding first quantity; the value of the first quantity minus 1 is taken as the corresponding maximum scale index M; and the first equal time scale is taken as the corresponding first time scale t. i=0 The second equally divided time scale is taken as the corresponding first time scale t. i=1 This process continues until the last equally divided time scale is taken as the corresponding first time scale t. i=M Up to; and by all the first time scales t obtained. i This forms the corresponding first time axis.

4. The data processing method for multiple data sources according to claim 2, characterized in that, The step of integrating and processing the data from the multiple data sources based on the time scale of the first time axis to generate a corresponding first integrated data sequence specifically includes: The total number of all data types in the multiple data sources is counted to obtain the corresponding total number of types, and the maximum data index N is set according to the total number of types. Initialize the first integrated data sequence according to the maximum scale index M and the maximum data index N; and in the initialized first integrated data sequence, process all the second data objects D. i This is denoted as the corresponding first invalid data object; wherein, the first integrated data sequence obtained after initialization includes the second data object D of the maximum scale index M. i ; each of the second data objects D obtained during initialization i Compared with the first time scale t i One-to-one correspondence; the second data object D obtained after initialization i Including a corresponding first time scale t i A second source type initialized to empty and a second data group with a fixed data length of N-2; the initialized second data group includes N-2 initialized empty second data d. i,j ; All the first data objects from the multiple data sources are sorted according to the chronological order of the first time to obtain the corresponding sequence of first data objects; The first data objects in the first data object sequence are traversed sequentially. During traversal, the currently traversed first data object is taken as the corresponding current first data object, the first source type of the first data source corresponding to the current first data object is taken as the corresponding current type, the first time of the current first data object is taken as the corresponding current data time, and the first time scale t on the first time axis corresponding to the current data time is taken as the current data time. i As the corresponding current time scale, and using the second data object D in the first integrated data sequence that corresponds to the current time scale, i As the corresponding current second data object; and to identify whether the current second data object is recorded as the first invalid data object; if the current second data object is recorded as the first invalid data object, then the second source type of the current second data object is set to the current type, and the second data d corresponding to the second data group of the current second data object is set according to one or more of the first data of the current first data object. i,j The settings are configured, and upon successful configuration, the current second data object is reclassified as the corresponding first valid data object. If the current second data object is not reclassified as the first invalid data object, the second source type of the current second data object is used as the corresponding first type. A type optimization process is performed on the first type and the current type according to a preset type optimization mode to obtain the corresponding first optimization result. When the first optimization result is the current type, the second source type and the second data group of the current second data object are cleared, and the second source type of the current second data object is set to the current type. The second data d in the second data group of the current second data object is then configured according to one or more of the first data from the current first data object. i,j Configure settings; When the traversal is complete, the first integrated data sequence with the data filling completed will be output as the corresponding data integration processing result.

5. The data processing method for multiple data sources according to claim 4, characterized in that, The step of performing type optimization processing on the first type and the current type according to a preset type optimization mode to obtain the corresponding first optimization result specifically includes: The preferred type mode is identified; the preferred type mode includes a first preferred mode, a second preferred mode, and a third preferred mode. When the type preference mode is the first preference mode, the current weather type is obtained as the corresponding first weather type; according to the first weather type, a preset data source level list is queried, and the first type priority sequence field of the first data source level record that matches the first weather type in the data source level list is extracted as the corresponding first type priority sequence; the sequence index of the third source type that matches the first type in the first type priority sequence is recorded as the corresponding first index, and the sequence index of the third source type that matches the current type is recorded as the corresponding current index; when the current index is less than the first index, the corresponding first preference result is set as the current type; when the current index is greater than the first index, the corresponding first preference result is set as the first type; the data source level list includes multiple first data source level records; the first data source level record includes the first weather type field and the first type priority sequence field; the first type priority sequence is composed of multiple third source types sorted; the third source types include lidar type, millimeter-wave radar type, camera type, target detection model type, and filter type; When the type preference mode is the second preference mode, for the second data object D in the first data object sequence whose second source type is the first type and which is recorded as the first valid data object, i The quantity is counted to obtain the corresponding second quantity; and the second data object D in the first data object sequence whose second source type is the current type and is recorded as the first valid data object is... i The quantities are counted to obtain a corresponding third quantity; when the third quantity is greater than the second quantity, the corresponding first preferred result is set as the current type; and when the third quantity is less than or equal to the second quantity, the corresponding first preferred result is set as the first type. When the type preference mode is the third preference mode, the second data object D, which is closest to the current second data object in the first data object sequence and is recorded as the first valid data object by a specified number H, is selected. i Form a corresponding second data object set; and for the second data object D in the second data object set whose second source type is the first type. i The quantity is statistically analyzed to obtain the corresponding fourth quantity; and the second data object D in the second data object set whose second source type is the current type is... i The number of items is counted to obtain the corresponding fifth number; when the fifth number is greater than the fourth number, the corresponding first preferred result is set as the current type; and when the fifth number is less than or equal to the fourth number, the corresponding first preferred result is set as the first type.

6. The data processing method for multiple data sources according to claim 2, characterized in that, The data correction process for the velocity type data in the first integrated data sequence specifically includes: In the first integrated data sequence, the second data d, whose data type is position coordinates, is... i,j The second data object D corresponding to the second data group that is not empty i Let V be the corresponding valid point object. r The index of the valid point object is r≥1; and it is composed of all the valid point objects V. r Sorting generates a corresponding sequence of valid point objects; The preset speed correction mode is identified; the speed correction mode includes a first correction mode and a second correction mode. When the velocity correction mode is the first correction mode, a linear correction method is used to correct each effective point object V in the effective point object sequence. r Perform speed correction; When the velocity correction mode is the second correction mode, a filtering correction method is used to correct each effective point object V in the effective point object sequence. r Perform speed correction.

7. The data processing method for multiple data sources according to claim 6, characterized in that, The linear correction method is used to adjust each valid point object V in the valid point object sequence. r Speed ​​correction includes: The first valid point object V r As the corresponding current valid point object; and the three nearest valid point objects V following the current valid point object in the valid point object sequence. r Let these be the corresponding last one, last two, and last three valid point objects; and based on the last one, last two, and last three valid point objects, apply the second data d, whose data type is speed, to the current valid point object. i,j Perform data correction processing; The second valid point object V r As the corresponding current valid point object; and the nearest valid point object V in the sequence of valid point objects preceding the current valid point object; r Let V be the corresponding previous valid point object; and let V be the two nearest valid point objects V after the current valid point object in the sequence of valid point objects. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous valid point object and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing; The third valid point object V r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the sequence of valid point objects. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous one and previous two valid point objects and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing; The fourth valid point object V r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the two nearest valid point objects V after the current valid point object in the sequence of valid point objects. r These are denoted as the corresponding next one and next two valid point objects; and based on the previous one and previous two valid point objects and the next one and next two valid point objects, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing; And so on, until the second-to-last valid point object V is... r As the corresponding current valid point object; and the two nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let V be the corresponding first and second valid point objects; and let V be the most recent valid point object V in the sequence of valid point objects after the current valid point object. r This is denoted as the corresponding next valid point object; and based on the previous and second previous valid point objects and the next valid point object, the second data d of data type speed in the current valid point object is... i,j Perform data correction processing; The last valid point object V r As the corresponding current valid point object; and the three nearest valid point objects V preceding the current valid point object in the valid point object sequence. r Let these be the corresponding first, second, and third valid point objects; and based on the first, second, and third valid point objects, assign the second data d, whose data type is speed, to the current valid point object. i,j Perform data correction processing.

8. The data processing method for multiple data sources according to claim 7, characterized in that, The second data d, which is of data type speed, in the current valid point object is determined based on the last one, the last two, and the last three valid point objects. i,j Data correction processing includes: The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, which is of data type position coordinate, from the last one, the last two, and the last three valid point objects. i,j Extract them as the corresponding next coordinate vector p1, next second coordinate vector p2, and next third coordinate vector p3; The time scale differences from the current valid point object to the next valid point object, from the next valid point object to the second valid point object, and from the second valid point object to the third valid point object are calculated to obtain the corresponding subsequent time difference Δt1, subsequent second time difference Δt2, and subsequent third time difference Δt3; and the longest time difference Δt is calculated from the time scale difference from the current valid point object to the third valid point object. max ; For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded; If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated. If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, the current effective point velocity vector v is estimated based on the current coordinate vector p0, the next coordinate vector p1, the next second coordinate vector p2, the next third coordinate vector p3, the next time difference Δt1, the next second time difference Δt2, and the next third time difference Δt3 to obtain the corresponding estimated velocity vector v. * , , , , ; and the second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

9. The data processing method for multiple data sources according to claim 7, characterized in that, The second data d, which is of data type speed, in the current valid point object is processed based on the previous valid point object and the next and second valid point objects. i,j Data correction processing includes: The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the next valid point object, and the next two valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first coordinate vector p1 and the second coordinate vector p2; The time scale differences from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the next two valid point objects are calculated to obtain the corresponding previous time difference Δt. -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the previous valid point object and the current valid point object and between the current valid point object and the next two valid point objects is taken as the corresponding longest time difference Δt. max ; For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded; If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated. If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The latter coordinate vector p1, the latter second coordinate vector p2, and the former time difference Δt -1 The subsequent time difference Δt1 and the subsequent second time difference Δt2 are used to estimate the velocity vector of the current effective point to obtain the corresponding estimated velocity vector v. * , , , , ; and the second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

10. The data processing method for multiple data sources according to claim 7, characterized in that, The second data d, which is of data type speed, in the current valid point object is processed according to the previous and second valid point objects and the next and second valid point objects. i,j Data correction processing includes: The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, the next valid point object, and the next two valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 The first coordinate vector p1 and the second coordinate vector p2; The time scale differences from the two previous valid point objects to the previous valid point object, from the previous valid point object to the current valid point object, from the current valid point object to the next valid point object, and from the next valid point object to the two subsequent valid point objects are calculated to obtain the corresponding first two time differences Δt. -2 Previous time difference △t -1 The maximum time difference is defined as Δt1 and Δt2, and the largest of the time scale differences between the two preceding valid point objects and the current valid point object and the current valid point object and the two subsequent valid point objects is taken as the corresponding longest time difference Δt. max ; For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded; If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated. If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The latter coordinate vector p1, the latter second coordinate vector p2, and the former two time differences Δt -2 The preceding time difference Δt -1 The subsequent time difference Δt1 and the subsequent second time difference Δt2 are used to estimate the velocity vector of the current effective point to obtain the corresponding estimated velocity vector v. * , , , , , ; and the second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

11. The data processing method for multiple data sources according to claim 7, characterized in that, The second data d, which is of data type speed, in the current valid point object is processed based on the previous, previous two, and next valid point objects. i,j Data correction processing includes: The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, and the next valid point object. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the next coordinate vector p1; The time scale differences from the two previous valid point objects to the previous valid point object, from the previous valid point object to the current valid point object, and from the current valid point object to the next valid point object are calculated to obtain the corresponding first two time differences Δt. -2 Previous time difference △t -1 The maximum time difference Δt1 is taken as the corresponding longest time difference Δt, which is the time difference between the two previous valid point objects and the current valid point object and the time difference between the current valid point object and the next valid point object. max ; For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded; If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated. If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The latter coordinate vector p1, and the former two time differences Δt -2 The preceding time difference Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector of the current effective point using the time difference Δt1. * , , , , ; and the second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

12. The data processing method for multiple data sources according to claim 7, characterized in that, The second data d, which is of data type speed, in the current valid point object is determined based on the previous one, two, and three valid point objects. i,j Data correction processing includes: The second data d, which is of data type position coordinate in the current valid point object. i,j Extract it as the corresponding current coordinate vector p0; and take the second data d, whose data type is position coordinate, from the previous valid point object, the previous two valid point objects, and the previous three valid point objects. i,j Extract it as the corresponding previous coordinate vector p -1 The first two coordinate vectors p -2 and the first three coordinate vectors p -3 ; The time scale differences from the first three valid point objects to the first two valid point objects, from the first two valid point objects to the previous valid point object, and from the previous valid point object to the current valid point object are calculated to obtain the corresponding first three time differences Δt. -3 The first two time differences △t -2 Time difference with the previous time Δt -1 The time scale difference between the first three valid point objects and the current valid point object is taken as the corresponding longest time difference Δt. max ; For the longest time difference Δt max Confirm whether the preset first interval threshold has been exceeded; If the longest time difference Δt is confirmed max If the first interval threshold is exceeded, the current data correction process will be terminated. If the longest time difference Δt is confirmed max If the first interval threshold is not exceeded, then based on the current coordinate vector p0 and the previous coordinate vector p0... -1 The first two coordinate vectors p -2 The first three coordinate vectors p -3 The first three time differences △t -3 The first two time differences Δt -2 and the previous time difference Δt -1 The estimated velocity vector v is obtained by estimating the velocity vector at the current effective point. * , , , , ; and the second data d with data type speed in the current valid point object. i,j Extract it as the corresponding current velocity vector v0; and compare the current velocity vector v0 with the estimated velocity vector v * The system identifies whether the vector difference meets the preset vector difference range; if yes, it exits the current data correction process; if no, it converts the second data d of the current valid point object, which is of data type velocity, into the corrected data. i,j Reset to the corresponding estimated velocity vector v * .

13. The data processing method for multiple data sources according to claim 6, characterized in that, The filtering correction method is used to process each valid point object V in the valid point object sequence. r Perform speed correction, specifically including The corresponding Kalman filter observation-state equation set is created based on the preset motion model and is denoted as the first observation-state equation set; the first observation-state equation set uses position coordinates as the observation and velocity as the state variable; From the second valid point object V in the sequence of valid point objects r The process begins by iterating through the valid point objects sequentially; during the iteration, the currently iterated valid point object V is... r As the corresponding current valid point object; and set the data type of the current valid point object to the second data d of the position coordinates. i,j As the observation at the current moment, the previous valid point object V of the current valid point object. r The data type is the second data d, which is the speed. i,j The observations at the current time and the state quantities at the previous time are used as the state quantities at the previous time; and the observations at the current time and the state quantities at the previous time are substituted into the first observation-state equation set to predict the state quantities at the current time to obtain the corresponding first predicted quantity v. p ; and set the data type of the current valid point object to the second data d of velocity. i,j Extracted as the corresponding first velocity vector v ’ ; and for the first velocity vector v ’ With the first prediction v p The system identifies whether the vector difference meets the preset vector difference range; if so, it proceeds to the next valid point object V. r Continue the traversal; otherwise, change the data type of the current valid point object to the second data d of velocity. i,j Reset to the first prediction value v p .

14. The data processing method for multiple data sources according to claim 2, characterized in that, The step of identifying the mask data sequences of various data sources based on the first integrated data sequence to obtain corresponding multi-class mask data sequences specifically includes: The number of the first data sources of the multiple types of data sources is counted to obtain the corresponding number of the first data sources; The length of the first mask data sequence is initialized to the maximum scale index M; the initialized first mask data sequence includes the first mask data Mask of the maximum scale index M. i Each of the first mask data Mask i The mask value is initialized to 0; the first mask data sequence corresponds one-to-one with the first data source; Each of the first data sources is traversed; during traversal, the current first data source is taken as the corresponding current data source, the first source type of the current data source is taken as the corresponding current source type, and the first mask data sequence corresponding to the current data source is taken as the corresponding current mask data sequence; and the second data object D in the first integrated data sequence whose second source type matches the current source type is taken as the current data sequence. i All are marked as the corresponding current second data object; and the first time scale t of each of the current second data objects is set. i The scale index i is extracted as the corresponding first scale index; and the first mask data Mask corresponding to each of the first scale indices in the current mask data sequence is used. i The mask value is reset to 1; At the end of the traversal, the corresponding multi-class mask data sequence is output, which is composed of all the first mask data sequences.

15. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-14; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-14.

Images