A method for labeling off-road ground natural driving dataset considering multi-dimensional characteristics

Abstract

The application provides a kind of off-road ground natural driving data set labeling method considering multidimensional characteristics, defines data label in combination with vehicle external perception features and ontology perception features, realizes the many-to-many mapping of ground image material category and ground physical feature category, so that off-road ground natural driving multi-source heterogeneous data set labeling is more dimensional, information is more perfect.The application of the off-road ground natural driving multi-source heterogeneous data set labeling method is helpful to realize the parameterization description of the physical features of different off-road ground categories, enhance the off-road ground environment cognition ability of intelligent off-road vehicle, and provide more valuable ground physical feature information for motion planning and control of intelligent off-road vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of off-road ground natural driving data set labeling method, especially to a kind of off-road ground natural driving data set labeling method considering multidimensional characteristics. BACKGROUND

[0002] With the development of vehicle intelligent auxiliary driving, the application of driving assistance technology has not been limited to urban roads, and can also help drivers improve the traffic capacity and efficiency of vehicles in off-road driving scenarios such as field rescue, material transportation, mining operations and agricultural tasks. The complex and changeable characteristics of off-road scenarios put higher requirements on the terrain perception ability of vehicles, and the improvement of vehicle perception ability is highly dependent on the quality of raw data. In related technologies, the ways of vehicle perception of ground type include external perception (i.e. image and point cloud) and intrinsic perception (i.e. vehicle dynamics response). However, external perception features and intrinsic perception features cannot be mapped one-to-one, and in current ground class labeling of data sets, the terrain is usually divided into several categories according to image material characteristics, lacking consideration of ground physical characteristics and geometric characteristics, which restricts the development of vehicle planning and control algorithms that rely on ground physical characteristics and vehicle-road interaction characteristics.

[0003] Therefore, it is necessary to establish an off-road ground natural driving data set labeling method considering multidimensional characteristics, which defines data labels by combining vehicle external perception features and intrinsic perception features, to provide more clear and accurate environment cognition for vehicle planning and control algorithms. SUMMARY

[0004] To solve the above technical problems, the present application provides an off-road ground natural driving data set labeling method considering multidimensional characteristics, comprising the following steps:

[0005] S10, obtaining off-road ground natural driving multi-source heterogeneous data set D multi The off-road ground natural driving multi-source heterogeneous data set contains at least two or more data sources, and the collected data has at least two or more different data formats, and the multi-source heterogeneous data has time synchronization basis.

[0006] The off-road ground natural driving multi-source heterogeneous data set D multi contains at least intrinsic perception signal D proprio And external perception signal D tero ;

[0007] The intrinsic perception signal D proprio contains at least driver manipulation behavior data, vehicle driving state data and vehicle-wheel dynamics response data inside the vehicle CAN bus;

[0008] The external perception signal Dtero It includes at least off-road terrain image data acquired by image acquisition equipment installed on the vehicle.

[0009] Preferably, the body perception signal D proprio It also includes a combined inertial navigation signal installed on the vehicle; the external sensing signal D tero It also includes point cloud data collected by binocular cameras or lidar installed on the vehicle.

[0010] The driver operation behavior data includes at least signals of steering wheel angle, steering angular velocity, braking pressure, engine / drive motor output torque, and engine / drive motor output speed; the vehicle driving status data includes at least signals of vehicle speed, vehicle longitudinal acceleration, vehicle lateral acceleration, wheel slip ratio control function, and vehicle stability control function; the vehicle-wheel dynamics response data includes at least signals of wheel speed of each wheel and vehicle suspension travel; the off-road ground image data is a ground image of the vehicle being driven or about to be driven by, acquired by an image acquisition device facing the ground or in front of the vehicle.

[0011] The aforementioned multi-source heterogeneous dataset D of natural off-road driving multi The dataset is divided into several continuous segments based on collection time and space, representing a multi-source heterogeneous dataset D of natural off-road driving. multi,cont The continuous off-road terrain natural driving multi-source heterogeneous dataset D multi,cont It involves collecting data over continuous time and space.

[0012] S20. Mileage labeling of multi-source heterogeneous data from natural off-road driving: Based on vehicle driving status information and image acquisition equipment installation location information, the mileage relative to the acquisition starting point at the time of body perception signal acquisition and the mileage range of the off-road surface area where external perception signals can be acquired relative to the acquisition starting point are estimated. The steps are as follows:

[0013] S21. For each continuous segment of the multi-source heterogeneous dataset D of natural driving on off-road terrain. multi,cont Starting from the data collection point, a sampling period of Δt is established. mile The time span of the mileage frame sequence D is the same as that of the continuous dataset. mile ={F mile ,TS mile}, TS mile The timestamp is used to represent the mileage frame. Based on the vehicle's driving status information and the timestamp, the mileage information F of the vehicle's location relative to the collection start point at the time of each mileage frame acquisition is estimated. mile The timestamp sequence of the mileage frames is the same as the timestamp sequence of the off-road terrain images.

[0014] S22, according to the timestamp TS of the body perception signal proprio , the mileage mark Tag of all body perception signals mile,proprio is the latest last mileage frame acquisition time ts mile,before The mileage information f of the vehicle location relative to the starting point of the collection mile,before ;

[0015] S23, according to the timestamp TS of the external perception signal tero , the mileage mark Tag of all external perception signals mile，tero is the latest last mileage frame acquisition time ts mile,before The mileage information f of the vehicle location relative to the starting point of the collection mile,before ;

[0016] S24, according to the installation position of the image acquisition device to estimate the ground longitudinal relative mileage range in the off-road ground image, which represents the nearest and farthest distance of the ground area visible in the off-road ground image in the vehicle forward direction relative to the current vehicle position;

[0017] S25, according to the mileage mark reference Tag of the off-road ground image mile,tero,base , the ground longitudinal relative mileage range (d min , d max ) in the off-road ground image, the mileage mark Tag of the off-road ground image mile,tero is the mileage range of the off-road ground image relative to the starting point of the collection.

[0018] S30, ground image material category labeling of off-road ground natural driving multi-source heterogeneous data, extracting uniform ground image material category off-road ground natural driving multi-source heterogeneous data segment D multi,uniqmatl :

[0019] S31, for each continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont , label the ground image material category of the off-road ground image in units of pixels; the ground image material category Y matl represents the conclusion obtained by visually judging the ground image according to the material characteristics;

[0020] S32, extract the continuous frame off-road ground image containing unique ground image material category and continuously in the same ground image material category, and the body perception signal contained in the corresponding mileage range of the mileage mark of the continuous frame off-road ground image

[0021] S33, label the ground image material category of each frame of body perception signal obtained in step S32 as the corresponding ground image material category, and form an off-road ground natural driving multi-source heterogeneous data paragraph D of uniform ground image material category multi,uniqmatl = {D proprio tero , TS proprio , TS tero , Tag mile,proprio , Tag mile,tero , Y matl uniqmatl .

[0022] S40, time-frequency domain feature extraction and first vehicle-wheel-road interaction feature set D interact construction:

[0023] S41, for each off-road ground natural driving multi-source heterogeneous data paragraph D of uniform ground image material category multi,uniqmatl , vehicle-wheel-road interaction state information estimation is carried out according to the body perception signal; and continuous vehicle-wheel-road interaction state data paragraphs are formed respectively;

[0024] S42, each continuous vehicle-wheel-road interaction state data paragraph is divided into a plurality of continuous data paragraphs with time length T, the mutual overlap length of which does not exceed 70%, and N vehicle-wheel-road interaction state samples P are formed in total interact = {p interact,1 , p interact,2 ,..., p interact,N};

[0025] S43, time domain feature and frequency domain feature of each vehicle-wheel-road interaction state sample are calculated, each vehicle-wheel-road interaction state sample p interact,i corresponds to a group of vehicle-wheel-road interaction features x i = (x i1 , x i2 ,...x in ), n is the number of features, and y matl,i corresponds to a unique ground image material category label;

[0026] S44, all vehicle-wheel-road interaction state samples contained in the off-road ground natural driving multi-source heterogeneous data set in step S10 and the corresponding ground image material category label form a first vehicle-wheel-road interaction feature set:

[0027]

[0028] Further, the vehicle-wheel-road interaction state information in step S41 includes any other information expressing the vehicle-wheel-road interaction state, including driving resistance and wheel speed noise: ​​

[0029] (1) The calculation formula of the driving resistance is:

[0030] F f = F t -F w -F i -F j

[0031] Wherein:

[0032] F t is the driving force, F w is the air resistance generated by the air acting on the vehicle body, F i is the slope resistance when the vehicle is running on a slope, F j is the acceleration resistance generated to overcome inertia when accelerating;

[0033] (2) Wheel speed noise:

[0034] Use a high-pass filter to extract the noise component in the wheel speed signal, and the left front wheel, right front wheel, left rear wheel and right rear wheel are respectively: v fl,noise , v fr,noise , v rl,noise , v rr,noise .

[0035] Further, the time domain feature and frequency domain feature calculation method of step S43 is as follows:

[0036] (1) Time domain feature

[0037] For one of the vehicle-wheel-road interaction state information S of a vehicle-wheel-road interaction state sample, the time domain feature includes but is not limited to:

[0038] (a) Mean value:

[0039] (b) Peak-peak value: peak2peak(S) = max(S)-min(S)

[0040] (c) Variance:

[0041] (d) Root mean square:

[0042] (2) Frequency domain feature

[0043] Perform a fast Fourier transform on the vehicle-wheel-road interaction state information S to obtain the frequency spectrum S(f) of the vehicle-wheel-road interaction state information. The frequency domain feature includes but is not limited to:

[0044] (a) Center of gravity frequency:

[0045] (b) Mean square frequency:

[0046] (c) Root mean square frequency: RMSF = MSF

[0047] S50, the mapping relationship between vehicle-wheel-road interaction features and ground physical feature categories is constructed, based on the first vehicle-wheel-road interaction feature set D. interact The main vehicle-wheel-road interaction features in the first vehicle-wheel-road interaction feature set are obtained through feature analysis methods, forming the second vehicle-wheel-road interaction feature set D′. interact The second vehicle-wheel-road interaction feature set D′ was analyzed using a clustering algorithm. interact The sample distribution in the dataset is analyzed to obtain n. phy There are 10 clusters, and each cluster generates a ground physical feature category label Y. phy The steps include:

[0048] S51, based on the first vehicle-wheel-road interaction feature set D interact Feature filtering is performed on the first vehicle-wheel-road interaction feature set, and the main vehicle-wheel-road interaction features x′=(x′) are extracted based on the dependency relationships between the vehicle-wheel-road interaction features. ·1 ,x′ ·2 ,...x′ ·m ), where m is the number of extracted features, forming the second vehicle-wheel-road interaction feature set:

[0049]

[0050] S52, For the second vehicle-wheel-road interaction feature set, those with the same ground image material category y matl Samples of type i are clustered to form several clusters;

[0051] S53. Perform pairwise similarity evaluation on clusters with different ground image material category labels, and merge clusters with high similarity.

[0052] S54. Regarding the n formed after the merger... phy Clusters Generate ground physical feature category labels Y one by one phy .

[0053] S60. Label the ground physical feature categories of multi-source heterogeneous data on natural off-road driving:

[0054] S61. According to step S50, the second vehicle-wheel-road interaction feature set D′ interact Each sample (x′) i ,y matl,i The corresponding original vehicle-wheel-road interaction state sample p interact,iAlso labeled as the same ground physical feature category;

[0055] S62, the vehicle-wheel-road interaction state sample p interact,i The off-road ground natural driving multi-source heterogeneous data segment is mapped back to the uniform ground image material category described in steps S30 and S40.

[0056] S63, the ground physical feature category label of each frame of body perception signal is determined by the ground physical feature category of the several vehicle-wheel-road interaction state samples {..., p interact,j-1 ,p interact,j ,p interact,j+1 ,...} vote. phy,j-1 ,y phy,j ,y phy,j+1 ,...} vote.

[0057] S70, the ground image material category Y matl and the ground physical feature category Y phy Probability mapping relationship:

[0058] According to the clustering result obtained in step S50, the ground physical feature category to which the sample contained in each ground image material category belongs is inductively counted, and a ground image material category and ground physical feature category probability mapping relationship table is formed.

[0059] S80, construction of the mapping relationship between the off-road ground physical feature category and the mechanical / geometric feature:

[0060] S81, for each vehicle-wheel-road interaction state sample p interact,i , the mechanical feature mc and the geometric feature gc of the ground are estimated, the mechanical feature includes but is not limited to the ground adhesion coefficient, and the geometric feature includes but is not limited to the ground unevenness;

[0061] S82, for each ground physical feature category Y phy The mechanical feature and the geometric feature estimated by the corresponding vehicle-wheel-road interaction state sample are fitted to a Gaussian distribution, and the probability distribution mapping mc phy,i ~ N (μ mc,i , σ mc,i ), gc phy,i ~ N (μ gc,i , σ gc,i ) of each ground physical feature category and the mechanical / geometric feature is obtained.

[0062] The beneficial effects of the method of the present application are:

[0063] The off-road ground natural driving data set labeling method considering multi-dimensional characteristics provided by the application, compared with the prior art, fully considers the problem that the external perception characteristics and the intrinsic perception characteristics of the vehicle on the off-road ground cannot be one-to-one corresponding, provides a ground physical feature category labeling method, realizes the many-to-many mapping of the ground image material category and the ground physical feature category, and makes the off-road ground natural driving multi-source heterogeneous data set labeling more dimensional and more perfect in information. Application of the off-road ground natural driving multi-source heterogeneous data set labeling method of the application helps to realize the parameterized description of the physical features of different off-road ground categories, enhances the off-road ground environment cognition ability of the intelligent off-road vehicle, and provides more valuable ground physical feature information for the motion planning and control of the intelligent off-road vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 It is a schematic diagram of the whole process of the method of the application.

[0065] Figure 2 It is a schematic diagram of the off-road ground natural driving multi-source heterogeneous data mileage labeling process of the application.

[0066] Figure 3 It is a schematic diagram of the off-road ground natural driving multi-source heterogeneous data mileage labeling of the application.

[0067] Figure 4 It is a schematic diagram of the off-road ground natural driving multi-source heterogeneous data paragraph extraction process of the application.

[0068] Figure 5 It is a schematic diagram of the off-road ground natural driving multi-source heterogeneous data paragraph extraction of the application.

[0069] Figure 6 It is a schematic diagram of the first vehicle-wheel-road interaction feature set construction of the application.

[0070] Figure 7 It is a schematic diagram of the first vehicle-wheel-road interaction feature set construction of the application.

[0071] Figure 8 It is a schematic diagram of the vehicle-wheel-road interaction feature and ground physical feature category mapping relationship construction of the application.

[0072] Figure 9 It is a schematic diagram of the ground image material category and ground physical feature category probability mapping relationship of the application.

[0073] Figure 10 It is a schematic diagram of the relationship between the data sets and feature sets at each level of the application. DETAILED DESCRIPTION

[0074] Reference is made to the drawingsFigure 1 As shown, the application provides a cross-country ground natural driving data set labeling method considering multi-dimensional characteristics, comprising the following steps:

[0075] S10, obtaining a cross-country ground natural driving multi-source heterogeneous data set:

[0076] The cross-country ground natural driving multi-source heterogeneous data set D multi is at least composed of the body perception signal D proprio and the external perception signal D tero . multi D proprio = {{D proprio ,TS tero},{D tero ,TS proprio}}, TS tero is the timestamp of the body perception signal, and TS proprio is the timestamp of the external perception signal.

[0077] The body perception signal D proprio at least contains driver manipulation behavior data, vehicle driving state data, and vehicle-wheel dynamics response data inside the vehicle CAN bus, and preferably also contains combined inertial navigation signals installed on the vehicle.

[0078] The external perception signal D tero at least contains off-road ground image data collected by image acquisition equipment installed on the vehicle, and preferably also contains point cloud data collected by binocular cameras or laser radars installed on the vehicle.

[0079] The driver manipulation behavior at least contains signals such as steering wheel angle, steering angle speed, brake pressure, engine / drive motor output torque, and engine / drive motor output speed to express the driver's driving operation; the vehicle driving state data at least contains signals such as vehicle driving speed, vehicle body longitudinal acceleration, vehicle body lateral acceleration, wheel slip rate control function, and vehicle body stability control function; the vehicle-wheel dynamics response data at least contains signals such as vehicle wheel speed and vehicle suspension movement stroke; and the off-road ground image data is the image of the ground on which the vehicle is driving or will drive collected by the image acquisition equipment facing the ground or the front of the vehicle.

[0080] The multi-source heterogeneous data set at least contains two or more data sources, the collected data has at least two or more different data formats, and the multi-source heterogeneous data has a time synchronization basis, generally, the time synchronization basis is the timestamp.

[0081] Generally, the cross-country ground natural driving multi-source heterogeneous data set D multiThe continuous off-road ground natural driving multi-source heterogeneous data set D can be divided into several segments according to the collection time and space multi,cont , the continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont is collected continuously in time and space.

[0082] S20, off-road ground natural driving multi-source heterogeneous data mileage marking, according to the vehicle driving state information and the installation position information of the image acquisition device, the relative collection starting point of the body perception signal collection time is estimated respectively. The mileage and the off-road ground area relative to the collection starting point of the external perception signal can be collected. Referring to Figure 2 , Figure 3 , the steps are as follows:

[0083] S21, for each segment of the continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont , from the collection starting point, a mileage frame sequence D mile with the same time span and continuous data set is established, the sampling period is Δt mile ={F mile , TS mile}, TS mile is the mileage frame timestamp; according to the vehicle driving state information and the mileage frame timestamp, the mileage information F mile of the vehicle at the collection time of each mileage frame relative to the collection starting point is estimated; preferably, the timestamp sequence of the mileage frame is the same as the timestamp sequence of the off-road ground image;

[0084] S22, according to the timestamp TS proprio of the body perception signal, the mileage marking Tag mile,proprio of all body perception signals is the mileage information f mile,before of the vehicle at the collection time of the nearest last mileage frame ts mile,before relative to the collection starting point;

[0085] S23, according to the timestamp TS tero of the external perception signal, the mileage marking reference of all external perception signals is the mileage information f mile,before of the vehicle at the collection time of the nearest last mileage frame ts mile,before relative to the collection starting point;

[0086] S24, according to the installation position of the image acquisition device, the ground longitudinal relative mileage range in the off-road ground image is estimated, which represents the nearest and farthest distance of the visible ground area in the off-road ground image in the vehicle forward direction relative to the current position of the vehicle. Preferably, the ground longitudinal relative mileage range in the off-road ground image can also be estimated in combination with the vehicle body posture;

[0087] S25, mileage marking reference Tag according to the off-road ground image mile,tero,base , ground longitudinal relative mileage range (d min , d max ) in the off-road ground image mile,tero is the mileage range relative to the starting point of collection in the off-road ground image.

[0088] S30, ground image material category marking of off-road ground natural driving multi-source heterogeneous data, and extraction of off-road ground natural driving multi-source heterogeneous data paragraph D of uniform ground image material category multi,uniqmatl . Referring to Figure 4 , Figure 5 , the steps are as follows:

[0089] S31, each continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont , the ground image material category is marked in units of pixels for the off-road ground image; the ground image material category Y matl represents the conclusion obtained by judging the ground category of the ground image according to the characteristics of color, texture and the like from the visual aspect;

[0090] S32, extraction of the continuous frame off-road ground image containing unique ground image material category and continuously in the same ground image material category, and the body perception signal contained in the corresponding mileage range of the mileage marking of the continuous frame off-road ground image;

[0091] S33, marking the ground image material category of each frame body perception signal obtained in step S32 as the corresponding ground image material category, forming off-road ground natural driving multi-source heterogeneous data paragraph D multi,uniqmatl ={D proprio ,D tero ,TS proprio ,TS tero ,Tag mile,proprio ,Tag mile,tero ,Y matl} uniqmatl ; generally, each continuous off-road ground natural driving multi-source heterogeneous data set can form several off-road ground natural driving multi-source heterogeneous data paragraphs of uniform ground image material category.

[0092] S40, time-frequency domain feature extraction and first vehicle-wheel-road interaction feature set D interact construction. Referring to Figure 6 , Figure 7 , the steps are as follows:

[0093] S41, the off-road ground natural driving multi-source heterogeneous data segment D of each uniform ground image material category multi,uniqmatl According to the body perception signal, vehicle-wheel-road interaction state information estimation is performed to form continuous vehicle-wheel-road interaction state data segments;

[0094] S42, each continuous vehicle-wheel-road interaction state data segment is divided into several time lengths T, and the continuous data segments with a mutual overlap length of not more than 70% are formed, and N vehicle-wheel-road interaction state samples, i.e., P interact interact,1 interact,2 interact,N} are formed;

[0095] S43, the time domain features and frequency domain features of each vehicle-wheel-road interaction state sample are calculated, each vehicle-wheel-road interaction state sample p interact,i corresponds to a group of vehicle-wheel-road interaction features x i = (x i1 , x i2 ,... x in ), n is the number of features, and corresponds to a unique ground image material category label y matl,i ;

[0096] S44, all vehicle-wheel-road interaction state samples and corresponding ground image material category labels contained in the off-road ground natural driving multi-source heterogeneous data set in step S10 form a first vehicle-wheel-road interaction feature set:

[0097]

[0098] Further, the vehicle-wheel-road interaction state information in step S41 includes driving resistance, wheel speed noise and other any information expressing vehicle-wheel-road interaction state:

[0099] (1) The calculation formula of driving resistance is:

[0100] F f = F t -F w -F i -F j

[0101] Wherein:

[0102] F t is the driving force, and for a vehicle with a front-rear dual motor driving configuration, the calculation formula is T M1 , T M2 are the output torques of the front and rear driving motors, i 0f , i 0r ​​​These are the transmission ratios of the front and rear main reducers, r. f r r Let η be the radius of the front and rear wheels, respectively. T For mechanical efficiency;

[0103] F w The air resistance generated by air acting on the car body is calculated using the following formula: C D Let A be the drag coefficient of the car, A be the size of the car's frontal projection area, and u be the vehicle's speed.

[0104] F i F is the gradient resistance of a vehicle traveling on a slope. j To overcome the acceleration drag caused by inertia during acceleration, taking a vehicle with a front-rear dual-motor drive configuration as an example, the calculation formula is as follows: M is the mass of the entire vehicle, a x This is the longitudinal acceleration signal output by the vehicle. These represent the output speeds of the front and rear drive motors, I. M1 I M2 These are the rotor inertia of the front and rear drive motors, respectively. The angular accelerations I of the front and rear wheels are respectively. whl,f I whl,r These are the moments of inertia of the front and rear wheels, respectively.

[0105] (2) Wheel speed noise:

[0106] A high-pass filter was used to extract the noise component from the wheel speed signal. The left front wheel, right front wheel, left rear wheel, and right rear wheel speeds were respectively: v fl,noise v fr,noise v rl,noise v rr,noise .

[0107] Furthermore, the calculation methods for the time-domain features and frequency-domain features described in step S43 are as follows:

[0108] (1) Temporal characteristics

[0109] For one vehicle-wheel-road interaction state information S in a vehicle-wheel-road interaction state sample, the temporal features include, but are not limited to:

[0110] (a) Average value:

[0111] (b) Peak-to-peak value: peak2peak(S) = max(S) - min(S)

[0112] (c) Variance:

[0113] (d) root mean square:

[0114] (2) Frequency domain features

[0115] The vehicle-wheel-road interaction state information S is subjected to a fast Fourier transform to obtain a frequency spectrum S(f) of the vehicle-wheel-road interaction state information. The frequency domain features include but are not limited to:

[0116] (a) Gravity frequency:

[0117] (b) Mean square frequency:

[0118] (c) Root mean square frequency: RMSF = MSF

[0119] S50, vehicle-wheel-road interaction feature and ground physical feature category mapping relationship construction, according to the first vehicle-wheel-road interaction feature set D interact , the main vehicle-wheel-road interaction features in the first vehicle-wheel-road interaction feature set are obtained through feature analysis method, the second vehicle-wheel-road interaction feature set D' interact , the distribution of the sample set in the second vehicle-wheel-road interaction feature set D' interact is analyzed through clustering algorithm, n phy clusters are obtained, and each cluster corresponds to a ground physical feature category label Y phy one by one. Referring to FIG. 8, the steps are as follows: Figures 8-10

[0120] S51, according to the first vehicle-wheel-road interaction feature set D interact , the main vehicle-wheel-road interaction features x' = (x' ·1 , x' ·2 ,... x' ·m ) are extracted according to the dependency relationship between the vehicle-wheel-road interaction features, m is the number of extracted features, and the second vehicle-wheel-road interaction feature set is formed:

[0121]

[0122] S52, the samples in the second vehicle-wheel-road interaction feature set with the same ground image material category y matl = i are clustered to form a plurality of clustering clusters;

[0123] S53, similarity evaluation is performed on the clustering clusters with different ground image material category labels two by two, and the clustering clusters with high similarity are merged;

[0124] S54, n phy clusters are formed after the merging is completed ​Generate ground physical feature category labels Y one by one phy .

[0125] In this embodiment, the feature extraction method described in step S51 employs factor analysis, and its steps are as follows:

[0126] (1) Calculation Mean and standard deviation μ of the first n columns j ,σ j For the first vehicle-wheel-road interaction feature set D, j = 1, 2, ..., n, j = 1, 2, ..., n. interact Standardized processing;

[0127] (2) Calculate the sample correlation matrix R;

[0128] (3) Calculate the eigenvalues ​​and eigenvectors of the correlation matrix;

[0129] (4) The number of principal factors m is determined based on the cumulative contribution rate;

[0130] (5) Calculate the relevant factor loading matrix;

[0131] (6) Determine the factor analysis model;

[0132] (7) The factor analysis model groups features according to their correlation, placing features with high correlation into the same group and features with low correlation into different groups;

[0133] (8) Based on the number of principal factors m determined in step (4), select the feature with the highest correlation to the principal factor from the feature groups corresponding to each principal factor as the extracted feature.

[0134] In this embodiment, the clustering method in step S52 is k-means clustering, and its steps are as follows:

[0135] (1) Initialization: For the second vehicle-wheel-road interaction feature set, select the ground image material category y with the same type. matl = sample subset of i Randomly select k sample points as the initial cluster centers.

[0136] (2) Cluster the samples: calculate the clustering of the samples respectively. Each second vehicle-wheel-road interaction feature sample x′ j =(x′) j1 ,x′ j2 ,...x′ jm To the cluster center The squared Euclidean distance d is used to determine the clustering result C based on the principle of minimizing the distance. (t) ;

[0137] (3) Update cluster center: for the clustering result C (t) , compute the sample mean within the cluster and take it as the new cluster center and continuously update according to the following objective function:

[0138]

[0139] (4) If the iteration converges or the loop termination condition is met, output the calculation result, otherwise repeat steps (2) and (3).

[0140] In this embodiment, the similarity evaluation method of the cluster in step S53 adopts the Calinski-Harabasz index (CH). For two evaluated clusters C i , C j and cluster centers m i , m j , the steps are as follows:

[0141] (1) Calculate compactness:

[0142]

[0143] N i , N j are the number of samples contained in the clusters C i , C j ;

[0144] (2) Calculate separation:

[0145] N i ||m i -m|| 2 +N j ||m j -m|| 2

[0146] m is the center of all samples contained in the clusters C i , C j , that is, the sample mean;

[0147] (3) Calculate the Calinski-Harabasz index:

[0148]

[0149] S60, ground physical feature category labeling of off-road ground natural driving multi-source heterogeneous data. Referring to Figure 7 , Figure 10 , the steps are as follows:

[0150] S61、According to step S50, the second vehicle-wheel-road interaction feature set D' interact Each sample (x' i ,y matl,i ) in the set corresponds to an original vehicle-wheel-road interaction state sample p interact,i , which is also labeled as the same ground physical feature category;

[0151] S62、Map the vehicle-wheel-road interaction state sample p interact,i Back to the off-road ground natural driving multi-source heterogeneous data paragraph of the uniform ground image material category described in steps S30 and S40;

[0152] S63、The ground physical feature category label of each frame of body perception signal is determined by the ground physical feature category {..., y phy,j-1 ,y phy,j ,y phy,j+1 ,...} voted by the ground physical feature category {..., y phy,j-1 ,y phy,j ,y phy,j+1 ,...} of the several vehicle-wheel-road interaction state samples {..., p interact,j-1 ,p interact,j ,p interact,j+1 ,...}.

[0153] S70、Construct the ground image material category Y matl and the ground physical feature category Y phy probability mapping relationship. Referring to Figure 9 , the steps are as follows:

[0154] According to the clustering results obtained in step S50, the ground physical feature categories to which the samples contained in each ground image material category belong are summarized and counted to form a ground image material category and ground physical feature category probability mapping relationship table.

[0155] S80、Mapping relationship construction between off-road ground physical feature categories and mechanical / geometric features. The steps are as follows:

[0156] S81、For each vehicle-wheel-road interaction state sample p interact,i , estimate the mechanical feature mc and the geometric feature gc of the ground, wherein the mechanical feature includes but is not limited to the ground adhesion coefficient, and the geometric feature includes but is not limited to the ground unevenness;

[0157] S82、For each ground physical feature category Y phy , the mechanical feature and the geometric feature estimated from the corresponding vehicle-wheel-road interaction state sample are fitted to a Gaussian distribution to obtain the probability distribution mapping mc phy,i ~ N(μ mc,i ,σ mc,i ), gc phy,i ~ N(μ gc,i ,σgc,i ).

[0158] Further, the Gaussian distribution is fitted in step S82, and the mechanical characteristics mc phy,i are taken as an example, the steps are:

[0159] (1) The mean value μ mc,i and the variance σ mc,i of the mechanical characteristics or geometric characteristics of the vehicle-wheel-road interaction state sample to be fitted are calculated.

[0160] (2) The Gaussian distribution mc phy,i ~ N(μ mc,i , σ mc,i ) is fitted.

[0161] The above is a specific description of the preferred embodiment of the present application, and the estimated vehicle-wheel-road interaction state information and the adopted time / frequency domain characteristics, the main feature extraction method, the clustering method and the clustering cluster similarity evaluation method include but are not limited to the specific scheme adopted.

Claims

1. A method for off-road ground natural driving dataset annotation considering multi-dimensional characteristics, characterized in that: The method comprises the following steps: S10, obtaining off-road ground natural driving multi-source heterogeneous dataset D multi ; The off-road ground natural running multi-source heterogeneous data set contains at least two or more data sources, the collected data has at least two or more different data formats, and the multi-source heterogeneous data has time synchronization basis; the off-road ground natural running multi-source heterogeneous data set D multi contains at least the body perception signal D proprio and the external perception signal D tero ; The body perception signal D proprio at least contains driver manipulation behavior data, vehicle running state data and vehicle-wheel dynamics response data inside the vehicle CAN bus; The external perception signal D tero at least contains off-road ground image data collected by an image collection device mounted on the vehicle; The off-road ground natural driving multi-source heterogeneous data set D multi The off-road ground natural driving multi-source heterogeneous data set D is divided into several continuous segments according to the collection time and space multi,cont The continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont is collected continuously in time and space; S20, off-road ground natural driving multi-source heterogeneous data mileage marking: According to the vehicle driving state information and the installation position information of the image acquisition device, the relative mileage of the body perception signal acquisition time point to the collection starting point and the off-road ground area relative mileage range of the external perception signal that can be collected to the collection starting point are estimated respectively; S30, ground image material category labeling is performed on the off-road ground natural running multi-source heterogeneous data, and an off-road ground natural running multi-source heterogeneous data segment D of a uniform ground image material category is extracted multi,uniqmatl : S31, for each continuous off-road ground natural driving multi-source heterogeneous data set D multi,cont annotating the ground image material category of the off-road ground image in units of pixels; the ground image material category Y matl indicates the conclusion obtained by visually judging the ground image according to the material characteristics. S32, extracting the continuous frame off-road ground image containing the unique ground image material category and continuously in the same ground image material category, and the body perception signal contained in the mileage range corresponding to the mileage marking of the continuous frame off-road ground image; S33, labeling the ground image material category of each frame of the body perception signal obtained in step S32 as the corresponding ground image material category, to form an off-road ground natural driving multi-source heterogeneous data paragraph D of uniform ground image material category multi,uniqmatl = {D proprio , D tero , TS proprio , TS tero , Tag mile,proprio , Tag mile,tero , Y matl} uniqmatl ; S40, time-frequency domain feature extraction and first vehicle-wheel-road interaction feature set D interact Construction: S41, the off-road ground natural driving multi-source heterogeneous data segment D of each uniform ground image material category multi,uniqmatl , vehicle-wheel-road interaction state information estimation is performed according to the ontology perception signal; continuous vehicle-wheel-road interaction state data segments are formed respectively; S42, each continuous vehicle-wheel-road interaction state data paragraph is divided into several time length T, the continuous data segment of the mutual overlap length not more than 70%, a total of N vehicle-wheel-road interaction state sample P is formed interact ={p interact,1 ,p interact,2 ,...,p interact,N} S43, calculate the time domain features and frequency domain features of each car-wheel-road interaction state sample p interact,i corresponding to a set of car-wheel-road interaction features x i i1 i2 in ), n is the number of features, and corresponds to a unique ground image material category label y matl,i ;​​​ S44, the off-road ground natural driving multi-source heterogeneous data set containing all the vehicle-wheel-road interaction state samples and the corresponding ground image material category marking form a first vehicle-wheel-road interaction feature set: S50, vehicle-wheel-road interaction feature and ground physical feature category mapping relationship construction: According to the first vehicle-wheel-road interaction feature set D interact , the main vehicle-wheel-road interaction features in the first vehicle-wheel-road interaction feature set are obtained through a feature analysis method to form a second vehicle-wheel-road interaction feature set D' interact , the distribution of the sample set in the second vehicle-wheel-road interaction feature set D' interact is analyzed through a clustering algorithm to obtain n phy clusters, and a ground physical feature category label Y phy is generated for each cluster. S60, ground physical feature category marking of off-road ground natural driving multi-source heterogeneous data: S61、According to step S50, the second vehicle-wheel-road interaction feature set D' interact Each sample (x i ,y matl,i ) in the second vehicle-wheel-road interaction feature set D' interact,i is also labeled as the same ground physical feature category as the corresponding original vehicle-wheel-road interaction state sample p S62, obtaining a vehicle-wheel-road interaction state sample p interact,i mapping the off-road ground natural driving multi-source heterogeneous data segment back to the uniform ground image material category described in steps S30, S40; S63, the ground-truth physical feature class label of each frame of body perception signal is voted by the ground-truth physical feature classes {..., y interact,j-1 ,p interact,j ,p interact,j+1 ,...} of the several car-wheel-road interaction state samples {..., p phy,j-1 ,y phy,j ,y phy,j+1 ,...} to which the frame of body perception signal belongs. S70, constructing the ground image material category Y matl with the ground physical feature category Y phy Probability mapping relationship: According to the clustering results obtained in step S50, the ground physical feature categories to which the samples contained in each ground image material category belong are statistically summarized, and a ground image material category and ground physical feature category probability mapping relationship table is formed; S80, off-road ground physical feature category and mechanical / geometric feature mapping relationship construction: S81, for each car-wheel-road interaction state sample p interact,i estimate the mechanical characteristics mc and the geometric characteristics gc of the ground, said mechanical characteristics including but not limited to the ground adhesion coefficient, said geometric characteristics including but not limited to the ground unevenness; S82, for each ground physical feature category Y phy The corresponding vehicle-wheel-road interaction state sample estimates the mechanical and geometric characteristics fitting Gaussian distribution, obtaining the probability distribution mapping mc of each ground physical feature category and mechanical / geometric characteristics phy,i ~ N(μ mc,i ,σ mc,i ), gc phy,i ~ N(μ gc,i ,σ gc,i ).

2. The off-road ground natural driving dataset labeling method considering multi-dimensional characteristics according to claim 1, characterized in that: In step S10, the body perception signal D proprio Also includes a combined inertial navigation signal mounted on the vehicle; the external perception signal D tero Also includes point cloud data collected by a binocular camera or laser radar mounted on the vehicle.

3. The method of claim 1, wherein the method further comprises: In step S10, the driver's steering behavior data at least includes signals such as steering wheel angle, steering angle speed, brake pressure, engine / drive motor output torque, and engine / drive motor output speed; the vehicle driving state data at least includes signals such as vehicle driving speed, vehicle body longitudinal acceleration, vehicle body lateral acceleration, vehicle wheel slip rate control function, and vehicle body stability control function; the vehicle-wheel dynamics response data at least includes signals such as vehicle wheel speed and vehicle suspension movement stroke; and the off-road ground image data is the ground image collected by the image acquisition device facing the ground or the front of the vehicle.

4. The method of claim 1, wherein the method further comprises: The off-road ground natural driving multi-source heterogeneous data mileage marking step in step S20 comprises: S21, a plurality of heterogeneous data sets D of natural driving on each section of continuous off-road ground multi,cont , a sampling period Δt is established from the starting point of collection mile , a sequence of mileage frames D with the same time span as the continuous data set mile ={F mile , TS mile}, TS mile is a mileage frame timestamp; mileage information F of a location of the vehicle at a collection time of each mileage frame relative to the starting point of collection is estimated according to vehicle driving state information and the mileage frame timestamp mile ; the timestamp sequence of the mileage frame is the same as the timestamp sequence of the off-road ground image S22, a timestamp TS of the body perception signal proprio , a mileage tag Tag of all body perception signals mile,proprio is the most recent last mileage frame acquisition time ts mile,before mileage information f of the location of the vehicle relative to the starting point of acquisition mile,before ; S23, according to the timestamp TS of the external perception signal tero , the mileage mark Tag of all external perception signals mile , tero are the latest last frame mileage frame acquisition time ts mile,before The mileage information f of the location where the vehicle is located relative to the starting point of acquisition mile,before ; S24, estimating the ground longitudinal relative mileage range in the off-road ground image according to the installation position of the image acquisition device, wherein the ground longitudinal relative mileage range in the off-road ground image represents the nearest and farthest distances of the visible ground area in the off-road ground image in the vehicle forward direction relative to the current position of the vehicle; S25, a mileage marking reference Tag according to the off-road ground image mile,tero,base , a ground longitudinal relative mileage range (d min ,d max ) in the off-road ground image mile,tero is a mileage range relative to the starting point of collection in the off-road ground image.

5. The method of claim 1, wherein the method further comprises: The vehicle-wheel-road interaction state information in step S41 includes any other information expressing the vehicle-wheel-road interaction state, including driving resistance and wheel speed noise: (1) The calculation formula of the driving resistance is: F f = F t - F w - F i - F j Wherein: F t F w F i F j F (2) Wheel speed noise: The noise components in the wheel speed signals are extracted using a high-pass filter, and the left front wheel, right front wheel, left rear wheel, and right rear wheel are respectively: fl,noise , fr,noise , rl,noise , rr,noise .

6. The method of claim 1, wherein the method further comprises: The time domain feature and frequency domain feature calculation method in step S43 is as follows: (1) Time domain feature For one vehicle-wheel-road interaction state information S of a vehicle-wheel-road interaction state sample, the time domain feature includes but is not limited to: (a) mean value: (b) Peak-peak value: peak2peak(S)=max(S)-min(S) (c) variance: (d) root mean square: (2) Frequency domain feature The vehicle-wheel-road interaction state information S is subjected to fast Fourier transform to obtain a frequency spectrum S(f) of the vehicle-wheel-road interaction state information, and the frequency domain features include but are not limited to: (a) center of gravity frequency: (b) mean square frequency: (c) Root mean square frequency: RMSF = MSF.

7. The method of claim 1, wherein the method further comprises: The step S50 described vehicle-wheel-road interaction feature and ground physical feature category mapping relationship construction step includes: S51, according to the first vehicle-wheel-road interaction feature set D interact , according to the first vehicle-wheel-road interaction feature set D ·1 , according to the first vehicle-wheel-road interaction feature set D ·2 , according to the first vehicle-wheel-road interaction feature set D ·m , according to the first vehicle-wheel-road interaction feature set D S52, having the same ground image material category y in the second vehicle-wheel-road interaction feature set matl = sample clustering of i, forming several clustering clusters; S53, similarity evaluation is carried out on two two clustering clusters with different ground image material category labels, and the clustering clusters with high similarity are merged; S54, generating the n phy cluster labels Y one by one phy .

Images