High-precision map intelligent correction and updating method based on multi-modal data online learning optimization

Abstract

The invention discloses a high-precision map intelligent correction and updating method based on multi-modal data online learning optimization, and the method achieves the unified sequence representation of map information through the construction of a topological structure of a map reference line and the discretization of a sequence point set, and improves the organization and processing efficiency of map data. CNN semantic segmentation and inverse perspective transformation are adopted, and a pose map optimization method is combined, so that precise perception and fusion of mine field road boundaries are realized, and the spatial precision of map data is improved; frenet coordinate system conversion is used for deducing the deviation state of the real-time movement track of the vehicle, the error amount of a route and a map reference line during real-time running of the vehicle is effectively represented, and the accuracy of a map updating learning model is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for intelligently correcting and updating high-precision maps based on multimodal data online learning optimization. Background Art

[0002] With the rapid development of autonomous driving technology, high-precision maps, as an important part of the autonomous driving system, play a vital role in the safety and reliability of the autonomous driving system due to their data accuracy and real-time update capabilities. In specific application scenarios, such as open-pit mines, due to the mixed structured and unstructured roads and frequent terrain changes, higher requirements are placed on the accuracy and update frequency of high-precision maps.

[0003] Traditional methods of making and updating high-precision maps rely mainly on manual geographic surveys and measurements, which are not only costly but also have long update cycles and are difficult to meet the needs of dynamically changing environments such as open-pit mines. In addition, the harsh environment in mining areas and the many safety hazards of manually driving mining trucks further highlight the potential benefits of autonomous driving technology in mining areas.

[0004] In recent years, some studies have begun to explore map updating techniques based on vehicle GNSS trajectories and data mining methods. For example, Wang et al. proposed a map updating method based on historical vehicle trajectories and geometric features. However, when dealing with complex terrain changes in open-pit mines, these methods are often only applicable to map boundary expansion and road network changes in simple scenarios, and the accuracy of updated map elements is insufficient. Summary of the invention

[0005] The purpose of the present invention is to provide a method for intelligent correction and updating of high-precision maps based on multimodal data online learning optimization in response to the technical defects existing in the prior art.

[0006] The technical solution adopted to achieve the purpose of the present invention is:

[0007] A method for intelligently correcting and updating high-precision maps based on multimodal data online learning optimization includes the following steps:

[0008] Step 1: According to the OpenDrive specification, the road structure, road structure connection relationship and polynomial ParamPoly3 provided by the static text of the initial high-precision map are used to complete the topological connection relationship of different road reference lines in the initial map, and the discrete point sequence (u i , v i ), taking the starting point (x0, y0) of each road structure as the discrete point sequence (u i , v i) to obtain the map reference line sequence (x i ,y i );

[0009] Step 2: Use the CNN semantic segmentation method to perceive the road boundary line of the original image and obtain the semantic pixel position [u, v]. Use the inverse perspective transformation to back-project the semantic pixel position [u, v] from the image plane to obtain the road boundary feature position [x v ,y v ,z v ];

[0010] Step 3: Use the pose graph optimization method to combine the vehicle's IMU and GPS data to obtain the vehicle's precise pose. The road boundary sequence (x i w ,y i w, , z i w );

[0011] Step 4: Transform the real-time vehicle trajectory sequence (x, y) and the map reference line sequence (x i ,y i ) is deduced to obtain the vehicle real-time motion trajectory sequence offset state quantity (S i , L i );

[0012] Step 5: Transform the road boundary sequence (x i w ,y i w ,,z i w ) and the map reference line sequence (x i ,y i ) to obtain the map information sequence, which is then combined with the vehicle real-time motion trajectory sequence offset state quantity (S i , L i ) to perform sequence fusion and obtain a multimodal sequence vector s i , the multimodal sequence vectors are combined to create the corresponding multimodal information sequence X for the vehicle real-time motion trajectory sequence i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1};

[0013] Step 6: Use the multimodal information sequence obtained in step 5 to train the dynamic multi-head self-attention Transformer module learning network, model the attention mechanism, output the sequence features, and obtain the updated map reference line sequence and road boundary sequences, and accurately quantify them using loss functions;

[0014] Step 7: In the actual operation scenario of the vehicle, the trained learning network is tested. According to the test results, the structure and parameters of the learning network are adjusted, and the map road curve is updated in real time as the next initial map, and the update cycle is repeated.

[0015] In the above technical solution, in step 1, according to the OpenDrive specification, firstly based on the initial mine map <road>Middle Multi-segment <geometry>The element S coordinate, heading angle hdg and length length get the starting coordinate S of each reference line start and the end coordinate S end , and evenly separate them to reconstruct each segment <geometry>The sequence S i =S start +i / n·(S end -S start ), where i is S i The index of the point in the sequence, n is the index of each segment <geometry>It is divided into n discrete points;

[0016] Combined with the coefficients aU, bU, cU, dU, aV, bV, cV, dV extracted from the corresponding polynomial ParamPoly3, the discrete point sequence (u i , v i ):

[0017]

[0018] Among them, w i Indicates S i The sequence is converted to the local uv coordinate system in the polynomial ParamPoly3 to calculate u i , v i The intermediate parameter point of

[0019] And based on the discrete point sequence (u i , v i ) per paragraph <geometry>The structure discrete point sequence (x0, y0) is the origin, and the absolute coordinate system map reference line sequence (x i ,y i ):

[0020]

[0021] In the above technical solution, in step 2, the semantic pixel position [u, v] is mapped from the image plane to the ground plane in the vehicle coordinate system by inverse perspective transformation to obtain the road boundary feature position in the vehicle coordinate system:

[0022]

[0023] Among them, π c (·) represents the camera’s distortion and projection model, is the back-projection function, which projects the pixel into space, [R c t c ] is the external parameter matrix of the camera relative to the center of the vehicle, [u, v] is the semantic pixel position, [x v ,y v ,z v ] is the road boundary feature position in the vehicle coordinate system, λ is a scalar, [] col:i It means taking the i-th column of the matrix, where i is 1, 2, and 4.

[0024] In the above technical solution, in step 3, the vehicle maps the collected semantic features from the vehicle coordinate system to the global coordinate system through the optimized posture information. Suppose the global posture of the vehicle at time i is defined as position p i and rotation R(q i ) indicates that R(q i ) is the quaternion q i Function converted to rotation matrix, road boundary feature sequence in vehicle coordinate system The road boundary sequence can be mapped to the global coordinate system through the following transformation:

[0025]

[0026] Through the above transformation, the locally observed features are integrated into a unified global map.

[0027] In the above technical solution, in step 4, the real-time motion trajectory sequence offset state quantity (S i , L i )for:

[0028]

[0029] Among them, θ is the map reference line sequence (x i ,y i ), α is the angle between the current position of the vehicle and the tangent direction of the reference line, and (x, y) is the real-time motion trajectory sequence of the vehicle.

[0030] In the above technical solution, in step 5, the multimodal sequence corresponds to the map reference line sequence number, the multimodal sequence is truncated according to k steps, and the multimodal sequence with the starting sequence number i is represented as X i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1 }, each multimodal sequence vector s i The map reference line sequence (x i ,y i ), the road boundary sequence in the global coordinate system (x i w ,y i w ,,z i w ), the vehicle real-time motion trajectory sequence offset state (S i , L i ), fused to get s i＝ {x i ,y i , x i w ,y i w , z i w , S i , L i }.

[0031] In the above technical solution, in step 6, the dynamic multi-head self-attention Transformer module includes three independent attention "heads", which respectively learn the semantic expressions of reference lines, boundaries, and road terrain in the sequence, and the outputs of the three independent attention "heads" are combined and spliced ​​to obtain:

[0032] MultiHead(Q,K,V)=Concat(head1,

[0033] ...,head h )W O

[0034] head i =Attention(QW i Q ,KW i K ,VW i V )

[0035]

[0036] Among them, MultiHead represents multi-head attention, Q, K and V represent query, key and value matrices respectively, Concat represents feature concatenation, head h is the hth attention head, W O It is the weight matrix used to multiply the output of multi-head self-attention. i is the i-th attention head, QW i Q ,KW i K and VW i V is the weight matrix of Q, K and V, Attention is calculated, softmax is a normalization method, d k is the dimension of the key vector, K T is the transpose of K, QK T This mechanism enables the dynamic multi-head self-attention Transformer module to process sequences in parallel, significantly improving computational efficiency.

[0037] In the above technical solution, in step 6, the mean square error (MSE) is used to quantify the accuracy of the updated map reference line sequence and road boundary sequence, and the following errors are used for evaluation: Among them, L map is the loss function of the network, is the updated map reference line sequence, p i is the real coordinate of the vehicle, N is the total number of discrete points in the map, and σ is a learnable noise parameter.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention realizes a unified sequence representation of map information and improves the organization and processing efficiency of map data by constructing a topological structure of map reference lines and discretizing a sequence point set.

[0040] 2. The present invention adopts CNN semantic segmentation and inverse perspective transformation, combined with the pose graph optimization method, to achieve accurate perception and fusion of mine road boundaries and improve the spatial accuracy of map data.

[0041] 3. The present invention uses the Frenet coordinate system transformation to derive the offset state of the vehicle's real-time motion trajectory, effectively characterizes the error between the vehicle's real-time running route and the map reference line, and improves the accuracy of the map update learning model.

[0042] 4. The present invention utilizes the learning network of Transformer, can make full use of multimodal data, continuously optimize the map update results through online learning, and improve the intelligence level of map update.

[0043] 5.L map It is the loss function of the network, which mainly minimizes the error of the predicted map reference line sequence and road boundary sequence and the noise part of the learning input data.

[0044] 6. The present invention verifies the effectiveness of the algorithm under actual vehicle operation scenarios, achieves high-precision and real-time map updates, and meets the strict requirements of autonomous driving vehicles for high-precision maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a diagram of the overall method architecture of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] A method for intelligently correcting and updating high-precision maps based on multimodal data online learning optimization includes the following steps:

[0048] Step 1: According to the OpenDrive specification, the road structure, road structure connection relationship and polynomial provided by the static text of the initial mine high-precision map are used to complete the topological connection relationship of different road reference lines in the initial map, and the discrete point sequence (u i , v i ), taking the starting point (x0, y0) of each road structure as the discrete point sequence (u i , v i ) to obtain the map reference line sequence (x i ,y i );

[0049] Specifically, according to the OpenDrive specification, firstly, based on the initial mine site high-precision map text <road>Provided <link> middle <predecessor>and <successor>Information, representation combined <road>as well as <junction>The road link information in the structure can be used to construct the topological connection relationship of different roads in the mine on the explicit map; then, according to the multi-segment reference line coordinate system <geometry>In <type>Get the corresponding reference line type (using the typical ParamPoly3 as an example), and then use the multi-segment <geometry>The element S coordinate, heading angle hdg and length length are used to obtain the starting coordinate S of each reference line. start and the end coordinate S end , and evenly separate them to reconstruct each segment <geometry>sequence S i =S start +i / n·(S end -S start ), where i is S i The index of the point in the sequence, n is the index of each segment <geometry>It is divided into n discrete points;

[0050] Combined with the coefficients aU, bU, cU, dU, aV, bV, cV, dV extracted from the corresponding polynomial ParamPoly3, the discrete point sequence (u i , v i ):

[0051]

[0052] Among them, w i Indicates S i The sequence is converted to the local uv coordinate system in the polynomial ParamPoly3 to calculate u i , v i The intermediate parameter point of

[0053] And based on the discrete point sequence (u i , v i ) Take the starting point (x0, y0) of each road structure as the origin and find the map reference line sequence (x i ,y i ):

[0054]

[0055] Step 2: Use the CNN semantic segmentation method to perceive the road boundary line in the original image and obtain the semantic pixel position [u, v]. Use the inverse perspective transformation to back-project the semantic pixel position [u, v] from the image plane to obtain the road boundary feature position [x v ,y v ,z v ];

[0056] Specifically, the semantic pixel position [u, v] is mapped from the image plane to the ground plane in the vehicle coordinate system through inverse perspective transformation, and the road boundary feature position [x v ,y v ,z v ]:

[0057]

[0058] Among them, π c (·) represents the camera’s distortion and projection model, is the back-projection function, which projects the pixel into space, [R c t c ] is the external parameter matrix of the camera relative to the center of the vehicle, [u, v] is the semantic pixel position, [x v ,y v ,z v ]Road boundary features in vehicle coordinate system, λ is a scalar, [] col:i It means taking the i-th column of the matrix, where i is 1, 2, and 4.

[0059] Step 3: Use the pose graph optimization method to combine the vehicle's IMU and GPS data to obtain the vehicle's precise pose. The road boundary sequence (x i w ,y i w, , z i w );

[0060] The vehicle maps the collected semantic features from the vehicle coordinate system to the global coordinate system through the optimized posture information. The road boundary sequence in the global coordinate system is:

[0061] Assume that the global position of the vehicle at time i is given by position p i and rotation R(q i ) indicates that R(q i ) is the quaternion q i Function converted to rotation matrix, road boundary feature sequence in vehicle coordinate system The road boundary sequence can be mapped to the global coordinate system through the following transformation:

[0062]

[0063] To integrate locally observed features into a unified global map.

[0064] Step 4: Transform the real-time trajectory sequence (x, y) of the vehicle and the map reference line sequence (x i ,y i ) is deduced to obtain the vehicle real-time motion trajectory sequence offset state quantity (S i , L i );

[0065] The vehicle's real-time motion trajectory sequence offset state quantity (S i , L i )for:

[0066]

[0067] Among them, θ is the map reference line sequence (x i ,y i ), α is the angle between the current position of the vehicle and the tangent direction of the reference line, (x, y) is the real-time motion trajectory sequence of the vehicle (i.e. the current position of the vehicle), and this process involves the position of the vehicle in the Frenet coordinate system and the tangent direction of the reference line, as well as the calculation of the lateral and longitudinal positions of the vehicle relative to the reference line.

[0068] Step 5: Transform the road boundary sequence (x i w ,y i w, , z i w ) and the map reference line sequence (x i ,y i ) to obtain the map information sequence, which is then combined with the vehicle real-time motion trajectory sequence offset state quantity (S i , L i ) and perform sequence fusion to obtain a multimodal sequence vector s i , the multimodal sequence vectors are combined to create the corresponding multimodal information sequence X for the vehicle real-time motion trajectory sequence i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1 };

[0069] Specifically, the multimodal information sequence corresponds to the map reference line sequence number, and the multimodal sequence is truncated according to k steps. The multimodal information sequence with the starting sequence number i is represented as X i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1 }, each multimodal sequence vector s i The map reference line sequence (x i ,y i ), the road boundary sequence in the global coordinate system (x i w ,y i w ,,z i w ), the vehicle real-time motion trajectory sequence offset state (S i , L i ), thus we get s i＝ {x i ,y i , x i w ,y i w , z i w , S i , L i }.

[0070] Step 6: Use the multimodal information sequence obtained in step 5 to train the dynamic multi-head self-attention Transformer module learning network, model the attention mechanism, output the sequence features, and obtain the updated map reference line sequence and road boundary sequences, and the loss function is used for accurate quantification.

[0071] The learning network of the dynamic multi-head self-attention Transformer includes three independent attention "heads", which respectively learn the semantic expression of reference lines, boundaries, and road terrain in the sequence, and the outputs of the three independent attention "heads" are combined and spliced ​​to obtain:

[0072] MultiHead(Q,K,V)=Concat(head1,

[0073] ...,head h )W O

[0074] head i =Attention(QW i Q ,KW i K ,VW i V )

[0075]

[0076] Among them, MultiHead represents multi-head attention, Q, K and V represent query, key and value matrices respectively, Concat represents feature concatenation, head h is the hth attention head, W O It is the weight matrix used to multiply the output of multi-head self-attention. i is the i-th attention head, QW i Q ,KW i K and VW i V is the weight matrix of Q, K and V, Attention is used for attention calculation, softmax is a normalization method, d k is the dimension of the key vector, K T is the transpose of K, QK T is the attention score, which determines the weight of each value when generating the output. This mechanism enables Transformer to process sequences in parallel, significantly improving computational efficiency. The mean square error (MSE) is used to quantify the accuracy of the updated map reference line sequence and road boundary sequence. For each discrete point on the map, the sum of the squares of the Euclidean distance between the predicted coordinates and the true coordinates is calculated, and a learnable noise parameter σ is added to obtain the following loss function: Among them, L map is the loss function of the network, is the updated map reference line sequence, p i is the real coordinate of the vehicle, N is the total number of discrete points in the map, and σ is a learnable noise parameter.

[0077] Step 7: In the actual operation scenario of the vehicle, the trained learning network is tested, including the maximum update error and average update error of the road centerline and boundary line. According to the test results, the structure and parameters of the learning network are adjusted, and the map road curve is updated in real time as the next initial map, and the update cycle is repeated.

[0078] In this embodiment, the vehicle may be a mining truck.

[0079] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / geometry> < / geometry> < / geometry> < / type> < / geometry> < / junction> < / road> < / successor> < / predecessor> < / road> < / geometry> < / geometry> < / geometry> < / geometry> < / road>

Claims

1. A high-precision map intelligent correction and update method based on multimodal data online learning optimization, characterized in that: The following steps are involved: Step 1: According to the OpenDrive specification, the road structure, road structure connection relationship and polynomial ParamPoly3 provided by the static text of the initial high-precision map are used to complete the topological connection relationship of different road reference lines in the initial map, and the discrete point sequence (u i , v i ), taking the starting point (x0, y0) of each road structure as the discrete point sequence (u i , v i ) to obtain the map reference line sequence (x i ,y i ); Step 2: Use the CNN semantic segmentation method to perceive the road boundary line of the original image and obtain the semantic pixel position [u, v]. Use the inverse perspective transformation to back-project the semantic pixel position [u, v] from the image plane to obtain the road boundary feature position [x v ,y v ,z v ]; Step 3: Use the pose graph optimization method to combine the vehicle's IMU and GPS data to obtain the vehicle's precise pose. The road boundary sequence (x i w ,y i w, , z i w ); Step 4: Transform the real-time vehicle trajectory sequence (x, y) and the map reference line sequence (x i ,y i ) is deduced to obtain the vehicle real-time motion trajectory sequence offset state quantity (S i , L i ); Step 5: Transform the road boundary sequence (x i w ,y i w, , z i w ) and the map reference line sequence (x i ,y i ) to obtain the map information sequence, which is then combined with the vehicle real-time motion trajectory sequence offset state quantity (S i , L i ) to perform sequence fusion and obtain a multimodal sequence vector s i , the multimodal sequence vectors are combined to create the corresponding multimodal information sequence X for the vehicle real-time motion trajectory sequence i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1 }; Step 6: Use the multimodal information sequence obtained in step 5 to train the dynamic multi-head self-attention Transformer module learning network, model the attention mechanism, output the sequence features, and obtain the updated map reference line sequence and road boundary sequences, and accurately quantify them using loss functions; Step 7: In the actual operation scenario of the vehicle, the trained learning network is tested. According to the test results, the structure and parameters of the learning network are adjusted, and the map road curve is updated in real time as the next initial map, and the update cycle is repeated.

2. The method according to claim 1, characterized in that: In step 1, according to the OpenDrive specification, firstly based on the initial mine map <road>Middle Multi-segment <geometry>The element S coordinate, heading angle hdg and length length get the starting coordinate S of each reference line start and the end coordinate S end , and evenly separate them to reconstruct each segment <geometry>The sequence S i =S start +i / n·(S end -S start ), where i is S i The index of the point in the sequence, n is the index of each segment <geometry> It is divided into n discrete points;< / geometry> < / geometry> < / geometry> < / road> Combined with the coefficients aU, bU, cU, dU, aV, bV, cV, dV extracted from the corresponding polynomial ParamPoly3, the discrete point sequence (u i , v i ).

3. The method according to claim 2, characterized in that The discrete point sequence (u i , v i ): Among them, w i Indicates S i The sequence is converted to the local uv coordinate system in the polynomial ParamPoly3 to calculate u i , v i The intermediate parameter point of And based on the discrete point sequence (u i , v i ) per paragraph <geometry>The structure discrete point sequence (x0, y0) is the origin, and the absolute coordinate system map reference line sequence (x i ,y i ).< / geometry> 4. The method according to claim 3, characterized in that The map reference line sequence (x i ,y i ):

5. The method according to claim 1, characterized in that In step 2, the semantic pixel position [u, v] is mapped from the image plane to the ground plane in the vehicle coordinate system by inverse perspective transformation to obtain the road boundary feature position in the vehicle coordinate system: Among them, π c (·) represents the camera’s distortion and projection model, is the back-projection function, which projects the pixel into space, [R c t c ] is the external parameter matrix of the camera relative to the center of the vehicle, [u, v] is the semantic pixel position, [x v ,y v ,z v ] is the road boundary feature position in the vehicle coordinate system, λ is a scalar, [] col:i It means taking the i-th column of the matrix, where i is 1, 2, and 4.

6. The method according to claim 1, characterized in that In step 3, the vehicle maps the collected semantic features from the vehicle coordinate system to the global coordinate system through the optimized posture information. Suppose the global posture of the vehicle at time i is defined as position p i and rotation R(q i ) indicates that R(q i ) is the quaternion q i Function converted to rotation matrix, road boundary feature sequence in vehicle coordinate system The road boundary sequence can be mapped to the global coordinate system through the following transformation:

7. The method according to claim 1, characterized in that In step 4, the vehicle's real-time motion trajectory sequence offset state quantity (S i , L i )for: Among them, θ is the map reference line sequence (x i ,y i ), α is the angle between the current position of the vehicle and the tangent direction of the reference line, and (x, y) is the real-time motion trajectory sequence of the vehicle.

8. The method according to claim 1, characterized in that In step 5, the multimodal sequence corresponds to the map reference line sequence number, and the multimodal sequence is truncated according to k steps. The multimodal sequence with the starting sequence number i is represented as X i k ={s i ,s i+1 ,…,s i+k-2 ,s i+k-1 }, each multimodal sequence vector s i The map reference line sequence (x i ,y i ), the road boundary sequence in the global coordinate system (x i w ,y i w, , z i w ), the vehicle real-time motion trajectory sequence offset state (S i , L i ), fused to get s i＝ {x i ,y i , x i w ,y i w , z i w , S i , L i }.

9. The method according to claim 1, characterized in that: In step 6, the dynamic multi-head self-attention Transformer module includes three independent attention "heads", which respectively learn the semantic expression of reference lines, boundaries, and road terrain in the sequence, and the outputs of the three independent attention "heads" are combined and spliced ​​to obtain: MultiHead(Q,K,V)=Concat(head1,...,head h )W O head i =Attention(QW i Q ,KW i K ,VW i V ) Among them, MultiHead represents multi-head attention, Q, K and V represent query, key and value matrices respectively, Concat represents feature concatenation, head h is the hth attention head, W O It is the weight matrix used to multiply the output of multi-head self-attention. i is the i-th attention head, QW i Q ,KW i K and VW i V is the weight matrix of Q, K and V, Attention is calculated, softmax is a normalization method, d k is the dimension of the key vector, K T is the transpose of K, QK T is the attention score.

10. The method according to claim 1, characterized in that In step 6, the mean square error (MSE) is used to quantify the accuracy of the updated map reference line sequence and road boundary sequence, and the following errors are used for evaluation: Among them, L map is the loss function of the network, is the updated map reference line sequence, p i is the real coordinate of the vehicle, N is the total number of discrete points in the map, and σ is a learnable noise parameter.

Images