A hinged vehicle trajectory planning method based on multi-camera surround view information

Abstract

This invention proposes a hinged vehicle trajectory planning method based on multi-camera surround view information. It estimates the relative pose of the cameras based on the surround view images, combines the independent semantic cognition of each camera to obtain a bird's-eye view semantic cognition map, and establishes a time-invariant global matching relationship with the semantic map. Based on the matching features, it completes high-precision estimation of the vehicle pose. By improving the accuracy of vehicle pose estimation and the long-term robustness of the matching features, it enhances environmental perception and localization accuracy, providing high-precision data to be fused for subsequent trajectory planning. This invention designs a vehicle trajectory planning method based on the bird's-eye view semantic cognition map and vehicle pose estimation. It fuses the constructed traffic strategy set with quasi-static representation and vehicle pose to complete trajectory planning, improving the real-time performance and completeness of trajectory planning. Tracking is performed based on the predicted trajectory, and the traffic strategy set is updated based on the tracking accuracy, improving the feasibility of trajectory planning.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to a hinged vehicle trajectory planning method based on multi-camera surround view information. Background Technology

[0002] Multi-camera surround-view systems are distributed across various levels of the articulated vehicle body, including both non-rigid and rigid connection systems. They provide omnidirectional, blind-spot-free perception of the vehicle's surroundings, offering rich multi-view images, and are therefore widely used for vehicle trajectory planning. The vehicle estimates its pose by matching multi-view images with a semantic map and solves for its trajectory through a search. However, current multi-view image-based vehicle trajectory planning methods typically match the geometric features of the images, exhibiting strong parameter dependence and poor robustness. Therefore, they are unsuitable for the dynamic, stochastic multi-view images provided by rigid connection systems, unlike trajectory planning methods for non-rigid systems.

[0003] Currently, vehicle trajectory planning methods for non-rigid systems typically employ a two-layer heuristic search approach to improve solution speed and enhance real-time performance. This two-layer heuristic search method comprises two search layers: the coarse solution obtained in the first layer is used to inspire a faster search in the second layer. Current methods include A*-optimal control, state lattice-optimal control, state lattice-graph search, and reinforcement learning-out-of-lattice search, but these suffer from poor real-time performance. Another search method improves real-time performance by switching between deterministic policies and search methods, but its applicability is limited.

[0004] Therefore, there is an urgent need for a vehicle trajectory planning method that can adapt to non-rigidly connected multi-camera surround systems and improve the feasibility and real-time performance of trajectory planning. Summary of the Invention

[0005] In view of this, the present invention provides a hinged vehicle trajectory planning method based on multi-camera surround view information, which can improve the feasibility, real-time performance and tracking accuracy of planning.

[0006] To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0007] A hinged vehicle trajectory planning method based on multi-camera surround view information, comprising the following steps:

[0008] S0. Search within the feasible state domain of the vehicle, which is composed of the spatiotemporal relationships of the poses of each level of the articulated vehicle, and obtain the set of traffic strategies offline.

[0009] S1. Multiple cameras are mounted on each level of the articulated vehicle body to form a multi-phase surround view system to collect surround view images of the vehicle.

[0010] Based on the panoramic image, the relative poses between each camera are estimated. Combining the relative poses between each camera, the independent semantic cognition of each camera to its own field of view environment is transformed into the ground coordinate system under the bird's-eye view, forming a bird's-eye view panoramic semantic cognition map of the driving environment.

[0011] S2. Establish a global mapping between the bird's-eye view semantic cognitive map and the ground semantic map, construct a vehicle pose prediction network, and use the vehicle pose prediction network to calculate the vehicle pose.

[0012] S3. Obtain a quasi-static representation of the driving environment based on the bird's-eye view semantic cognitive map.

[0013] S4. Starting from the vehicle pose, the quasi-static representation of the driving environment and the traffic strategy set are integrated to obtain the strategy chain, and the predicted trajectory of the vehicle is obtained based on the strategy chain.

[0014] Furthermore, it also includes S5: tracking the vehicle according to the predicted trajectory and calculating the tracking accuracy, and determining whether to update the passage strategy set based on the tracking accuracy. The update method is as follows: when the tracking accuracy is not lower than the expectation, the passage strategy set is not updated, and the process returns to S1 to continue vehicle trajectory planning until the vehicle reaches its destination; when the tracking accuracy is lower than the expectation, the predicted trajectory is replaced with the vehicle's actual trajectory, and the process returns to S1 to continue vehicle trajectory planning until the vehicle reaches its destination.

[0015] Furthermore, the method for estimating the relative pose between cameras is as follows: a graph neural network is used to perform feature association calculation on the panoramic images acquired by each camera to obtain the relative pose between cameras; a region common-view transfer mechanism is used to perform loop closure optimization on the relative pose to obtain a preliminary estimation result of the relative pose; a Kalman filter method is used to fuse the preliminary estimation result of the relative pose of the cameras with the historical data of the relative pose between different levels of the vehicle body to obtain the relative pose estimation result of the cameras.

[0016] Furthermore, the vehicle pose prediction network is constructed as follows:

[0017] A bird's-eye view semantic feature encoder is established based on the bird's-eye view spatial semantic cognition map to obtain the surrounding view features corresponding to each surround view image from the bird's-eye view perspective. A semantic map encoder is established based on the semantic map to obtain the semantic features of the driving environment. A Transformer model integrating self-attention and cross-attention mechanisms is established, using semantic features and surrounding view features as inputs and the matching features of the two inputs as the target output to train the model, resulting in a Transformer model that reflects the global mapping relationship between the bird's-eye view surrounding view semantic cognition map and the semantic map. The matching features output by this Transformer model are used as inputs, and the tractor pose is used as the output to train the neural network, resulting in a vehicle pose prediction network.

[0018] Furthermore, the vehicle pose is calculated as follows: the tractor pose is obtained based on the vehicle pose prediction network, and the pose of each vehicle body is deduced based on the hinge angles between each vehicle body and the vehicle kinematic model to obtain the vehicle pose.

[0019] Furthermore, the specific method for obtaining the traffic strategy set is as follows: A vehicle-road coupling model is used to obtain the spatiotemporal relationships of vehicle poses at various levels corresponding to different driving environments, forming the vehicle's state feasible domain; environmental parameters of the driving environment are randomly sampled, and vehicle traffic strategies within the state feasible domain of that driving environment are searched to obtain preliminary solutions for traffic strategies; the preliminary solutions for traffic strategies are screened according to preset strategy evaluation indicators, and the similarity between traffic strategies is calculated; the preliminary solutions for traffic strategies are clustered according to the similarity to obtain the traffic strategy set; The vehicle-road coupling model is constructed as follows: the spatiotemporal relationship between the tractor pose and the trailer poses at various levels is deduced using the historical running trajectory of the tractor and the hinge angles between vehicle bodies at various levels under each historical running trajectory, and the vehicle's kinematic model; a vehicle-road coupling model is established based on the coupling relationship between the driving environment and the spatiotemporal relationship.

[0020] Furthermore, the specific method of S4 is as follows: taking the vehicle pose as the starting point, the quasi-static representation of the driving environment and the traffic strategy set are sequentially matched to obtain the strategy chain; each strategy node in the strategy chain is relaxed to obtain the corresponding reachable set; sampling is performed in the reachable set to obtain the position corresponding to each strategy node to form the initial trajectory; the initial trajectory is smoothed to obtain the predicted trajectory of the vehicle.

[0021] Beneficial effects:

[0022] 1. This invention proposes a hinged vehicle trajectory planning method based on multi-camera surround view information. It estimates the relative pose of the cameras based on the surround view images, combines the independent semantic cognition of each camera to obtain a bird's-eye view semantic cognition map, and establishes a time-invariant global matching relationship with the semantic map. Based on the matching features, it completes high-precision estimation of the vehicle pose. By improving the accuracy of vehicle pose estimation and the long-term robustness of the matching features, it enhances environmental perception and positioning accuracy, providing high-precision data to be fused for subsequent trajectory planning. This invention designs a vehicle trajectory planning method based on the bird's-eye view semantic cognition map and vehicle pose estimation. It fuses the constructed traffic strategy set with quasi-static representation and vehicle pose to complete trajectory planning, improving the real-time performance and completeness of trajectory planning. Tracking is performed based on the predicted trajectory, and the traffic strategy set is updated based on the tracking accuracy, improving the feasibility of trajectory planning.

[0023] 2. This invention extracts surround view features from surround view images based on bird's-eye view semantic cognition maps. The surround view features are used as a measure of surround view perception of the driving environment. Compared with the surround view features corresponding to the surround view images from the perspective of each camera itself, it can better distinguish the semantic cognition of different objects and complete object recognition.

[0024] 3. This invention constructs a trajectory search method based on traffic strategy inspiration, which improves the real-time performance and completeness of trajectory planning. It establishes an update mechanism for trajectory planning and tracking accuracy, and replaces the predicted trajectory with the real trajectory when the tracking accuracy is lower than expected, so as to correct the trajectory planning in a timely manner. This results in better feasibility and real-time performance.

[0025] 4. This invention employs a region co-view transfer mechanism to perform loop closure optimization on relative poses and combines historical data of vehicle relative poses for compensation. This enables accurate estimation of the relative poses of non-rigidly connected cameras, laying the foundation for semantic fusion based on independent semantic cognition of cameras.

[0026] 5. This invention obtains the matching features of semantic features and surround view features by establishing a Transformer model. The vehicle pose prediction network is obtained through the matching features. Unlike the geometric feature matching used in the prior art, this invention achieves deep matching of surround view image and semantic map through the matching of high-level features (such as semantic features), which improves the long-term robustness of the matching features and thus improves the positioning accuracy.

[0027] 6. This invention establishes a Transformer model that integrates self-attention and cross-attention mechanisms to obtain matching features that reflect the global mapping relationship between the bird's-eye view semantic cognitive map and the semantic map, thereby obtaining a vehicle pose prediction network that can better match the semantic features of the surround view image and the semantic map to more accurately estimate the vehicle pose.

[0028] 7. This invention uses quasi-static representations of vehicle pose and driving environment to perform sequential matching with a set of traffic strategies. In the short time domain, the driving environment is continuously updated based on the strategy obtained from the previous matching step to determine the next strategy, gradually obtaining a strategy chain to obtain the predicted trajectory, resulting in higher planning accuracy. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 A flowchart illustrating the method for vehicle pose estimation.

[0031] Figure 3 A flowchart illustrating the method for vehicle trajectory planning. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Multiple cameras are mounted on each level of the articulated vehicle body to form a multi-phase surround-view system, which acquires images of the environment during vehicle operation, achieving omnidirectional perception of the vehicle's surroundings. Based on the surround-view images acquired by the multi-phase surround-view system, this invention provides a trajectory planning method for articulated vehicles based on multi-camera surround-view information, the steps of which are as follows: Figure 1 As shown, it includes camera pose estimation, bird's-eye view semantic cognition of the environment, quasi-static representation of dynamic scenes, vehicle pose estimation, construction and updating of traffic policy sets, and trajectory planning and tracking, specifically:

[0034] Step 1, Camera pose estimation:

[0035] An articulated vehicle is known to consist of a multi-stage body, including a single-stage tractor and at least one stage trailer. Each stage of the body is equipped with a multi-camera surround-view system consisting of multiple cameras to capture surround-view images of the vehicle. For example... Figure 2 As shown, a graph neural network is used to perform feature association calculation on the panoramic images acquired by each camera to obtain the relative pose between the cameras; a region common-view transfer mechanism is used to perform loop closure optimization on the relative pose to obtain the preliminary estimation result of the relative pose; a Kalman filter method is used to fuse the preliminary estimation result of the relative pose of the cameras with the historical data of the relative pose between the vehicle bodies at each level to complete the real-time estimation of the relative pose between the cameras.

[0036] Step 2: Obtaining the bird's-eye view semantic cognitive map:

[0037] like Figure 2 As shown, when estimating the relative poses between cameras, both convolutional neural networks and graph neural networks are used to achieve independent semantic cognition of each camera's own field of view environment. Based on the relative poses between cameras, the semantic description models contained in the independent semantic cognition are transformed into the ground coordinate system under the bird's-eye view and fused and matched to form a bird's-eye view semantic cognition map of the driving environment.

[0038] Step 3: Pose estimation of each vehicle body level:

[0039] like Figure 2As shown, a global mapping is established between the bird's-eye view semantic cognitive map and the ground semantic map. A vehicle pose prediction network is constructed, and the vehicle pose prediction network is used to estimate the tractor pose. Then, the pose of the trailer is inferred by combining the hinge angles between vehicle levels and the vehicle kinematic model, thus completing the pose estimation of each vehicle level. Specifically: First, a bird's-eye view semantic feature encoder is established based on the bird's-eye view spatial semantic cognitive map to obtain the surrounding features corresponding to each surround view image from the bird's-eye view perspective; a semantic map encoder is established based on the semantic map to obtain the semantic features of the driving environment; a Transformer model integrating self-attention and cross-attention mechanisms is established, using semantic features and surrounding features as inputs and the matching features of the two inputs as the target output to train the model, resulting in a Transformer model that reflects the global mapping relationship between the bird's-eye view semantic cognitive map and the semantic map; the matching features output by this Transformer model are used as inputs, and the tractor pose is used as the output to train the neural network, resulting in the vehicle pose prediction network. When estimating the pose of the tractor unit, the matching features of the surround view image are input into the vehicle pose prediction network to obtain the tractor unit pose estimation result. Then, based on the hinge angles between each vehicle body and the tractor unit pose estimation result, combined with the hinged vehicle kinematics model, the poses of subsequent vehicle bodies are deduced to achieve full vehicle pose estimation. Figure 2 As shown, the semantic features and look-around features mentioned in this invention are in the form of high-dimensional structured vectors.

[0040] Step 4: Construction and updating of the access strategy set:

[0041] like Figure 3 As shown, a vehicle-road coupling model is used to obtain the spatiotemporal relationship of vehicle poses at various levels corresponding to different driving environments, forming the vehicle's state feasible domain. The environmental parameters of the driving environment are randomly sampled, and the vehicle travel strategy within the state feasible domain under the driving environment is searched to obtain the initial solution of the travel strategy. The initial solution of the travel strategy is screened according to the preset strategy evaluation index, and the similarity between the travel strategies is calculated based on the point vector method. The initial solution of the travel strategy is clustered according to the similarity to obtain the travel strategy set.

[0042] Step 5: Dynamic trajectory planning and tracking:

[0043] like Figure 3 As shown, based on the bird's-eye view spatial semantic cognitive map, a quasi-static representation of the driving environment is constructed based on the spatiotemporal compression theory. Taking the vehicle pose as the starting point, the quasi-static representation of the driving environment and the traffic strategy set are integrated to obtain the strategy chain. Each strategy node in the strategy chain is relaxed to obtain the corresponding reachable set. The position of each strategy node is sampled in the reachable set to form the initial trajectory. The initial trajectory is smoothed to obtain the predicted trajectory of the vehicle.

[0044] Furthermore, this invention employs a trajectory tracking control algorithm to track the vehicle's actual trajectory according to the predicted trajectory and calculate the tracking accuracy. Based on the tracking accuracy, it determines whether to update the traffic strategy set. The update method is as follows: the tracking accuracy is compared with the expected value to update the traffic strategy set. If the tracking accuracy is not lower than the expected value, the traffic strategy set is not updated, and vehicle trajectory planning continues based on the surround view images within the next planning time period until the vehicle reaches its destination. If the tracking accuracy is lower than the expected value, the predicted trajectory is replaced with the vehicle's actual trajectory, and vehicle trajectory planning continues based on the surround view images within the next planning time period until the vehicle reaches its destination. The vehicle-road coupling model is prior knowledge, constructed as follows: the spatiotemporal relationship between the tractor's pose and the trailer's pose is deduced using the tractor's historical operating trajectory, the hinge angles between different vehicle levels under each historical operating trajectory, and the vehicle's kinematic model; the vehicle-road coupling model is established based on the coupling relationship between the driving environment and the spatiotemporal relationship.

[0045] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hinged vehicle trajectory planning method based on multi-camera surround view information, characterized in that the steps include... include: S0. Search within the feasible state domain of the vehicle, which is composed of the spatiotemporal relationships of the poses of each level of the articulated vehicle, and obtain the set of traffic strategies offline. S1. Multiple cameras are mounted on each level of the articulated vehicle body to form a multi-phase surround view system to collect surround view images of the vehicle. Based on the panoramic image, the relative poses between each camera are estimated. Combining the relative poses between each camera, the independent semantic cognition of each camera to its own field of view environment is transformed into the ground coordinate system under the bird's-eye view, forming a bird's-eye view panoramic semantic cognition map of the driving environment. S2. Establish a global mapping between the bird's-eye view semantic cognitive map and the ground semantic map, construct a vehicle pose prediction network, and use the vehicle pose prediction network to calculate the vehicle pose. S3. Obtain a quasi-static representation of the driving environment based on the bird's-eye view semantic cognitive map; S4. Starting with the vehicle pose, the quasi-static representation of the driving environment and the traffic strategy set are fused to obtain the strategy chain, and the predicted trajectory of the vehicle is obtained based on the strategy chain. The method for estimating the relative pose between each camera is as follows: a graph neural network is used to perform feature association calculation on the panoramic images collected by each camera to obtain the relative pose between each camera; a region common-view transfer mechanism is used to perform loop closure optimization on the relative pose to obtain the preliminary estimation result of the relative pose; a Kalman filter method is used to fuse the preliminary estimation result of the relative pose of the camera with the historical data of the relative pose between each level of the vehicle body to obtain the relative pose estimation result of the camera. The vehicle pose prediction network is constructed as follows: A bird's-eye view semantic feature encoder is established based on the bird's-eye view spatial semantic cognition map to obtain the surrounding view features corresponding to each surround view image from the bird's-eye view perspective; a semantic map encoder is established based on the semantic map to obtain the semantic features of the driving environment. A Transformer model integrating self-attention and cross-attention mechanisms is established. Semantic features and surround view features are used as inputs, and the matching features of the two inputs are used as the target output to train the model, resulting in a Transformer model that reflects the global mapping relationship between the bird's-eye view semantic cognitive map and the semantic map. The matching features output by this Transformer model are used as inputs, and the tractor pose is used as the output to train the neural network, resulting in a vehicle pose prediction network.

2. The method as described in claim 1, characterized in that, It also includes S5: Tracking the vehicle according to the predicted trajectory and calculating the tracking accuracy, and determining whether to update the passage strategy set based on the tracking accuracy. The update method is as follows: if the tracking accuracy is not lower than the expectation, the passage strategy set is not updated, and the system returns to S1 to continue vehicle trajectory planning until the vehicle reaches its destination; if the tracking accuracy is lower than the expectation, the actual trajectory of the vehicle is used to replace the predicted trajectory, and the system returns to S1 to continue vehicle trajectory planning until the vehicle reaches its destination.

3. The method according to any one of claims 1-2, characterized in that, The vehicle pose is calculated as follows: the tractor pose is obtained based on the vehicle pose prediction network, and the pose of each vehicle body is deduced based on the hinge angles between each vehicle body and the vehicle kinematic model to obtain the vehicle pose.

4. The method according to any one of claims 1-2, characterized in that, The specific method for obtaining the traffic strategy set is as follows: A vehicle-road coupling model is used to obtain the spatiotemporal relationships of vehicle poses at various levels corresponding to different driving environments, forming the vehicle's state feasible domain; environmental parameters of the driving environment are randomly sampled, and vehicle traffic strategies within the state feasible domain of that driving environment are searched to obtain preliminary solutions for traffic strategies; the preliminary solutions for traffic strategies are screened according to preset strategy evaluation indicators, and the similarity between traffic strategies is calculated; the preliminary solutions for traffic strategies are clustered according to the similarity to obtain the traffic strategy set; wherein, the vehicle-road coupling model is constructed as follows: the spatiotemporal relationship between the tractor pose and the trailer poses at various levels is deduced using the historical running trajectory of the tractor and the hinge angles between vehicle bodies at various levels under each historical running trajectory, and the vehicle's kinematic model; a vehicle-road coupling model is established based on the coupling relationship between the driving environment and the spatiotemporal relationship.

5. The method according to any one of claims 1-2, characterized in that, The specific method of S4 is as follows: taking the vehicle pose as the starting point, performing serialization matching on the quasi-static representation of the driving environment and the traffic strategy set to obtain a strategy chain; relaxing each strategy node in the strategy chain to obtain the corresponding reachable set; sampling in the reachable set to obtain the position corresponding to each strategy node to form an initial trajectory. The initial trajectory is smoothed to obtain the predicted trajectory of the vehicle.

Images