Multi-marshalling hinged vehicle BEV environment perception construction method

Abstract

The invention provides a construction method based on multi-marshalling hinged vehicle BEV environment perception. The method specifically comprises the following steps that 1, a multi-view camera is installed at the joint of multiple marshalling vehicles; 2, camera pose calibration in marshalling and between marshalling is completed; 3, angle pose transformation and association of cameras between marshalling are achieved; step 4, detecting a target captured by the multi-view camera by using a deep learning target detection network; step 5, obtaining a projection matrix calibrated by the multi-marshalling camera; step 6, projecting the detected target position information to the BEV sensing map, and constructing the BEV sensing map of the single-marshalling articulated vehicles; and step 7, according to the hinging angle information and the single-marshalling BEV perception map, splicing to generate a global BEV perception map of the multi-marshalling hinging vehicles. According to the method, efficient association of target sensing positions of multi-marshalling hinged vehicles and accurate splicing of a global BEV sensing map are realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a multi-camera BEV environment perception technology for multi-unit vehicles used in large-scale projects. Background Art

[0002] Currently, articulated multi-unit vehicles have a high risk of traffic accidents due to large blind spots. Existing BEV environmental perception technology is mainly used in small vehicles, and there is still a lack of end-to-end BEV perception methods for engineering vehicles. The current relevant patents are as follows:

[0003] The first category, such as Chinese invention patent 202210902286.7, proposes an environmental perception method, device, equipment and storage medium based on a multi-eye bird's-eye view perspective, which involves the field of intelligent driving technology. The main process of this method includes: inputting the image to be perceived generated by preprocessing into the environmental perception model; inputting the internal parameter data, external parameter data of the multi-eye sensor and the relevant data representing the online data enhancement strategy during the training of the environmental perception model into the model; using the environmental perception model to encode the image to be perceived, generate a numbered image, and perform matrix transformation on the sensor internal parameter, external parameter and data enhancement strategy related data to generate a heat map; further matrix transformation of the numbered image and heat map, calculate the voxel features based on the vehicle coordinate system; input the voxel features into the 3D target detection head, the drivable area segmentation head and the lane line segmentation head respectively, and obtain the multi-task reasoning results of environmental perception. Although this technology realizes multi-task environmental perception, its application scope is mainly for small cars and cannot effectively meet the needs of large vehicles.

[0004] The second category, such as Chinese invention patent 202310331701.2, proposes a method for calculating the drivable area from the perspective of BEV based on pixel position and camera coordinate system. The main steps of this method include: calculating the coordinates of the pixel position in the image coordinate system; deriving the plane equation of the ground plane in the camera coordinate system based on the plane equation of the ground plane in the target coordinate system and the external parameters of the camera; using the plane equation of the ground plane in the camera coordinate system, the camera internal parameters and the image coordinates of the pixel position, calculate the coordinates of the pixel position corresponding to the ground in the camera coordinate system; based on the coordinates of the ground pixel position in the camera coordinate system and the camera external parameters, deduce the coordinates of the ground position in the target coordinate system; combine the coordinates of the ground position in the target coordinate system and the pixel coordinates of the drivable area image to determine the drivable area from the perspective of BEV. This method can efficiently determine the drivable area from the perspective of BEV, but does not consider the problem of camera distortion, and has high performance requirements for hardware equipment.

[0005] The third category, such as Chinese invention patent 202310735397.8, proposes a visual BEV environment perception method for semi-trailers based on variable-angle fisheye cameras. The main steps of this method include: calibrating the pose of the cameras on the front and the car body; using the angle sensor to obtain the relative position relationship between the front and the car body, and completing the angle pose transformation and association of the cameras on the front and the car body; using a deep learning target detection network to detect targets in the fisheye cameras on the front and rear of the vehicle; projecting the target position information detected by a single fisheye camera into a top view; splicing the perception situation map of the front and rear cameras to generate a variable-angle perspective perception situation map of the semi-trailer. This method realizes the association of target perception positions of semi-trailers with angle rotation and the splicing of global BEV perception maps. However, it has limitations when dealing with semi-trailer turning scenarios.

[0006] The fourth category is China's invention patent 202311670801.4, a method for BEV perception and prediction of vehicle surrounding targets based on omnidirectional vision. This method constructs an end-to-end neural network based on BEV perception and prediction. The core feature is to explicitly convert features into polar coordinates for perception and prediction. The specific steps are as follows: annotate the three-dimensional data collected by the lidar and generate BEV bird's-eye view labels; build a model to extract two-dimensional features from the images collected by the camera, and convert these features into polar coordinates to generate BEV feature maps; fuse data from multiple cameras to generate BEV views in a unified reference coordinate system; predict the future frames of the target through a future prediction model based on the conditions of the current frame and historical frames; convert the prediction information from the polar coordinate system to the Cartesian coordinate system, and decode the semantic segmentation and future optical flow information through a multi-head decoder to output the final prediction result. This method achieves accurate perception and prediction of target position and behavior by fusing multi-source data and inter-frame information.

[0007] The fifth category is China's invention patent 202311183641.0, a method, device and technical solution based on BEV perception applied to vehicles. The specific process of the BEV perception method includes the following steps: acquiring image data and point cloud data of the vehicle's surrounding environment, extracting their features, and generating image features and point cloud features respectively; processing the image features and point cloud features, and fusing them to generate target BEV features, wherein the processing includes feature fusion and temporal feature enhancement; based on the target fused BEV features, performing target detection and road segmentation on the vehicle's surrounding environment, and outputting the BEV perception results of the vehicle's surrounding environment. This method achieves high-precision perception of the vehicle's surrounding environment through the effective fusion and enhancement of image and point cloud features, and has significant application value. Summary of the invention

[0008] This patent is designed to solve the perception problems caused by excessive distortion and blind spots of multi-cameras in special working conditions such as turning for engineering vehicles and articulated multi-unit vehicles. To this end, this patent designs a multi-camera BEV environment perception method to effectively eliminate the perception blind spots of multi-unit articulated vehicles on intelligent rails and improve the comprehensiveness and accuracy of vehicle environment perception.

[0009] The technical solution of the present invention makes full use of multi-cameras and realizes the camera pose inertia of the vehicle group through group calibration. Carry out camera pose calibration within and between groups to determine the pose parameters of each camera, and project the targets detected in the group camera into the top view through target detection, and then use the articulated angle sensor to realize the association and fusion of the perception results of different vehicle groups, thereby improving the global environment perception ability of multiple vehicles. The specific steps are as follows:

[0010] Step 1: Install the multi-eye camera on the articulated multi-unit vehicle;

[0011] Step 2: Carry out camera pose calibration within and between groups;

[0012] Step 2.1: By carrying out camera pose calibration within and between marshalings, the pose angle of the marshaling articulated vehicle caused by the superposition of rotation angles is accurately obtained, providing accurate pose estimation for BEV perception of articulated vehicles;

[0013] Step 2.2, perform BEV perception target detection of a single-unit articulated vehicle, and return the category and distance information of the target;

[0014] Step 3: Transform and associate camera angles and poses between groups;

[0015] Step 4: Use the deep learning target detection network to detect targets in the multi-camera;

[0016] Step 5: Project the detected target position information in the multi-camera into the BEV perception map to construct a single-unit articulated vehicle BEV perception map;

[0017] Step 6: Use the articulation angle information to associate and share the perception information of different vehicles.

[0018] Step 7: Construct a multi-group BEV perception map;

[0019] Step 8: Perform boundary prediction and semantic segmentation of the target, and project the detection and segmentation results into the BEV space.

[0020] Furthermore, in step 3, multi-coordinated articulated vehicles will face significant uncertainty in target perception by fisheye cameras and the construction of BEV perception maps relies on the fusion of panoramic fisheye cameras and ordinary rearview cameras. In addition, when fisheye cameras stitch images after correction, they are affected by the camera calibration posture parameters and the dynamic rotation of the coordinated vehicles, which will produce dynamic blind spots.

[0021] Furthermore, in step 4, simulation technology is used to simulate the turning process of the articulated multi-unit vehicle, so as to determine the optimal posture of the multi-eye fisheye camera and the electronic rearview mirror camera.

[0022] Furthermore, in step 5, target detection, edge prediction and semantic segmentation are performed in the severely distorted fisheye camera to overcome the interference of the large distortion of the fisheye camera on the stretching of the small target shape, improve the accuracy of small target detection in the fisheye camera and the rearview camera, and thus build a robust BEV projection perception map.

[0023] Furthermore, in step 7, the target in the camera can be mapped to the BEV perception map through the projection matrix. The target detected in the image is projected to the BEV map, and the boundary line where the target in the camera intersects with the ground is predicted, so as to calculate the distance of the target and perform the projection.

[0024] The beneficial effects of the present invention are:

[0025] (1) Different from the camera calibration of general passenger cars, multi-unit articulated vehicles need to simultaneously perform camera pose calibration within the unit and rotation transformation correction between units. Camera pose is the core parameter for constructing the BEV perception 2D projection. Therefore, accurate pose estimation of cameras in different units is crucial for generating high-quality BEV perception maps. However, the superposition of pose calibration errors within the unit and articulation rotation deviations between units often leads to significant distortion and blind spots in the BEV perception scene. To address this problem, the present invention provides a method that can provide accurate visual camera pose parameters for multi-camera target detection and BEV perception projection in multi-unit articulated vehicles.

[0026] (2) The 2D panoramic view and electronic rearview mirror of multi-unit vehicles have limited perception range, targets are prone to distortion, and the stitching blind area is large. In contrast, BEV perception not only expands the perception range by projecting farther targets into the top view, but also significantly reduces the blind area and improves the detection ability of moving obstacles. Therefore, the construction of the BEV map requires the long-distance target detection results of the panoramic fisheye camera and the electronic rearview mirror to be projected into the top view. However, due to the severe lens distortion of the fisheye camera, the deformation of the image edge and the distant target makes it difficult to accurately estimate the ground contact line through the traditional target detection model. At the same time, due to the high installation position of the electronic rearview mirror, there is also a high missed detection rate for the detection of small targets at a long distance in the field of view. In response to these problems, the present invention proposes a method for realizing target detection, classification and segmentation of fisheye cameras and electronic rearview mirrors, thereby providing accurate perception target projection information for constructing a large-viewing-angle, low-blind-area BEV map of multi-unit vehicles.

[0027] The accompanying drawings are as follows

[0028] Figure 1 Multi-unit articulated vehicle BEV perception flow chart DETAILED DESCRIPTION

[0029] This embodiment provides a method for evaluating the environment perception of an articulated multi-unit vehicle with a multi-camera BEV, as shown in the overall process, which specifically includes the following steps:

[0030] Step 1: Install the multi-camera on multiple vehicles;

[0031] Step 2: calibrate the camera of multiple articulated vehicles;

[0032] Step 2.1: By carrying out camera pose calibration within and between marshalings, the pose angle of the articulated vehicle in the marshaling caused by the superposition of rotation angles is accurately obtained, providing accurate pose estimation for BEV perception of articulated vehicles.

[0033] Step 2.2, perform BEV perception target detection of a single-unit articulated vehicle, and return the category and distance information of the target;

[0034] Step 3: Under special working conditions such as vehicle turning, simulate and generate a BEV perception map, and collect blind spot data of the fisheye camera;

[0035] BEV perception of multi-unit articulated vehicles faces two uncertainties: First, there is significant uncertainty in target perception between fisheye cameras and electronic rearview mirror cameras. Since electronic rearview mirror cameras are installed at a high position, the targets in their field of view are usually small; while distant targets in fisheye cameras are not only smaller in size, but also have serious distortion problems, which leads to a high missed detection rate in the target perception model. Secondly, the construction of BEV perception maps relies on the fusion of panoramic fisheye cameras and ordinary rearview cameras. However, when fisheye cameras perform image stitching after correction, dynamic blind spots will be generated due to the influence of camera calibration posture parameters and dynamic rotation of the unitized vehicles. This dynamic blind spot may pose a potential threat to the safety and reliability of BEV perception.

[0036] Step 4: Achieve accurate pose estimation of the multi-eye fisheye camera and the electronic rearview mirror camera in the dynamic articulated vehicle to optimize the inter-group camera pose error caused by the superposition of rotation angles;

[0037] The simulation technology is used to simulate the turning process of articulated multi-unit vehicles to determine the optimal positions of the multi-eye fisheye camera and the electronic rearview mirror camera.

[0038] Step 5: Use the distorted fisheye camera to detect the target;

[0039] Target detection, edge prediction and semantic segmentation are performed in fisheye cameras with severe distortion to overcome the interference of large distortion of fisheye cameras on the stretching of small target shapes, improve the accuracy of small target detection in fisheye cameras and rearview cameras, and thus build a robust BEV projection perception map.

[0040] Step 6: Obtain the projection matrix of the multi-group calibration;

[0041] Step 7: The detected target is projected onto the BEV map to obtain the distance of the target;

[0042] The target in the camera can be mapped to the BEV perception map through the projection matrix. In order to project the target detected in the image to the BEV map, it is necessary to obtain the distance between the target and the ground. The most direct method is to predict the boundary line where the target in the camera intersects with the ground, thereby calculating the distance of the target and projecting it.

[0043] Step 8: Perform boundary prediction and semantic segmentation of the target, and project the detection and segmentation results into the BEV space;

[0044] The distorted image of the fisheye camera causes the feature distribution to change, and the traditional rectangular convolution kernel cannot effectively describe the features of the distorted target; the electronic rearview mirror camera faces the problem that the perceived target viewing angle is different from that of an ordinary camera. These problems lead to uncertainty in the detection results of the perceived target, which in turn affects the accuracy of long-distance perception when the target in the camera is projected onto the BEV map. Therefore, for the target perception of multi-group multi-eye cameras, the present invention proposes a target detection method based on a rotating frame, and applies a variable convolution kernel to the target detection and segmentation model of the multi-eye camera. In addition, for the boundary prediction problem, an angled boundary prediction method is proposed, which significantly improves the accuracy of boundary prediction, thereby improving the accuracy of target detection, segmentation and distance prediction in multi-group BEV perception.

Claims

1. The present invention uses a multi-camera installed on an articulated multi-coordinate vehicle to realize BEV perception of the vehicle. Its characteristic is that the collaborative perception of multiple vehicles is realized through the end-to-end BEV perception and the association of the vehicle group information of the multi-camera. The system can obtain the articulation angle information of the articulated vehicle in real time, and transmit and share the BEV perception projection information between the vehicles based on the articulation angle, thereby realizing the active and collaborative perception of the vehicle group.

2. According to claim 1, the articulated multi-unit vehicle has active perception capabilities, which can detect the categories and distance information of moving pedestrians, vehicles and other obstacles around the vehicle, and project this information into the BEV perception map of the vehicle, providing key decision-making and control support for intelligent driving.

3. According to claim 1, the system also has a collaborative perception function, which realizes the dynamic transmission of camera posture through the marshaling articulation angle information, and integrates the perception information with the angle information to achieve dynamic collaborative perception of the marshaling vehicles. The articulation angle information is obtained in real time by the vehicle-mounted angle sensor and is used for the optimization and sharing of perception information.

4. According to claim 1, the system can directly map the type and distance information of the detected target into the top view to complete the BEV perception of the marshaled articulated vehicle. The target types include common obstacles such as pedestrians and vehicles, and the target distance information is predicted and calculated based on the geometric relationship between the detected target and the ground contact line.

5. The method for constructing environment perception of a multi-unit articulated vehicle BEV according to claim 2, characterized in that: The integrated model is not limited to YOLO, Faster-RCNN, SSD, and Transformer-based detection models.

Images