Unmanned aerial vehicle cooperative automatic driving vehicle BEV visual angle fusion method

Abstract

The invention discloses an unmanned aerial vehicle cooperative automatic driving vehicle BEV visual angle fusion method, and relates to the technical field of automatic driving. The method specifically includes controlling the unmanned aerial vehicle and the target vehicle to keep synchronous movement; acquiring first image data, point cloud data and second image data at the same moment, and converting the first image data, the point cloud data and the second image data into points in a world coordinate system; respectively carrying out feature extraction on the data, and correspondingly generating an image feature graph of a BEV view angle, a BEV view angle graph with height information and an RGB image feature graph; and constructing a shallow neural network to fuse the BEV view angle image with height information, the BEV view angle image feature map and the RGB image feature map into a uniformly represented multi-modal feature map, so that the autonomous vehicle adapts to a complex meteorological environment. Therefore, the self-driving automobile adapts to a complex meteorological environment, and the perception capability of the self-driving automobile to the environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for integrating the perspectives of an autonomous driving vehicle (BEV) in collaboration with a drone. Background Art

[0002] Autonomous driving technology mainly includes three core links: environmental perception, trajectory prediction and decision planning. In actual autonomous driving application scenarios, the vehicle first uses the environmental perception system to capture dynamic and static information about the surrounding environment, including the historical driving trajectories of other vehicles, real-time changing environmental factors, stationary obstacles, and road signs. By combining these perception data with high-precision maps, the vehicle can build a detailed scene understanding model. Next, the trajectory prediction system predicts the possible movement trajectories of other vehicles and intelligent agents in the short term in the future based on this scene information. Finally, the decision planning system comprehensively uses the results of environmental perception and trajectory prediction to reasonably plan the driving route and behavior of the autonomous driving vehicle to ensure safe and efficient driving.

[0003] There are currently two main technical routes in the environmental perception technology of intelligent vehicles. One is based on cameras and adopts a pure visual perception solution; the other is a multi-sensor fusion perception technology solution that mainly uses lidar and supplemented by other sensors. Both methods perceive the surrounding environment from the perspective of the vehicle itself. However, due to the limitations of the perspective and the principles of the sensor itself and its perception range, these methods can only perceive unobstructed objects within a certain range, and it is difficult to effectively perceive obstructed or distant objects. This results in the scene information reproduced by the perception system alone being incomplete, and the accuracy needs to be improved.

[0004] In order to solve this problem, the prior art discloses a method for collecting environmental information for self-driving cars on public roads. The method uses a drone to accompany the test vehicle for a long time and steadily, uses the drone's onboard environmental information collection equipment to collect road information ahead, pre-processes the collected video frames, and performs lane line inspection, pedestrian target detection and tracking, and vehicle target detection and tracking on the pre-processed video frames. Although it effectively avoids the situation where the vehicle's own sensor's perception range is not far enough and is blocked by road obstacles and other traffic participants, it is a simple collection of environmental information and target detection and tracking based on traditional algorithms. It uses a single modality (visual) information processing, which is easily affected by weather and other conditions and cannot be applied to complex meteorological environments.

[0005] Therefore, how to make autonomous vehicles adapt to complex meteorological environments and improve their ability to perceive the environment has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The main purpose of the present invention is to provide a UAV-coordinated autonomous vehicle BEV perspective fusion method, which aims to enable the autonomous vehicle to adapt to complex meteorological environments and improve the autonomous vehicle's ability to perceive the environment.

[0007] In order to achieve the above object, the present invention proposes a method for fusion of perspectives of an autonomous driving vehicle (BEV) in cooperation with a drone, comprising the following steps: Controlling the UAV and the target vehicle to keep synchronous movement, and making the camera of the UAV collect first image data, the on-board radar collect point cloud data, and the on-board camera collect second image data; Acquire first image data, point cloud data, and second image data at the same time, construct a world coordinate system according to the vehicle-mounted camera, convert the first image data into first standard image data in the world coordinate system, convert the point cloud data into point cloud standard data in the world coordinate system, and convert the second image data into second standard image data in the world coordinate system; Performing feature extraction on the first standard image data, the point cloud standard data, and the second standard image data, respectively, to generate a BEV perspective image feature map, a BEV perspective map with height information, and an RGB image feature map; A shallow neural network is constructed to fuse the BEV perspective map with high-degree information, the BEV perspective image feature map, and the RGB image feature map into a multimodal feature map with unified representation, so that the autonomous driving vehicle can adapt to complex meteorological environments.

[0008] In one embodiment of the present application, controlling the UAV and the target vehicle to maintain synchronous movement includes the following steps: The UAV is defined as a rover station, the target vehicle is defined as a reference station, and the precise coordinates of the UAV are obtained by using real-time dynamic carrier phase differential positioning; The UAV is controlled to keep synchronous movement with the target vehicle according to the precise coordinates.

[0009] In one embodiment of the present application, after controlling the UAV and the target vehicle to move synchronously according to the precise coordinates, it also includes obtaining first image data collected by the UAV, obtaining the target vehicle reference point position according to the first image data, and determining whether the reference point position is the same as a preset reference point. If they are the same, it indicates that the position between the UAV and the target vehicle is accurate.

[0010] In one embodiment of the present application, acquiring first image data, point cloud data, and second image data at the same time comprises the following steps: Adding timestamps to the first image data collected by the drone's camera, the point cloud data collected by the vehicle-mounted radar, and the second image data collected by the vehicle-mounted camera; The time stamps of the first image data collected by the drone camera, the point cloud data collected by the vehicle radar, and the second image data collected by the vehicle camera are accurately aligned through time synchronization technology; The first image data, the point cloud data, and the data corresponding to the preset timestamp in the second image data are acquired according to the preset timestamp.

[0011] In one embodiment of the present application, the first image data is converted into the first standard image data in the world coordinate system, and the second image data is converted into the second standard image data in the world coordinate system, and the conversion formula is as follows: ; The image pixel coordinate system is (u, v); the world coordinate system is (X W , Y W , Z W ); the pixel unit length size is d x and d y ; The pixel coordinates of the principal point are (u0, v0); f is the normalized focal length; R is the rotation matrix; T is the translation vector.

[0012] In one embodiment of the present application, the point cloud data is converted into point cloud standard data in the world coordinate system, and the conversion formula is as follows: ; Among them, (x c ,y c ,z c ) is the coordinate of the point under the target vehicle coordinates; (x l ,y l ,z l ) is the coordinate of the point under the laser radar; R is the rotation matrix; T is the translation matrix; ; in, is the heading angle ; is the pitch angle ; Roll angle ; ; Among them, (x, y, z) is the three-dimensional coordinate of the origin of the target vehicle coordinate system.

[0013] In one embodiment of the present application, extracting features from the first standard image data and the second standard image data includes the following steps: downsampling the first standard image data and the second standard image data; Performing feature extraction on the downsampled first standard image data and the second standard image data through a convolutional neural network; After the feature extraction is completed, the multi-scale image in the first standard image data and the multi-scale image in the second standard image data are fused respectively through the feature pyramid network to generate the image feature map of the BEV perspective and the RGB image feature map.

[0014] In one embodiment of the present application, feature extraction of point cloud standard data includes the following steps: voxelize point cloud standard data; The voxelized point cloud standard data grid is column-encoded, converted into a pseudo-image and high-dimensional features are extracted; A feature pyramid network is used to fuse multi-scale images with high-dimensional features.

[0015] In one embodiment of the present application, when constructing a shallow neural network, the shallow neural network is trained through regression and classification losses.

[0016] In one embodiment of the present application, before fusing the BEV viewing angle map with height information, the BEV viewing angle image feature map, and the feature map of the RGB image into a unified multimodal feature map, the method further includes: The BEV perspective map with height information, the BEV perspective image feature map, and the RGB image feature map are reduced in dimension through principal component analysis.

[0017] By adopting the above technical solution, through the fusion of multimodal sensor data, the system can obtain all-round scene information around the vehicle, comprehensively analyze environmental changes from a bird's-eye view and a first-person perspective, and ensure the perception ability of autonomous vehicles in complex scenes. The multimodal feature map that integrates point cloud and image data improves the adaptability of the system under complex meteorological conditions and reduces the risk of single sensor failure. By using a unified transformation of the world coordinate system, the spatial consistency of different sensor data is ensured, and redundant information is removed through dimensionality reduction methods such as PCA, which improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 It is a schematic structural diagram of the first embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the following specific embodiments are only used to explain the present invention and do not constitute a limitation to the present invention.

[0020] like Figure 1 As shown, in order to achieve the above purpose, the present invention proposes a UAV-coordinated autonomous driving vehicle BEV perspective fusion method, comprising the following steps: Controlling the UAV and the target vehicle to keep synchronous movement, and making the camera of the UAV collect first image data, the on-board radar collect point cloud data, and the on-board camera collect second image data; Acquire first image data, point cloud data, and second image data at the same time, construct a world coordinate system according to the vehicle-mounted camera, convert the first image data into first standard image data in the world coordinate system, convert the point cloud data into point cloud standard data in the world coordinate system, and convert the second image data into second standard image data in the world coordinate system; Performing feature extraction on the first standard image data, the point cloud standard data, and the second standard image data, respectively, to generate a BEV perspective image feature map, a BEV perspective map with height information, and an RGB image feature map; A shallow neural network is constructed to fuse the BEV perspective map with high-degree information, the BEV perspective image feature map, and the RGB image feature map into a multimodal feature map with unified representation, so that the autonomous driving vehicle can adapt to complex meteorological environments.

[0021] Specifically, in the operation of the system, it is necessary to first realize the synchronous motion control of the UAV and the target vehicle. The control module of the UAV is located on the target vehicle, and the control module of the UAV communicates with the vehicle-mounted system in real time to realize the flight control of the UAV, so that the UAV maintains the relative position with the target vehicle, ensuring that the UAV and the target vehicle can collect data at the same time and space range.

[0022] While the drone is maintaining a stable flight, its camera collects image data of the vehicle and its surroundings from a bird's-eye view. This angle is mainly used to capture a large range of scenes around the vehicle, providing a bird's-eye view. The on-board LiDAR obtains high-precision point cloud data by scanning the surrounding environment. This data contains three-dimensional information about objects around the vehicle for accurate spatial perception. The on-board camera mainly collects image data from the first-person perspective of the vehicle to capture details of the road and environment ahead. The image data from this perspective complements the drone's bird's-eye view data, providing information from multiple angles.

[0023] In order to ensure the consistency of multimodal sensor data, the system must first synchronize the time and convert the spatial coordinates of the data from the drone camera, vehicle radar and vehicle camera.

[0024] When synchronizing data, time synchronization technologies such as NTP (Network Time Protocol) are used to ensure that the three sensors collect data at the same time. This step ensures that in the subsequent fusion process, the data can be accurately aligned to reflect the environmental status at the same time point.

[0025] A unified world coordinate system is constructed based on the onboard camera, with the vehicle as the reference. This is the core coordinate system of the entire system, which is used to uniformly describe the position and form of multimodal data.

[0026] The bird's-eye view image data of the drone is mapped to the world coordinate system through coordinate transformation to obtain the first standard image data. The point cloud data of the vehicle-mounted radar is also converted to the point cloud standard data in the world coordinate system to ensure the spatial consistency of data from different sensors. The image data of the vehicle-mounted camera is also converted to generate the second standard image data consistent with the world coordinate system.

[0027] After data normalization, it is necessary to extract features from data of different modalities so that they can be fused in subsequent steps.

[0028] Extract features from the bird's eye view image data collected by drones to generate BEV (Bird's Eye View) image feature maps. Use advanced image processing methods such as convolutional neural networks (such as ResNet) to extract key features in the image.

[0029] For the point cloud data collected by the vehicle-mounted radar, the geometric details are retained through voxelization, and the voxel grids are spliced ​​into columns and encoded to generate a pseudo image. The spatial features of the pseudo image are extracted through 2D convolution to generate a BEV perspective feature map with height information.

[0030] Extract the features of the RGB image from the first-person perspective image of the on-board camera to generate an RGB image feature map. This process can use a multi-layer convolutional neural network to capture the details and texture information in the image.

[0031] After feature extraction is completed, the system fuses the multimodal features by constructing a shallow neural network. A shallow neural network is constructed to directly learn the mapping from one feature space to another, ensuring that features from different sensors are uniformly expressed in the same space. The shallow neural network is trained through regression or classification loss functions so that it can fuse the BEV point cloud map with height information, the BEV image feature map, and the RGB image feature map into a unified multimodal feature map. Ultimately, the system generates a unified representation of the multimodal feature map that can cover all-round scene information around the vehicle and is suitable for downstream autonomous driving tasks such as target detection, path planning, and prediction.

[0032] By adopting the above technical solution, through the fusion of multimodal sensor data, the system can obtain all-round scene information around the vehicle, comprehensively analyze environmental changes from a bird's-eye view and a first-person perspective, and ensure the perception ability of autonomous vehicles in complex scenes. The multimodal feature map that integrates point cloud and image data improves the adaptability of the system under complex meteorological conditions and reduces the risk of single sensor failure. By using a unified transformation of the world coordinate system, the spatial consistency of different sensor data is ensured, and redundant information is removed through dimensionality reduction methods such as PCA, which improves computing efficiency.

[0033] In one embodiment of the present application, controlling the UAV and the target vehicle to maintain synchronous movement includes the following steps: The UAV is defined as a rover station, the target vehicle is defined as a reference station, and the precise coordinates of the UAV are obtained by using real-time dynamic carrier phase differential positioning; The UAV is controlled to keep synchronous movement with the target vehicle according to the precise coordinates.

[0034] Specifically, the UAV is defined as the mobile station and the target vehicle as the base station.

[0035] The positioning system on the target vehicle acts as a base station, providing high-precision position information. The task of the base station is to be fixed at a known position and send differential correction data to the rover (drone). These correction data are calculated based on the difference between the known position and the measured position, thereby improving the positioning accuracy of the rover.

[0036] As a mobile station, the drone calculates its precise current position based on the differential correction data sent by the base station. Real-time dynamic carrier phase differential positioning (RTK) technology is the core of this step. It uses the carrier phase observations provided by the Global Navigation Satellite System (GNSS) to improve positioning accuracy through differential calculation.

[0037] The drone works with the GNSS system on the target vehicle. The base station (target vehicle) receives satellite signals and calculates the precise absolute position, and then sends the error information to the rover (drone) via wireless communication. The rover calculates the precise position coordinates by comparing the received satellite signals with the error information sent by the base station.

[0038] The control system of the drone receives the position information from the base station (target vehicle) in real time and calibrates it based on its own position information. By adjusting the flight parameters (such as speed, position, direction, etc.) in real time, the drone is ensured to always maintain a stable relative position with the target vehicle.

[0039] The longitudinal flight trajectory of the drone is achieved by tracking the driving trajectory of the vehicle. The longitudinal speed of the drone is adjusted according to the real-time speed of the vehicle to ensure that the two are always synchronized. The drone calculates the lane direction of the target vehicle to ensure that it flies along the lane normal direction. The drone minimizes lateral displacement to avoid excessive deviation from the vehicle and ensure the stability of image data and point cloud data collection. The drone maintains a relatively constant flight altitude to avoid visual changes caused by frequent altitude adjustments and ensure a consistent perspective during data collection.

[0040] With the above technical solution, the UAV can always maintain a stable relative position and attitude with the target vehicle through the centimeter-level accuracy provided by RTK technology, ensuring that the viewing angle and distance will not shift when collecting environmental data, which helps to generate accurate 3D scene reconstruction.

[0041] In one embodiment of the present application, after controlling the UAV and the target vehicle to move synchronously according to the precise coordinates, it also includes obtaining first image data collected by the UAV, obtaining the target vehicle reference point position according to the first image data, and determining whether the reference point position is the same as a preset reference point. If they are the same, it indicates that the position between the UAV and the target vehicle is accurate.

[0042] Specifically, the drone's camera continuously captures images of the vehicle and its surroundings during flight to ensure that the vehicle's position and posture on the road are captured. The images collected by the drone are preprocessed, such as image denoising, color correction, image enhancement, etc., to ensure the accuracy of subsequent feature extraction and reference point recognition.

[0043] Once the drone collects the first image data, the system will use image processing algorithms to extract the reference point position of the target vehicle from these images. The reference point is a certain part of the vehicle with obvious characteristics, such as a certain mark on the roof, a fixed position on the parking platform, or a key feature point of the vehicle structure. This point is pre-defined and used to monitor the relative position between the drone and the vehicle.

[0044] The key feature points of the target vehicle are extracted from the image through computer vision algorithms. Then, the preset reference point positions are found through specific detection algorithms. The detection process can use the geometric shape, color, texture and other features of the reference points for identification and positioning.

[0045] After successfully acquiring the actual position of the vehicle's reference point, the system compares the actual reference point coordinates with the preset standard reference point coordinates. Due to environmental changes and sensor accuracy, the system needs to set a certain tolerance range. If the position deviation of the reference point is within the tolerance range, the system can assume that the relative position between the drone and the vehicle is accurate; if it exceeds the preset tolerance range, it indicates that the position relationship between the drone and the vehicle is inaccurate, and the system may need to be adjusted. If the actual reference point position is the same as the preset reference point, it means that the position between the drone and the vehicle is accurate and no further adjustment is required. If the system determines that there is a significant deviation between the reference point position and the preset reference point, the system will issue a control command to adjust the drone's flight attitude or position until the relative position between the drone and the target vehicle reaches the required accuracy.

[0046] Using the above technical solution, the system can accurately calibrate the relative position of the drone and the target vehicle based on visual feedback through image recognition technology. This image-based benchmark calibration method can supplement the shortcomings of GPS / RTK positioning and further improve the accuracy of relative position.

[0047] In one embodiment of the present application, acquiring first image data, point cloud data, and second image data at the same time comprises the following steps: Adding timestamps to the first image data collected by the drone's camera, the point cloud data collected by the vehicle-mounted radar, and the second image data collected by the vehicle-mounted camera; Through time synchronization technology, the timestamps of the first image data collected by the drone camera, the point cloud data collected by the vehicle-mounted radar, and the second image data collected by the vehicle-mounted camera are accurately aligned; The first image data, the point cloud data, and the data corresponding to the preset timestamp in the second image data are acquired according to the preset timestamp.

[0048] Specifically, during the flight of the drone, the camera it carries will continuously collect image data of the vehicle and its surroundings. These image data are called first image data. During the acquisition process, each frame of the image is accompanied by a timestamp of the acquisition time. The on-board radar is responsible for collecting three-dimensional spatial information around the vehicle and outputs point cloud data. Point cloud data is output by frame or time interval, so each point cloud frame will also be stamped with a corresponding timestamp. The on-board camera is installed on the vehicle and is responsible for collecting RGB images in front of or around the vehicle. These image data are also timestamped to indicate the specific time of image acquisition.

[0049] The system will set up a timestamp marking mechanism for the data stream of each sensor. The timestamp is marked by the built-in clock of the hardware system where the sensor is located, ensuring that each frame of image or each set of point cloud data can be accurately recorded when it is collected.

[0050] In order to ensure that the image data of the drone camera, the point cloud data of the vehicle-mounted radar and the image data of the vehicle-mounted camera are at the same time, time synchronization technology is needed. The synchronization technology used is NTP (Network Time Protocol), which can ensure that the data timestamps of all sensors are unified under a global clock framework.

[0051] When the system starts, the various sensors of the drone and vehicle are time-calibrated through the Network Time Protocol (NTP). In this way, the sensor times of the drone and vehicle can be aligned, ensuring that data collection is carried out under a unified time standard.

[0052] During the data collection process, the system will continuously detect the time synchronization of each sensor. If it detects that the timestamp error exceeds the predetermined range (such as a few milliseconds), the system will adjust the clocks of each sensor through a feedback mechanism to realign them.

[0053] When acquiring sensor data, the system will extract the data frame of the corresponding time from the first image data of the drone camera, the point cloud data of the vehicle radar, and the second image data of the vehicle camera according to the preset timestamp. The preset timestamp can be a key moment set in the mission planning, or it can be automatically generated according to the system processing speed.

[0054] With the above technical solution, since the various sensors of drones and vehicles may not collect data at exactly the same time, the precise alignment of timestamps ensures that the data frames of different sensors correspond to the same time point. This can avoid data deviations caused by time differences between sensors and ensure the accuracy of subsequent data fusion.

[0055] In one embodiment of the present application, the first image data is converted into the first standard image data in the world coordinate system, and the second image data is converted into the second standard image data in the world coordinate system, and the conversion formula is as follows: ; The image pixel coordinate system is (u, v); the world coordinate system is (X W , Y W , Z W ); the pixel unit length size is d x and d y ; The pixel coordinates of the principal point are (u0, v0); f is the normalized focal length; R is the rotation matrix; T is the translation vector.

[0056] Using the above technical solution, the formula can accurately map the image data to the world coordinate system by converting the image coordinates and the world coordinates. Through this method, the position of each pixel captured by the camera in the real world can be accurately determined, providing accurate geometric information for subsequent data processing. Each point in the image is associated with a point in the real world through the focal length and image sensor parameters (such as pixel size and principal point position) to ensure the consistency of image information with the physical environment.

[0057] In one embodiment of the present application, the point cloud data is converted into point cloud standard data in the world coordinate system, and the conversion formula is as follows: ; Among them, (x c ,y c ,z c ) is the coordinate of the point under the target vehicle coordinates; (x l ,y l ,z l ) is the coordinate of the point under the laser radar; R is the rotation matrix; T is the translation matrix; ; in, is the heading angle ; is the pitch angle ; Roll angle ; ; Among them, (x, y, z) is the three-dimensional coordinate of the origin of the target vehicle coordinate system.

[0058] By adopting the above technical solution, the point cloud data can be accurately converted from the radar coordinate system to the world coordinate system of the target vehicle by using the rotation matrix R and the translation matrix T. This conversion method fully considers the heading angle (yaw), pitch angle (pitch) and roll angle (roll) of the vehicle, ensuring that the point cloud data can still maintain accurate consistency with the coordinates of the vehicle at different angles and positions.

[0059] In one embodiment of the present application, extracting features from the first standard image data and the second standard image data includes the following steps: downsampling the first standard image data and the second standard image data; Performing feature extraction on the downsampled first standard image data and the second standard image data through a convolutional neural network; After the feature extraction is completed, the multi-scale image in the first standard image data and the multi-scale image in the second standard image data are fused respectively through the feature pyramid network to generate the image feature map of the BEV perspective and the RGB image feature map.

[0060] Specifically, the first standard image data collected by the drone and the second standard image data collected by the vehicle-mounted camera are downsampled. The main purpose of this step is to reduce the resolution of the image, thereby reducing the computational complexity and reducing the computational resource consumption during subsequent feature extraction.

[0061] In actual operation, you can choose to reduce the width and height of the image by a certain ratio (for example, to half or one quarter of the original image size) to reduce the amount of data while ensuring that sufficient image details are retained to support subsequent feature extraction.

[0062] The convolutional neural network (CNN) is used to extract features from the downsampled image. The convolutional neural network can effectively capture local patterns and global features in the image and extract image features at different levels.

[0063] This application uses a classic deep convolutional neural network architecture, such as ResNet. The advantage of ResNet is that it solves the problem of gradient vanishing in deep networks by introducing residual connections, and can extract richer features. Through the processing of multiple convolutional layers, feature maps with different depths are generated. The feature maps of each layer describe different information of the image, such as edges, shapes, textures, etc.

[0064] After feature extraction is completed, the extracted multi-scale image features are fused using the Feature Pyramid Network (FPN). The function of FPN is to combine features of different levels in the image to generate multi-scale feature representations, so that the model can better handle objects of different sizes in the image.

[0065] FPN fuses image features through a bottom-up path and a top-down path. The bottom-up path extracts features layer by layer, and the top-down path upsamples and fuses low-resolution features step by step to eventually generate a richer feature map. This process can enhance the ability to recognize objects of different sizes in the image. The feature pyramid network generates image feature maps and RGB image feature maps for BEV perspectives for images of different scales of the first standard image data and the second standard image data.

[0066] After FPN processing, two important feature maps are finally generated: one is the image feature map based on the BEV perspective collected by the drone, and the other is the RGB image feature map based on the vehicle-mounted camera. The BEV perspective image feature map can provide a top-down bird's-eye view, while the RGB image feature map provides support for traditional driving perspective information.

[0067] The image feature map from the BEV perspective can provide a more comprehensive environmental perspective due to the advantages of the drone's perspective, especially in target detection and scene perception, which can more accurately determine the position and movement of objects. The RGB image feature map is more used for traditional image analysis, such as forward obstacle detection and traffic light recognition.

[0068] By adopting the above technical solution, image features of different scales can be fused by using a feature pyramid network, which enables the system to effectively handle objects of different sizes. For autonomous driving, objects on the road (such as vehicles, pedestrians, obstacles, etc.) exist in various scales, and FPN can better cope with this diversity and enhance the detection capabilities of small and distant targets. At the same time, the solution combines the bird's-eye view perspective (BEV) of the drone and the conventional driving perspective (RGB) of the on-board camera, and provides a full range of scene perception capabilities through the fusion of multi-perspective information. The BEV perspective can provide a global view around the vehicle, which is helpful for accurate path planning and environmental monitoring, while the RGB perspective provides detailed forward perspective information. The combination of the two enables the autonomous driving system to perceive the surrounding environment more accurately in complex environments, improving the accuracy and robustness of the perception system.

[0069] In one embodiment of the present application, feature extraction of point cloud standard data includes the following steps: voxelize point cloud standard data; The voxelized point cloud standard data grid is column-encoded, converted into a pseudo-image and high-dimensional features are extracted; A feature pyramid network is used to fuse multi-scale images with high-dimensional features.

[0070] Specifically, voxelization is to discretize continuous point cloud data and divide the point cloud into fixed-size 3D grids or voxels according to a predetermined resolution. The points contained in each voxel will be aggregated to generate a feature value representing the entire voxel.

[0071] During implementation, it is necessary to set the resolution size of a voxel grid. The higher the resolution, the fewer points in each voxel, but the overall feature information is retained more finely; while lower resolution will lose some detail information, but the computational cost is lower. In practical applications, the resolution and computational complexity can be balanced according to the requirements of the vehicle driving scene. The purpose of voxelization is to regularize the sparse point cloud data to facilitate subsequent convolution operations.

[0072] For the voxelized point cloud standard data, the idea of ​​the Pillar Feature Net (PFN) module of the PointPillars model is used to splice the voxelized grid on the z-axis to form a columnar code, which is then converted into a pseudo-image format and 2D convolution is performed to extract high-dimensional features.

[0073] The Feature Pyramid Network (FPN) fuses multi-scale features, allowing the system to simultaneously use features at different resolutions for perception, thereby improving the ability to detect objects of different sizes. The core idea of ​​the Feature Pyramid Network is to upsample and downsample the feature maps generated by multi-layer convolution, and combine high-resolution detail information with low-resolution global information through jump connections, and finally generate a fused multi-scale feature map.

[0074] By adopting the above technical solution, point cloud data is converted from sparse, irregular three-dimensional space to regular two-dimensional pseudo-image through voxelization and columnar encoding. This process can effectively reduce the complexity of point cloud data, reduce the data from three-dimensional to two-dimensional, and perform columnar encoding after voxelization, reducing the complexity of subsequent convolution, thereby greatly improving the computational efficiency and maintaining good feature extraction accuracy. At the same time, voxelization allows the detailed information in the point cloud to be retained, avoiding the loss of too much information.

[0075] In one embodiment of the present application, when constructing a shallow neural network, the shallow neural network is trained through regression and classification losses.

[0076] Specifically, regression loss is used to predict continuous values, such as the specific position and distance of an object. In autonomous driving scenarios, regression loss can be used to accurately estimate the spatial position or motion trajectory of targets such as vehicles and pedestrians. Classification loss is used for category recognition in target detection to help the model distinguish different types of targets (such as vehicles, pedestrians, obstacles, etc.). By using regression and classification losses at the same time, shallow neural networks can learn two types of tasks at the same time, which not only improves the model's feature expression ability, but also enhances its adaptability and solves multi-task problems in complex scenarios.

[0077] In one embodiment of the present application, before fusing the BEV viewing angle map with height information, the BEV viewing angle image feature map, and the feature map of the RGB image into a unified multimodal feature map, the method further includes: The BEV perspective map with height information, the BEV perspective image feature map, and the RGB image feature map are reduced in dimension through principal component analysis.

[0078] Specifically, principal component analysis (PCA) is a commonly used dimensionality reduction algorithm that can reduce the dimension of data while retaining the main information. The specific steps include: 1. Convert the original high-dimensional feature map into a matrix representation. 2. Calculate the covariance matrix of the feature map. 3. Calculate the eigenvalues ​​and eigenvectors of the covariance matrix. 4. Select the principal component with the largest eigenvalue, perform data projection, compress the original data dimension, retain most of the useful information and remove redundancy.

[0079] Through these steps, the BEV view map, BEV feature map and RGB feature map are all reduced in dimension to obtain a low-dimensional feature map that is smaller in size but still retains most of the key information.

[0080] By adopting the above technical solution, the feature graph after dimensionality reduction can reduce the processing complexity while maintaining key information, so that the system can process data more quickly and meet the high speed and response time requirements of real-time applications such as autonomous driving. The original high-dimensional data contains a lot of redundant information, which not only increases the computational burden, but also may interfere with subsequent feature fusion. Through PCA, this redundant information can be removed and the most important principal components for the task can be retained.

[0081] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for fusion of BEV perspectives in cooperation with drones, characterized in that: The following steps are involved: Controlling the UAV and the target vehicle to keep synchronous movement, and making the camera of the UAV collect first image data, the on-board radar collect point cloud data, and the on-board camera collect second image data; Acquire first image data, point cloud data, and second image data at the same time, construct a world coordinate system according to the vehicle-mounted camera, convert the first image data into first standard image data in the world coordinate system, convert the point cloud data into point cloud standard data in the world coordinate system, and convert the second image data into second standard image data in the world coordinate system; Performing feature extraction on the first standard image data, the point cloud standard data, and the second standard image data, respectively, to generate a BEV perspective image feature map, a BEV perspective map with height information, and an RGB image feature map; A shallow neural network is constructed to fuse the BEV perspective map with high-degree information, the BEV perspective image feature map, and the RGB image feature map into a multimodal feature map with unified representation, so that the autonomous driving vehicle can adapt to complex meteorological environments.

2. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: Controlling the UAV and the target vehicle to keep moving synchronously includes the following steps: The UAV is defined as a rover station, the target vehicle is defined as a reference station, and the precise coordinates of the UAV are obtained by using real-time dynamic carrier phase differential positioning; The UAV is controlled to keep synchronous movement with the target vehicle according to the precise coordinates.

3. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 2, characterized in that: After controlling the UAV and the target vehicle to move synchronously according to the precise coordinates, the method also includes obtaining first image data collected by the UAV, obtaining the reference point position of the target vehicle according to the first image data, and determining whether the reference point position is the same as a preset reference point. If they are the same, it indicates that the position between the UAV and the target vehicle is accurate.

4. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: Acquiring the first image data, the point cloud data, and the second image data at the same time comprises the following steps: Adding timestamps to the first image data collected by the camera of the drone, the point cloud data collected by the vehicle-mounted radar, and the second image data collected by the vehicle-mounted camera respectively; The time stamps of the first image data collected by the drone camera, the point cloud data collected by the vehicle radar, and the second image data collected by the vehicle camera are accurately aligned through time synchronization technology; The first image data, the point cloud data, and the data corresponding to the preset timestamp in the second image data are acquired according to the preset timestamp.

5. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: The first image data is converted into the first standard image data in the world coordinate system, and the second image data is converted into the second standard image data in the world coordinate system, and the conversion formula is as follows: ； The image pixel coordinate system is (u, v); the world coordinate system is (X W , Y W , Z W ); the pixel unit length size is d x and d y ; The pixel coordinates of the principal point are (u0, v0); f is the normalized focal length; R is the rotation matrix; T is the translation vector.

6. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: The point cloud data is converted into point cloud standard data in the world coordinate system. The conversion formula is as follows: ； Among them, (x c ,y c ,z c ) is the coordinate of the point under the target vehicle coordinates; (x l ,y l ,z l ) is the coordinate of the point under the laser radar; R is the rotation matrix; T is the translation matrix; ； in, is the heading angle ; is the pitch angle ; Roll angle ; ； Among them, (x, y, z) is the three-dimensional coordinate of the origin of the target vehicle coordinate system.

7. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: Extracting features from the first standard image data and the second standard image data comprises the following steps: downsampling the first standard image data and the second standard image data; Performing feature extraction on the downsampled first standard image data and the second standard image data through a convolutional neural network; After the feature extraction is completed, the multi-scale image in the first standard image data and the multi-scale image in the second standard image data are fused respectively through the feature pyramid network to generate the image feature map of the BEV perspective and the RGB image feature map.

8. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: Feature extraction of point cloud standard data includes the following steps: voxelize point cloud standard data; The voxelized point cloud standard data grid is column-encoded, converted into a pseudo-image and high-dimensional features are extracted; A feature pyramid network is used to fuse multi-scale images with high-dimensional features.

9. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: When building a shallow neural network, the shallow neural network is trained through regression and classification losses.

10. The method for fusion of perspectives of autonomous driving vehicles (BEVs) coordinated by drones as claimed in claim 1, characterized in that: Before fusing the BEV view map with height information, the BEV view image feature map, and the feature map of the RGB image into a multimodal feature map with unified representation, the following steps are also included: The BEV perspective map with height information, the BEV perspective image feature map, and the RGB image feature map are reduced in dimension through principal component analysis.

Images