Vehicle perception optimization method and device based on unmanned aerial vehicle, vehicle and storage medium

Abstract

The invention discloses a vehicle perception optimization method and device based on an unmanned aerial vehicle, a vehicle and a storage medium, and relates to the technical field of vehicle information processing, and the method comprises the steps: obtaining first environment data collected by an unmanned aerial vehicle sensing module and second environment data collected by a vehicle end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the unmanned aerial vehicle and the vehicle; performing cross-view feature analysis according to the relative pose transformation information, the first environment data and the second environment data to obtain fusion perception features; and generating a vehicle environment perception result based on the fused perception features. According to the method, the dynamic pose estimation between the unmanned aerial vehicle and the vehicle is combined with the feature pre-fusion architecture, and the fusion obstacle of inconsistent space-time coordinate systems between dynamic platforms is overcome, so that the cross-view feature-level pre-fusion of the perception data of the unmanned aerial vehicle and the vehicle is realized, and the perception precision and the perception view range are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle information processing, in particular to a vehicle perception optimization method and device based on a UAV, a vehicle and a storage medium. BACKGROUND

[0002] Traditional vehicle perception is limited by the range of vision, and there are often blind spots at bends, and it is easy to be blocked by obstacles, forming a dead angle of perception. At present, unmanned aerial vehicle cameras are generally used to improve the vehicle's perception ability of the surrounding environment.

[0003] However, in the existing technology of assisting vehicle operation with UAV data, a target-level post-fusion strategy is mostly adopted, which leads to a slow fusion process of UAV data and vehicle data, and further causes high target missing detection rate and insufficient perception accuracy.

[0004] Therefore, how to effectively improve the perception accuracy and perception field of view of the vehicle based on UAV data has become a difficult problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide a vehicle perception optimization method and device based on a UAV, a vehicle and a storage medium, which aims to solve the technical problem of how to effectively improve the perception accuracy and perception field of view of the vehicle based on UAV data.

[0006] To achieve the above-mentioned purpose, the present application provides a vehicle perception optimization method based on a UAV, which comprises: acquiring first environment data collected by a UAV sensing module and second environment data collected by a vehicle-end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the UAV; performing cross-view angle feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fusion perception features; generating a vehicle environment perception result based on the fusion perception features.

[0007] In an embodiment, the step of performing cross-view angle feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fusion perception features comprises: performing image feature analysis on the first environment data and the second environment data to obtain first image features corresponding to the UAV and second image features corresponding to the vehicle; performing cross-view angle feature alignment processing on the first image features based on the relative pose transformation information to obtain view angle alignment features with the vehicle view angle as a reference benchmark; The view angle alignment feature is fused with the second image feature to obtain a fused perception feature.

[0008] In an embodiment, the step of performing cross-view feature alignment processing on the first image feature according to the relative pose transformation information to obtain a view angle alignment feature taking a vehicle view angle as a reference includes: The relative pose transformation information is geometrically encoded to generate a relative pose vector; A query vector is generated based on the relative pose vector, and the first image feature is converted into a key vector and a value vector; Cross-attention operation is performed on the query vector, the key vector and the value vector to obtain a view angle alignment feature taking a vehicle view angle as a reference.

[0009] In an embodiment, the step of fusing the view angle alignment feature with the second image feature to obtain a fused perception feature includes: The view angle alignment feature and the second image feature are spliced in the channel dimension to generate a spliced feature; A preset weight operation is performed on the spliced feature to generate a real-time fusion weight; The view angle alignment feature and the second image feature are fused by weighting based on the real-time fusion weight to obtain a fused perception feature.

[0010] In an embodiment, the step of performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between a vehicle and a UAV includes: UAV calibration conversion data and vehicle calibration conversion data are obtained; Dynamic pose matching is performed based on the first environment data and the second environment data to obtain a relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system; The relative pose transformation information between the vehicle and the UAV is generated according to the UAV calibration conversion data, the vehicle calibration conversion data and the relative pose transformation matrix.

[0011] In an embodiment, the step of performing dynamic pose matching based on the first environment data and the second environment data to obtain a relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system includes: A preset coordinate system conversion is performed on the first environment data and the second environment data to obtain a first composite inverse transformation matrix and a second composite inverse transformation matrix; A relative attitude matrix of the UAV relative to the vehicle in a unified navigation coordinate system is generated based on the first composite inverse transformation matrix and the second composite inverse transformation matrix; generate a relative translation vector of the UAV compared to the vehicle in a unified navigation coordinate system, based on a navigation center coordinate of the vehicle; generate a relative pose transformation matrix according to the relative attitude matrix and the relative translation vector.

[0012] In an embodiment, the step of obtaining the UAV calibration conversion data and the vehicle calibration conversion data comprises: performing intrinsic parameter calibration conversion on the UAV camera and the vehicle camera respectively to obtain UAV camera conversion data and vehicle camera conversion data; performing extrinsic parameter calibration on the UAV body and the UAV camera according to the UAV camera conversion data to obtain the UAV calibration conversion data; performing extrinsic parameter calibration on the vehicle body and the vehicle camera based on the vehicle camera conversion data to obtain the vehicle calibration conversion data.

[0013] In addition, to achieve the above object, the present application further provides a vehicle perception optimization method based on a UAV, which comprises: a data acquisition module, configured to acquire first environment data collected by a UAV sensing module and second environment data collected by a vehicle end sensing module; a pose conversion module, configured to perform dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the UAV; a feature analysis module, configured to perform cross-view feature analysis according to the relative pose transformation information, the first environment data and the second environment data to obtain fused perception features; a perception module, configured to generate a vehicle environment perception result based on the fused perception features.

[0014] In addition, to achieve the above object, the present application further provides a vehicle, which comprises a memory, a processor, and a vehicle perception optimization method based on a UAV stored in the memory and executable on the processor, and the vehicle perception optimization method based on a UAV is configured to implement the steps of the vehicle perception optimization method based on a UAV as described above.

[0015] In addition, to achieve the above object, the present application further provides a storage medium, which stores a vehicle perception optimization method based on a UAV, and the vehicle perception optimization method based on a UAV is executed by a processor to implement the steps of the vehicle perception optimization method based on a UAV as described above.

[0016] The application provides a vehicle perception optimization method and device based on a UAV, a vehicle and a storage medium. The method comprises the following steps: acquiring first environment data collected by a UAV sensing module and second environment data collected by a vehicle end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the UAV and the vehicle; performing cross-view feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fused perception features; and generating a vehicle environment perception result based on the fused perception features.

[0017] The application first synchronously acquires multi-modal sensor data of the UAV and the vehicle. Then, the relative position and attitude between the two are calculated in real time to perform dynamic pose estimation, so as to ensure accurate spatio-temporal alignment in a dynamic scene and provide a geometric basis for subsequent feature fusion. Next, the dynamic pose information is used to intelligently align and fuse the high-altitude overhead features of the UAV and the flat-view features of the vehicle at the feature level, so as to improve the accuracy of feature fusion through cross-view feature analysis. Finally, the enhanced environment perception result is output by decoding the fused features. Therefore, the application does not need prior knowledge based on target detection to fuse the UAV data and the vehicle data, but combines the dynamic pose estimation between the UAV and the vehicle with the feature pre-fusion architecture to overcome the fusion obstacle of inconsistent spatio-temporal coordinate systems between dynamic platforms, so as to realize cross-view feature level pre-fusion of the UAV and vehicle perception data, which can not only improve the feature fusion efficiency, but also significantly improve the perception accuracy and expand the vehicle perception field of view. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application together with the specification.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, brief introductions will be given to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.

[0020] Figure 1 A flowchart of a first embodiment of the vehicle perception optimization method based on a UAV of the application; Figure 2 A first flowchart of a second embodiment of the vehicle perception optimization method based on a UAV of the application; Figure 3 A second flowchart of the second embodiment of the vehicle perception optimization method based on a UAV of the application; Figure 4 A third flowchart of the second embodiment of the vehicle perception optimization method based on a UAV of the application; Figure 5 A flowchart of a third embodiment of the vehicle perception optimization method based on a UAV of the present application is provided. Figure 6 A process diagram of the vehicle perception optimization method based on a UAV of the present application is provided. Figure 7 A module structure diagram of the vehicle perception optimization method device based on a UAV of the present application is provided. Figure 8 A device structure diagram of the hardware operating environment involved in the vehicle perception optimization method based on a UAV in the embodiment of the present application is provided.

[0021] The purpose implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0023] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0024] The main solution of the present application is: obtaining first environment data collected by a UAV sensing module and second environment data collected by a vehicle end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the UAV; performing cross-view feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fusion perception features; and generating a vehicle environment perception result based on the fusion perception features.

[0025] At present, in the technology of assisting vehicle operation with UAV data, a target-level post-fusion strategy is mostly used, that is, UAV data and vehicle data are fused based on the target objects detected by both, which leads to a slow fusion process of UAV data and vehicle data, cannot solve the problem of spatio-temporal alignment in dynamic scenes, cannot realize effective perception of objects that cannot be detected by the vehicle, and further causes the problems of high target missing detection rate and insufficient perception accuracy. Therefore, how to effectively improve the perception accuracy and perception field of view of the vehicle based on UAV data has become a problem to be solved.

[0026] To solve the problem, the application first synchronously acquires multi-modal sensor data of the unmanned aerial vehicle and the vehicle. Then, the relative position and attitude between the two are calculated in real time for dynamic pose estimation, ensuring accurate spatio-temporal alignment in a dynamic scene and providing a geometric basis for subsequent feature fusion. Next, the dynamic pose information is used to intelligently align and fuse the high-altitude overhead features of the unmanned aerial vehicle with the flat-view features of the vehicle at the feature level, improving the accuracy of feature fusion through cross-view feature analysis. Finally, the enhanced environmental perception results are output by decoding the fused features. Therefore, the application does not need prior knowledge based on target detection for unmanned aerial vehicle data and vehicle data fusion, but combines dynamic pose estimation between the unmanned aerial vehicle and the vehicle with a feature pre-fusion architecture to overcome the fusion obstacles of inconsistent spatio-temporal coordinate systems between dynamic platforms, thereby realizing cross-view feature-level pre-fusion of unmanned aerial vehicle and vehicle perception data. This not only improves feature fusion efficiency, but also significantly improves perception accuracy and expands the vehicle's perception field of view.

[0027] It should be noted that the execution subject of the present embodiment can be a vehicle, a main controller of the vehicle, a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an unmanned aerial vehicle-based vehicle perception optimization device capable of realizing the above functions, etc. The present embodiment does not make specific limitations on this. The following takes the main controller of the vehicle as an example to illustrate the present embodiment and the following embodiments.

[0028] Based on this, the present embodiment provides an unmanned aerial vehicle-based vehicle perception optimization method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the unmanned aerial vehicle-based vehicle perception optimization method of the present application is shown in the figure.

[0029] In the present embodiment, the unmanned aerial vehicle-based vehicle perception optimization method comprises steps S10-S40: Step S10, acquiring first environment data collected by an unmanned aerial vehicle sensing module and second environment data collected by a vehicle end sensing module; It is easy to understand that the above-mentioned unmanned aerial vehicle sensing module can be a multi-sensor combination carried on the unmanned aerial vehicle, including an unmanned aerial vehicle camera, an inertial measurement unit (IMU) of the unmanned aerial vehicle, a real-time dynamic positioning (RTK) module of the unmanned aerial vehicle, a laser radar, etc. Therefore, the vehicle environment raw data under the high-altitude view output by the unmanned aerial vehicle sensing module, i.e. the above-mentioned first environment data, can include first image data collected by the unmanned aerial vehicle camera, angular velocity / acceleration data collected by the IMU, WGS-84 coordinate data collected by the RTK, and point cloud data collected by the laser radar, etc.

[0030] Correspondingly, the vehicle-end sensing module can be a sensor combination installed on the vehicle, and can include a surround-view camera, a front-view main camera, a vehicle-mounted IMU, a vehicle-mounted RTK (using the same reference station signal as the UAV RTK), a millimeter-wave radar, and the like. The vehicle-end sensing module collects vehicle surrounding environment data under a ground view, i.e., the second environment data, which can also include second image data collected by each vehicle camera, angular velocity / acceleration data of the IMU, WGS-84 coordinate data of the RTK, and detection data collected by the millimeter-wave radar. In addition, the first environment data and the second environment data need to have the same timestamp (which can be synchronized based on GPS (Global Positioning System) time service) to ensure the spatio-temporal consistency of the two.

[0031] In step S20, dynamic pose estimation is performed based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the UAV. In step S30, cross-view feature analysis is performed according to the relative pose transformation information, the first environment data, and the second environment data to obtain fused perception features. In step S40, a vehicle environment perception result is generated based on the fused perception features.

[0032] It can be understood that, in the process of dynamic pose estimation performed by the main controller, a configured dynamic real-time relative pose estimation framework can be used to fuse the above-mentioned heterogeneous sensor data in different dimensions, i.e., the first environment data and the second environment data, to generate relative pose transformation information representing the real-time spatial relative position and real-time relative attitude relationship between the UAV and the vehicle, and to determine the relative motion state of 6 degrees of freedom (3 translational quantities and 3 rotational quantities) between the vehicle and the UAV.

[0033] It should be noted that, in combination with the relative pose transformation information, the main controller can perform geometric alignment, correlation matching, and fusion processing on the UAV high-altitude view features corresponding to the first environment data and the vehicle ground view features corresponding to the second environment data through a cross-attention mechanism, i.e., perform the above-mentioned cross-view feature analysis, to eliminate the feature misalignment caused by the view angle difference between the UAV collected data and the vehicle-end collected data.

[0034] At this time, the unified feature representation obtained through the cross-view feature analysis, i.e., the above-mentioned fused perception features, effectively integrates the complementary information of the two views, which contains both the wide-area view information of the UAV and the detailed feature information of the vehicle.

[0035] Finally, the host controller can generate a final output vehicle surrounding environment understanding result based on the fused perception data, i.e., the vehicle environment perception result described above. Illustratively, the embodiment can input the fused perception features into a decoder (which can adopt a CNN (Convolutional Neural Network) + lightweight Transformer hybrid architecture), and output a final vehicle environment perception result in combination with target detection, semantic segmentation, etc. In the embodiment, the vehicle environment perception result can include the position / speed / trajectory of obstacles (vehicles, pedestrians, cyclists), lane lines, traffic signs, road boundaries, construction areas, and identification and positioning of abnormal road conditions such as landslides / flooding, thereby effectively expanding the vehicle environment perception range.

[0036] In summary, compared with the prior art target-level post-fusion analysis scheme for fusing drone data and vehicle data based on target detection prior knowledge, the embodiment overcomes the fusion obstacle of inconsistent space-time coordinate systems between dynamic platforms by combining dynamic pose estimation between the drone and the vehicle with a feature pre-fusion architecture, thereby realizing cross-view feature-level pre-fusion of drone and vehicle perception data, which not only improves feature fusion efficiency, but also significantly improves perception accuracy and expands the vehicle perception field of view.

[0037] The embodiment provides a vehicle perception optimization method based on a drone, which includes: acquiring first environment data collected by a drone sensing module and second environment data collected by a vehicle-end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the drone; performing cross-view feature analysis based on the relative pose transformation information, the first environment data, and the second environment data to obtain fused perception features; and generating a vehicle environment perception result based on the fused perception features. Compared with the prior art target-level post-fusion analysis scheme for fusing drone data and vehicle data based on target detection prior knowledge, the embodiment overcomes the fusion obstacle of inconsistent space-time coordinate systems between dynamic platforms by combining dynamic pose estimation between the drone and the vehicle with a feature pre-fusion architecture, thereby realizing cross-view feature-level pre-fusion of drone and vehicle perception data, which not only improves feature fusion efficiency, but also significantly improves perception accuracy and expands the vehicle perception field of view.

[0038] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above description, and will not be repeated hereinafter.

[0039] It is easy to understand that the first environment data collected by the unmanned aerial vehicle and the second environment data collected by the vehicle end sensing module both exist data from different coordinate systems (for example, pixel coordinates, camera coordinate system, body coordinate system, vehicle coordinate system and navigation coordinate system, etc.), in order to realize the subsequent feature fusion process, the data from different coordinate systems need to be aligned. However, the traditional sensor fusion relies on static calibration, but in the running process of the vehicle, there is dynamic relative motion between the unmanned aerial vehicle and the vehicle, which causes real-time change of the space-time coordinate system, and the observation angle difference between the two is significant (the unmanned aerial vehicle provides a top-down wide-area view, and the vehicle provides a flat-view detailed view). Therefore, how to unify the space-time reference system in real time and accurately determine the relative positioning between the two to overcome the fusion challenges brought by the external parameter drift and the inconsistent view angle has become a difficult problem to be overcome.

[0040] Therefore, on the basis of the first embodiment, please refer to Figure 2 , Figure 2 This is the first flowchart of the second embodiment of the vehicle perception optimization method based on the unmanned aerial vehicle of the present application. In this embodiment, step S20 includes steps S21-S23: Step S21, acquiring unmanned aerial vehicle calibration conversion data and vehicle calibration conversion data; It is understood that in order to solve the problem of relative pose matching between the unmanned aerial vehicle and the vehicle, multiple coordinate system conversions are required in this embodiment. Among them, the above-mentioned unmanned aerial vehicle calibration conversion data can be a rotation matrix and a translation vector determined by internal and external parameter calibration in advance, which is the rotation matrix of the unmanned aerial vehicle camera relative to the unmanned aerial vehicle body. And the translation vector The vehicle calibration conversion data can be a rotation matrix And a translation vector

[0041] At this time, in a feasible implementation manner, please refer to Figure 3 , Figure 3 This is the second flowchart of the second embodiment of the vehicle perception optimization method based on the unmanned aerial vehicle of the present application. In this embodiment, step S21 can include steps A1-A3: Step A1, performing internal parameter calibration conversion on the unmanned aerial vehicle camera and the vehicle camera respectively to obtain unmanned aerial vehicle camera conversion data and vehicle camera conversion data; It is easy to understand that this embodiment requires converting the image points (pixel coordinates) captured by the UAV camera or vehicle camera to the UAV's own and the vehicle's own body coordinate system (B-frame). First, the main controller can achieve an effective conversion from UAV / vehicle pixel coordinates to UAV / vehicle camera coordinate system through the calibration internal parameters of the UAV camera and vehicle camera. The converted coordinates are the aforementioned UAV camera conversion data and vehicle camera conversion data.

[0042] Therefore, the process of internal parameter calibration and conversion for UAV cameras and vehicle cameras described above can be divided into the following three steps: First, obtain the intrinsic parameter matrix generated in advance by the UAV camera and vehicle camera through intrinsic parameter calibration. The calibration internal parameters present in this intrinsic parameter matrix may include: pixel focal length in the x and y directions ( (in pixels), coordinates of the image center point in pixel coordinate system ( ) and distortion coefficient vector D, etc.

[0043] Second, the distortion coefficient vector D is used to represent the pixel coordinates of the data collected by the UAV camera and the vehicle camera. Perform distortion correction to obtain undistorted coordinates. This ensures accurate pixel positioning, unaffected by lens distortion. This distortion correction process can be implemented using mature library functions, such as OpenCV's cv2.undistortPoints. The distortion correction process can be expressed as follows (this process is applicable to both drones and vehicles): (1) Third, perform a coordinate transformation from pixel to camera coordinate system, that is, the corrected pixel coordinates... This is converted to 3D points in the camera coordinate system. At this point, the origin of the camera coordinate system is at the camera's optical center, and the Z-axis is along the optical axis. This coordinate transformation process is also applicable to both drone cameras and vehicle cameras. The transformation formula is as follows: (2) In the formula, These are the point coordinates in the camera coordinate system (corresponding to the above-mentioned UAV camera conversion data and vehicle camera conversion data), that is, the point coordinates of the real object in the camera coordinate system. The position of the point in the left-right direction of the camera (unit: meters), positive right and negative left; The position of the point in the vertical direction of the camera, positive up and negative down; denoted as the distance of the point along the optical axis, positive in front and negative behind.

[0044] Step A2: Based on the UAV camera conversion data, perform external parameter calibration on the UAV body and the UAV camera to obtain UAV calibration conversion data; Step A3: Based on the vehicle camera conversion data, perform extrinsic parameter calibration on the vehicle body and the vehicle camera to obtain vehicle calibration conversion data.

[0045] It is easy to understand that this embodiment can achieve the transformation from the UAV / vehicle camera coordinate system to the UAV / vehicle body B-frame through extrinsic parameter calibration, thereby obtaining the coordinates and pose of the real object relative to the body / vehicle. The specific formula is as follows (applicable to both UAVs and vehicles): (3) In the formula, These are the coordinates of a point in the drone's body coordinate system or the vehicle's vehicle coordinate system. For the rotation matrix from the camera coordinate system of the drone camera to the body coordinate system of the drone body / For the rotation matrix from the vehicle camera to the vehicle coordinate system of the vehicle body; These are the coordinates of a point in the camera coordinate system of the drone / vehicle; Let $\mathbf$ be the translation vector from the camera coordinate system of the UAV camera to the body coordinate system of the UAV body, and $\mathbf$ be the translation vector from the camera coordinate system of the vehicle camera to the vehicle coordinate system of the vehicle body.

[0046] The UAV's body coordinate system is a coordinate system (B system) established with the center of gravity as the origin, with the X-axis pointing in the direction of the UAV's movement, the Y-axis pointing to the left, and the Z-axis pointing vertically upward (right-handed coordinate system); the vehicle's body coordinate system can be a coordinate system (B system) established with the center of the vehicle's rear axle as the origin, with the X-axis pointing in the direction of the vehicle's movement, the Y-axis pointing to the left, and the Z-axis pointing vertically upward (right-handed coordinate system).

[0047] At this point, referring to formula (3), it can be seen that in this embodiment, the rotation matrix of the UAV camera relative to the UAV body can be obtained during the external parameter calibration process of clarifying the relative relationship between the camera and the body / vehicle body. Translation vector And the vehicle camera relative to the vehicle body Translation vector .

[0048] Step S22: Based on the first environmental data and the second environmental data, perform dynamic pose matching to obtain the relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system; It is easy to understand that in the collaborative operation of UAVs and vehicles, since the poses of the two are constantly changing dynamically, this embodiment can rely on a unified high-precision global coordinate system as a common reference frame to realize the real-time relative pose calculation of UAVs and vehicles.

[0049] At this point, in the above dynamic pose matching process, this embodiment can use the Global Navigation Satellite System (GNSS) represented by GPS as a spatial reference to uniformly transform the local coordinates (B-frame) of the UAV body and the vehicle body to the navigation coordinate system (G-frame), such as the WGS-84 geodetic coordinate system, which can correspond to the above unified navigation coordinate system.

[0050] However, vehicle coordinate systems (B-frame) often use the ENU (East-North-Sky) coordinate system, while UAV body coordinate systems (B-frame) commonly use the NED (North-East-Ground) coordinate system. Therefore, there are differences in the definition of their body coordinate axes. Furthermore, UAVs need to consider the Earth's curvature and elevation changes during navigation and positioning, typically requiring modeling based on geodetic coordinate systems such as WGS-84. Since vehicles often approximate planar motion within a local area, a projection transformation from geodetic coordinates to local planar coordinates is also necessary.

[0051] Therefore, this embodiment can solve the real-time relative motion state of the UAV and the vehicle in a unified coordinate system based on the first environmental data and the second environmental data corresponding to the UAV and the vehicle, respectively, according to the different coordinate system transformation process mentioned above, and determine the above relative pose transformation matrix.

[0052] In one feasible implementation, refer to Figure 4 , Figure 4 This is a third flowchart illustrating the second embodiment of the vehicle perception optimization method based on unmanned aerial vehicles (UAVs) of this application. In this embodiment, step S22 may include steps B1 to B4: Step B1: Perform a preset coordinate system transformation on the first environmental data and the second environmental data to obtain a first composite inverse matrix and a second composite inverse matrix; It is easy to understand that, based on the above analysis, the preset coordinate system transformation process mentioned in this embodiment requires the transformation from the UAV's NED coordinate system to the vehicle's ENU coordinate system, and the transformation from the UAV / vehicle coordinate system to the navigation coordinate system (G system). In this case, the aforementioned first composite inversion matrix can be a composite rotation matrix that transforms the UAV's body coordinate system (B system) to the navigation coordinate system (G system), representing the UAV's attitude relative to the G system. Similarly, the aforementioned second composite inversion matrix can be a composite rotation matrix that transforms the vehicle's body coordinate system (B system) to the navigation coordinate system (G system), representing the vehicle's attitude relative to the G system, and its construction method is consistent with the first composite inversion matrix.

[0053] It is important to understand that both the first and second composite inverse matrices mentioned above can be obtained by multiplying the Euler angle rotation matrices in reverse order. Firstly, the transformation process from the NED coordinate system of the UAV body to the ENU coordinate system of the vehicle body can be represented by the following formula: (4) In the formula, For rotation matrix, These are the initial UAV body coordinates corresponding to the first environmental data. These are the converted coordinates of the UAV body corresponding to the first environmental data.

[0054] Secondly, in the aforementioned transformation from the body / vehicle coordinate system to the navigation coordinate system (G system), the core objective is to resolve the orientation alignment issue. At this point, the main controller can rotate the body / vehicle coordinate system around the Z-axis (corresponding to the yaw angle ψ), then around the Y-axis (corresponding to the pitch angle θ), and finally around the X-axis (corresponding to the roll angle φ), following the Euler angle rotation sequence. This yields the rotation matrix for each axis. , and Then, by multiplying the three matrices in reverse order, we can obtain the composite inverse matrix of the B-system to G-system transformation. The specific formula is as follows:

[0055] (5) At this point, the formula for converting B-system coordinates to G-system coordinates is as follows: (6) Therefore, for the initial environmental data collected by the UAV in the body coordinate system, the coordinate transformation from the B-frame to the G-frame can be performed using the following formula: (7) For the second environmental data collected by the vehicle and located in the vehicle coordinate system, the coordinate system transformation from B-frame to G-frame can be performed using the following formula: (8) At this time, the above This can be the first composite inversion matrix corresponding to the drone. This is the second composite inverse matrix corresponding to the vehicle.

[0056] Step B2: Generate the relative attitude matrix of the UAV to the vehicle in a unified navigation coordinate system based on the first composite inverse matrix and the second composite inverse matrix; It is important to understand that the aforementioned relative attitude matrix can be a matrix representing the actual attitude relationship between the UAV and the vehicle in the G-frame, reflecting their rotational relationship. This relative attitude matrix... It can be obtained by multiplying two composite inverse matrices, as shown in the following formula: (9) Step B3: Using the navigation center coordinates of the vehicle as a reference, generate the relative translation vector of the UAV relative to the vehicle in a unified navigation coordinate system. It should be noted that, although this embodiment can be achieved through the above... and To achieve uniformity in the ENU coordinate system, with all coordinate axes aligned (East-North-Sky), but... and The corresponding coordinate origin is still at the centroid of their respective carriers (UAV body or vehicle body), so the relative position cannot be calculated directly. At this time, it is also necessary to realize the transformation from the UAV navigation coordinate system to the vehicle local plane coordinate system.

[0057] It is easy to understand that the WGS-84 coordinates (L_fly, λ_fly, h_fly) and (L_car, λ_car, h_car) of the phase center of the drone's antenna, obtained by the GNSS receiver, are the navigation center of the drone and the vehicle, respectively, and both correspond to the absolute position of the drone and the vehicle on Earth.

[0058] At this point, the main controller can use the vehicle's navigation center coordinates (L_car, λ_car, h_car) as a reference, which is the origin of the local ENU coordinate system to be matched, to determine the position vector of the UAV relative to the vehicle in the local ENU coordinate system, and represent it as (E_fly, N_fly, U_fly), where E is the east coordinate, N is the north coordinate, and U is the sky coordinate.

[0059] For example, the main controller can directly call the coordinate transformation library function GeographicLib to convert the UAV's WGS-84 coordinates (L_fly, λ_fly, h_fly) into the ENU coordinate system with the vehicle origin as the reference. The GeographicLib library function internally handles the Earth's curvature and directly outputs the UAV's position coordinates relative to the vehicle (E_fly, N_fly, U_fly), which can be expressed by the following formula: (10) In the formula, The aforementioned relative translation vector represents the distance between the absolute coordinates of the UAV and the vehicle (or the lever value between the UAV and the vehicle).

[0060] Step B4: Generate a relative pose transformation matrix based on the relative pose matrix and the relative translation vector.

[0061] Finally, the main controller can combine the relative translation vectors mentioned above. and relative attitude matrix Determine the relative pose transformation matrix of the UAV and vehicle in the B-frame coordinate system under a unified G-frame. The calculation process is expressed as follows: (11) Step S23: Generate relative pose transformation information between the vehicle and the UAV based on the UAV calibration transformation data, the vehicle calibration transformation data, and the relative pose transformation matrix.

[0062] It is important to understand that the main controller can generate relative pose transformation information between the vehicle and the UAV based on the relative pose transformation matrix and the acquired extrinsic parameter matrix, i.e., the aforementioned UAV calibration transformation data and vehicle calibration transformation data. The calculation process is as follows:

[0063]

[0064] =

[0065] (12) In the formula, This refers to relative pose transformation information; This is the extrinsic parameter matrix for the vehicle camera; This is the transformation matrix from the navigation coordinate system to the vehicle coordinate system; This is the transformation matrix from the body coordinate system to the navigation coordinate system; This is the extrinsic parameter matrix for the UAV camera.

[0066] In summary, this embodiment can introduce a standardized dynamic real-time relative pose estimation framework based on the above formulas (1) to (12). The data conversion between different coordinate systems is performed in the order of "pixel → camera coordinate system → body coordinate system / vehicle coordinate system → navigation system". This unifies the heterogeneous data such as GPS, IMU, and visual features collected by the sensors of the UAV and the vehicle into the WGS-84 navigation coordinate system. Combined with Euler angle rotation matrix and relative translation calculation, the relative pose transformation information between the vehicle and the UAV is obtained, thereby adapting to the motion state changes of the UAV and the vehicle in real time and solving the fundamental challenge of spatiotemporal alignment between mobile platforms.

[0067] This embodiment discloses performing intrinsic parameter calibration transformations on the UAV camera and vehicle camera respectively to obtain UAV camera conversion data and vehicle camera conversion data; performing extrinsic parameter calibration on the UAV body and UAV camera based on the UAV camera conversion data to obtain UAV calibration conversion data; and performing extrinsic parameter calibration on the vehicle body and vehicle camera based on the vehicle camera conversion data to obtain vehicle calibration conversion data. A preset coordinate system transformation is performed on the first environmental data and the second environmental data to obtain a first composite inverse matrix and a second composite inverse matrix; a relative attitude matrix of the UAV relative to the vehicle in a unified navigation coordinate system is generated based on the first composite inverse matrix and the second composite inverse matrix; a relative translation vector of the UAV relative to the vehicle in the unified navigation coordinate system is generated using the vehicle's navigation center coordinates as a reference; and a relative pose transformation matrix is ​​generated based on the relative attitude matrix and the relative translation vector. Relative pose transformation information between the vehicle and the UAV is generated based on the UAV calibration conversion data, the vehicle calibration conversion data, and the relative pose transformation matrix. This embodiment can accurately estimate the 6-DOF relative motion between the UAV and the vehicle online and at high frequency, transforming the relative pose between the UAV and the vehicle into a state variable that needs to be continuously estimated. It can adapt to the changes in the motion state of the UAV and the vehicle in real time, thereby solving the fundamental challenge of spatiotemporal alignment between mobile platforms.

[0068] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the contents that are the same as or similar to those in the first and second embodiments described above can be referred to the above description and will not be repeated hereafter.

[0069] It's important to understand that drones typically engage in rapid and frequent maneuvers, such as agile turns, hovering, and climbing, while vehicles also perform sharp maneuvers like braking and turning. Therefore, directly performing feature fusion analysis on the image features of drones and vehicles (including their coordinate information) requires traversing all regions of both images, resulting in low computational efficiency and low matching accuracy.

[0070] Therefore, to solve this problem, this embodiment can introduce a cross-attention mechanism into the image fusion of UAV and vehicle, thereby enabling selective and dynamic attention to the most relevant information from different perspectives (attention to the pose of the UAV camera relative to the vehicle camera) during the feature fusion process. This avoids the need to search for correlations from completely unrelated locations during the image fusion process. Instead, it only requires subtle compensation and enhancement of the vehicle image data based on the UAV image data to quickly generate more accurate and information-rich image fusion results.

[0071] Based on this, please refer to Figure 5 , Figure 5This is a flowchart illustrating the third embodiment of the vehicle perception optimization method based on unmanned aerial vehicles (UAVs) of this application. Figure 5 As shown, in this embodiment, step S30 may include steps S31 to S33: Step S31: Perform image feature analysis on the first environmental data and the second environmental data to obtain the first image features corresponding to the UAV and the second image features corresponding to the vehicle; It is easy to understand that in this embodiment, the main controller may use a deep learning model, such as a CNN backbone network, to extract discriminative information from the image data contained in the first environment data and the second environment data, including edges, textures, target contours, high-level semantic features, etc.

[0072] The feature representation extracted from the image data of the first environmental data, namely the first image features that can reflect the global environmental features from the UAV's high-altitude perspective, such as large-scale lane distribution and long-distance obstacle clusters, can be represented as follows: The second image features extracted from the image data of the second environmental data can reflect the environmental details from the vehicle's ground perspective, such as the texture of nearby obstacles and the text information of traffic signs, and can be represented as... .

[0073] Step S32: Perform cross-view feature alignment processing on the first image features according to the relative pose transformation information to obtain view alignment features with the vehicle view as the reference. It should be noted that the main controller can perform spatial geometric correction on the first image features based on the relative pose transformation information, so as to realize the coordinate mapping under the vehicle's viewpoint, that is, to perform the above-mentioned cross-viewpoint feature alignment processing, so as to eliminate the feature misalignment caused by the difference in viewpoint between the UAV image data and the vehicle image data.

[0074] At this point, as one possible implementation method, in this embodiment, step S32 includes steps C1 to C3: Step C1: Perform geometric condition encoding on the relative pose transformation information to generate a relative pose vector; Understandably, to facilitate subsequent interaction with image features from drones and vehicles, this embodiment can map the high-dimensional matrix data of the relative pose transformation matrix into a low-dimensional, compact vector representation through a neural network (such as an encoder composed of a multi-layer perceptron (MLP) or a lightweight Transformer), generating the aforementioned relative pose vector that retains key geometric information. .

[0075] Step C2: Generate a query vector based on the relative pose vector, and convert the first image features into a key vector and a value vector; Step C3: Perform cross-attention operation on the query vector, the key vector, and the value vector to obtain the view alignment feature with the vehicle's view as the reference.

[0076] It is important to understand that in the process of aligning UAV image features to vehicle image features, this embodiment can guide the fusion of UAV image features through a cross-attention network. Here, the key vector K and value vector V can be feature vectors obtained by projecting the first image features. The key vector K can be used to calculate the relevance with the query vector, representing the "index" information of the first image feature; the value vector V can be used for weighted summation based on relevance weights, representing the "value" information of the first image feature.

[0077] The query vector Q mentioned above can be a guide vector generated based on the relative pose vector, used to query feature information related to the vehicle's perspective in the key vector, and its dimension must be consistent with the key vector and the value vector.

[0078] Therefore, the master controller can make the cross-attention network operate according to geometric conditions, i.e., the relative pose vector. The generated query vector Q actively queries the drone image features related to vehicle image features, and interacts with the drone image feature key vector K and value vector V to obtain drone image features aligned to the vehicle's viewpoint, i.e., the aforementioned viewpoint aligned features. The corresponding cross-attention operation process is as follows:

[0079] (13) In the formula, , and All of these are learnable projection matrices in the cross-attention model.

[0080] It is understood that in this embodiment, a learnable projection matrix can be used. For relative pose vector Perform linear transformation to generate query vector ; through learnable projection matrices For the first image features Generate key vectors by performing linear transformations ; through learnable projection matrices For the first image features Perform linear transformations to generate value vectors .

[0081] At this point, the main controller can calculate the relevance score between the query vector Q and the key vector K based on formula (12). Then, the relevance scores are normalized using the Softmax function to obtain attention weights. Finally, the value vector V is weighted and fused according to the attention weights to obtain the view alignment features that enable the UAV to be specifically aligned to the vehicle's view. .

[0082] In this embodiment, when performing feature fusion using the cross-attention mechanism, the query vector guides the focus to feature regions related to the vehicle's perspective. Therefore, compared to existing full-image traversal methods, the feature fusion efficiency of this embodiment can be significantly improved. At the same time, by introducing attention weights to dynamically adapt to changes in the geometric relationship between the UAV and the vehicle, and by further optimizing the filtering of irrelevant information through a learnable projection matrix, the subsequent feature fusion process can maintain high alignment accuracy and high fusion speed even in high-frequency jitter scenarios such as UAV maneuvers (turning, climbing) and vehicle driving (braking, turning).

[0083] Step S33: Perform feature fusion on the viewpoint alignment feature and the second image feature to obtain fused perception features.

[0084] It is easy to understand that the reliability of drone features and vehicle features varies greatly under different scenarios (such as occlusion, sunny / rainy / foggy conditions, close-range / long-range distances). If a fixed weight is used to fuse the image features of the two, it cannot adapt to dynamic changes. Therefore, in this embodiment, the viewpoint alignment features of the drone and the second image features of the vehicle can be dynamically weighted and fused.

[0085] At this point, as one possible implementation method, in this embodiment, step S33 includes steps D1~D3: Step D1: The viewpoint alignment feature and the second image feature are spliced ​​together in the channel dimension to generate spliced ​​features; Step D2: Perform a preset weight calculation on the splicing features to generate real-time fusion weights; Step D3: Based on the real-time fusion weight, the viewpoint alignment feature and the second image feature are weighted and fused to obtain the fused perception feature.

[0086] It should be noted that the main controller can align the viewpoint with features. With second image features The features are superimposed along the feature channel dimension to generate a high-dimensional feature tensor with expanded channel count, i.e., the aforementioned spliced ​​features. Then, during the aforementioned preset weight calculation process, this embodiment can learn the importance distribution of the two original features in the spliced ​​features through a neural network, adaptively determine the reliability of the two features under different scenarios, and generate a real-time fusion weight G for subsequent weighted fusion. For example, this embodiment can use a convolutional layer and an activation function to perform the aforementioned preset weight calculation. In this case, the calculation process of the real-time fusion weight G can be as follows: (14) In the formula, For channel splicing, features from different sources are spliced ​​together along the channel dimension. These are convolutional layers, typically 1x1 convolutions, used to learn how to calculate weights from concatenated features; The sigmoid activation function compresses each value into the (0, 1) interval.

[0087] At this point, a value of 1 for the aforementioned real-time fusion weight G can represent "complete trust". A value of 0 indicates "complete trust". ".

[0088] Finally, the main controller can adaptively weight and integrate the view alignment features and the second image features according to the real-time fusion weights G to generate fused perception features. The corresponding calculation process can be seen as follows: (15) in, This indicates element-wise multiplication.

[0089] In summary, this embodiment uses the real-time estimated dynamic extrinsic parameters (i.e., the relative pose transformation matrix) as key inputs and integrates them into the cross-attention feature fusion process of UAV image data and vehicle image data. This allows the feature fusion process to always focus on the geometrically aligned region between the UAV and the vehicle, thereby improving the efficiency of feature fusion in dynamic environments. Meanwhile, this embodiment eliminates the feature misalignment problem caused by viewpoint differences through cross-view feature alignment processing, and dynamically adjusts the weights of the two features. For example, it enhances the feature weight of the drone in occluded scenarios and enhances the feature weight of the vehicle in close-range scenarios, so that the two features can be effectively fused in the same geometric space, thereby effectively enhancing the accuracy of feature fusion.

[0090] This embodiment discloses image feature analysis of first and second environmental data to obtain first image features corresponding to the UAV and second image features corresponding to the vehicle; geometric conditional encoding of relative pose transformation information to generate a relative pose vector; generation of a query vector based on the relative pose vector, and conversion of the first image features into key and value vectors; cross-attention operation on the query vector, key vector, and value vector to obtain a view alignment feature with the vehicle's viewpoint as a reference; concatenation of the view alignment feature and the second image feature along the channel dimension to generate a concatenated feature; pre-weighted calculation of the concatenated feature to generate a real-time fusion weight; and weighted fusion of the view alignment feature and the second image feature based on the real-time fusion weight to obtain a fused perception feature.

[0091] Therefore, this embodiment can eliminate the feature misalignment problem caused by viewpoint differences through cross-view feature alignment processing, so that the two features can be effectively fused in the same geometric space, thereby effectively integrating the global information of the high-altitude viewpoint and the detailed information of the ground viewpoint, and improving the information density of the fused perception features. In addition, the adaptive weighted feature fusion strategy proposed in this embodiment can effectively adapt to real-time operation scenarios and improve the perception robustness in different scenarios.

[0092] For example, to help understand the technical concept or principle of the UAV-based vehicle perception optimization method after combining this embodiment with the above embodiments one to three, please refer to Figure 6 , Figure 6 A schematic diagram illustrating the process of the vehicle perception optimization method based on UAVs in this application is provided below: like Figure 6 As shown, this application constructs a complete perception fusion closed-loop system through the collaborative work of multiple sensors of UAV and vehicle. It consists of UAV and vehicle-side sensing modules (camera, IMU, RTK, etc.); UAV and vehicle-side calibration modules; UAV and vehicle-side sensor abstraction modules, vehicle-side positioning modules, perception modules, and vehicle-side fusion modules with spatial cross-attention mechanisms corresponding to the main controller, so as to realize dynamic pose matching and target front fusion of vehicle and UAV.

[0093] In practice, after takeoff, the drone can establish a communication connection with the vehicle and synchronize the time between the drone and the vehicle via GPS timing to ensure that the error between their data does not exceed 1ms (millimeter). Then, the drone follows the vehicle along a preset trajectory (maintaining a height of 50-100 meters directly above the vehicle, with a horizontal displacement of no more than 5 meters). The drone's sensing module and the vehicle's sensing module simultaneously start data acquisition, continuously outputting first environmental data and second environmental data.

[0094] Then, the first and second environmental data can be preprocessed through the drone and vehicle-side sensor abstraction module of the main controller, such as image distortion correction, IMU data denoising, and RTK coordinate smoothing.

[0095] Next, the vehicle-side positioning module of the main controller adopts the configured dynamic relative pose estimation framework, and integrates IMU data and RTK data from the preprocessed first and second environmental data to obtain the relative pose transformation information between the vehicle and the UAV.

[0096] Subsequently, the vehicle-side fusion module of the main controller, based on the generated relative pose transformation information, guides the first and second image features generated by the perception module through a spatial cross-attention mechanism to perform cross-view feature analysis of geometric perspectives, generating fused perception features that eliminate feature misalignment caused by viewpoint differences. Finally, based on the fused perception features, the main controller can identify targets such as obstacles and traffic signs using object detection algorithms such as YOLOv8, and extract structural information such as lane lines and road boundaries using semantic segmentation algorithms, integrating them to obtain a complete vehicle environment perception result, which is then output to the vehicle's autonomous driving decision-making system.

[0097] Therefore, this application solves the core problem of real-time changes in sensor extrinsic parameters between two motion platforms, UAV and vehicle, through a dynamic real-time relative pose estimation framework. On this basis, it innovatively integrates dynamic extrinsic parameters into a cross-view attention mechanism, achieving deep fusion at the feature level, thereby significantly improving the field of view (FOV) and accuracy of perception.

[0098] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the vehicle perception optimization method based on UAVs in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0099] This application also provides a vehicle perception optimization method and apparatus based on unmanned aerial vehicles (UAVs), please refer to... Figure 7 , Figure 7 This is a schematic diagram of the module structure of the vehicle perception optimization method device based on unmanned aerial vehicles (UAVs) according to an embodiment of this application. In this embodiment, the vehicle perception optimization method device based on UAVs includes: The data acquisition module 701 is used to acquire the first environmental data collected by the UAV sensor module and the second environmental data collected by the vehicle-side sensor module. The pose conversion module 702 is used to perform dynamic pose estimation based on the first environmental data and the second environmental data to obtain relative pose transformation information between the vehicle and the drone. Feature analysis module 703 is used to perform cross-view feature analysis based on the relative pose transformation information, the first environmental data and the second environmental data to obtain fused perception features; The perception module 704 is used to generate vehicle environment perception results based on the fused perception features.

[0100] As one possible implementation, in this embodiment, the feature analysis module 703 is further configured to perform image feature analysis on the first environmental data and the second environmental data to obtain the first image feature corresponding to the UAV and the second image feature corresponding to the vehicle; perform cross-view feature alignment processing on the first image feature according to the relative pose transformation information to obtain view alignment features with the vehicle view as a reference; and perform feature fusion on the view alignment features and the second image feature to obtain fused perception features.

[0101] As one possible implementation, in this embodiment, the feature analysis module 703 is further configured to perform geometric condition encoding on the relative pose transformation information to generate a relative pose vector; generate a query vector based on the relative pose vector, and convert the first image features into a key vector and a value vector; perform cross-attention operation on the query vector, the key vector, and the value vector to obtain a view alignment feature with the vehicle viewpoint as the reference.

[0102] As one possible implementation, in this embodiment, the feature analysis module 703 is further configured to stitch the viewpoint alignment feature and the second image feature in the channel dimension to generate stitched features; perform preset weight calculation on the stitched features to generate real-time fusion weights; and perform weighted fusion of the viewpoint alignment feature and the second image feature based on the real-time fusion weights to obtain fused perception features.

[0103] As one possible implementation, in this embodiment, the pose transformation module 702 is further configured to acquire UAV calibration transformation data and vehicle calibration transformation data; perform dynamic pose matching based on the first environmental data and the second environmental data to obtain the relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system; and generate relative pose transformation information between the vehicle and the UAV based on the UAV calibration transformation data, the vehicle calibration transformation data and the relative pose transformation matrix.

[0104] As one possible implementation, in this embodiment, the pose transformation module 702 is further configured to perform a preset coordinate system transformation on the first environmental data and the second environmental data to obtain a first composite inverse matrix and a second composite inverse matrix; generate a relative attitude matrix of the UAV relative to the vehicle in a unified navigation coordinate system based on the first composite inverse matrix and the second composite inverse matrix; generate a relative translation vector of the UAV relative to the vehicle in a unified navigation coordinate system based on the navigation center coordinates of the vehicle; and generate a relative pose transformation matrix according to the relative attitude matrix and the relative translation vector.

[0105] As one possible implementation, in this embodiment, the pose conversion module 702 is further used to perform intrinsic parameter calibration conversion on the UAV camera and the vehicle camera respectively to obtain UAV camera conversion data and vehicle camera conversion data; perform extrinsic parameter calibration on the UAV body and the UAV camera based on the UAV camera conversion data to obtain UAV calibration conversion data; and perform extrinsic parameter calibration on the vehicle body and the vehicle camera based on the vehicle camera conversion data to obtain vehicle calibration conversion data.

[0106] The UAV-based vehicle perception optimization method and apparatus provided in this application, employing the UAV-based vehicle perception optimization method in the above embodiments, can solve the technical problem of how to effectively improve the perception accuracy and perception field of view of vehicles based on UAV data. Compared with the prior art, the beneficial effects of the UAV-based vehicle perception optimization method and apparatus provided in this application are the same as those of the UAV-based vehicle perception optimization method provided in the above embodiments, and other technical features in the UAV-based vehicle perception optimization method and apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0107] This application provides a vehicle, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the UAV-based vehicle perception optimization method described in Embodiment 1 above.

[0108] The following is for reference. Figure 8 It shows a structural diagram of a vehicle suitable for implementing the embodiments of this application, that is, a device structural diagram of the hardware operating environment involved in the vehicle perception optimization method based on UAV in the embodiments of this application. Figure 8 The vehicle shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments described in this application. Figure 8As shown, the vehicle may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for vehicle operation. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the vehicle to communicate wirelessly or wiredly with other devices to exchange data. Although vehicles with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0109] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment disclosed in this application includes a UAV-based vehicle perception optimization method program product, which includes a UAV-based vehicle perception optimization method program carried on a computer-readable medium, the UAV-based vehicle perception optimization method program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the UAV-based vehicle perception optimization method program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the UAV-based vehicle perception optimization method program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0110] The vehicle provided in this application employs the UAV-based vehicle perception optimization method described in the above embodiments, which solves the technical problem of how to effectively improve the perception accuracy and field of view of a vehicle based on UAV data. Compared with the prior art, the beneficial effects of the vehicle provided in this application are the same as those of the UAV-based vehicle perception optimization method provided in the above embodiments, and other technical features of the vehicle are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0112] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0113] This application provides a storage medium having computer-readable program instructions (i.e., a UAV-based vehicle perception optimization method program) stored thereon, the computer-readable program instructions being used to execute the UAV-based vehicle perception optimization method in the above embodiments.

[0114] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of the storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0115] The aforementioned storage medium may be included in the vehicle or may exist independently without being installed in the vehicle.

[0116] The aforementioned storage medium carries one or more programs, which, when executed by the vehicle, enable the vehicle to improve its perception accuracy and perception field of view.

[0117] The program code for the UAV-based vehicle perception optimization method for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and UAV-based vehicle perception optimization method program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0120] The readable storage medium provided in this application is a storage medium storing computer-readable program instructions (i.e., the UAV-based vehicle perception optimization method program) for executing the aforementioned UAV-based vehicle perception optimization method. This solves the technical problem of how to effectively improve the perception accuracy and field of view of vehicles based on UAV data. Compared with the prior art, the beneficial effects of the storage medium provided in this application are the same as those of the UAV-based vehicle perception optimization method provided in the above embodiments, and will not be repeated here.

[0121] The above are only some embodiments of this application and do not limit the scope of the solution of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method for optimizing vehicle perception based on unmanned aerial vehicles, the method comprising: The method comprises: acquiring first environment data collected by a UAV sensing module and second environment data collected by a vehicle end sensing module; performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between a vehicle and a UAV; performing cross-view feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fused perception features; generating a vehicle environment perception result based on the fused perception features.

2. The drone-based vehicle perception optimization method of claim 1, wherein, The step of performing cross-view feature analysis based on the relative pose transformation information, the first environment data and the second environment data to obtain fused perception features comprises: performing image feature analysis on the first environment data and the second environment data to obtain first image features corresponding to the UAV and second image features corresponding to the vehicle; performing cross-view feature alignment processing on the first image features based on the relative pose transformation information to obtain view alignment features with the vehicle view angle as a reference benchmark; performing feature fusion on the view alignment features and the second image features to obtain fused perception features.

3. The UAV-based vehicle perception optimization method of claim 2, wherein, The step of performing cross-view feature alignment processing on the first image features based on the relative pose transformation information to obtain view alignment features with the vehicle view angle as a reference benchmark comprises: performing geometric condition coding on the relative pose transformation information to generate a relative pose vector; generating a query vector based on the relative pose vector and converting the first image features into a key vector and a value vector; performing cross-attention operation on the query vector, the key vector and the value vector to obtain view alignment features with the vehicle view angle as a reference benchmark.

4. The UAV-based vehicle perception optimization method of claim 2, wherein, The step of performing feature fusion on the view alignment features and the second image features to obtain fused perception features comprises: splicing the view alignment features and the second image features in a channel dimension to generate spliced features; performing preset weight operation on the spliced features to generate real-time fusion weights; performing weighted fusion on the view alignment features and the second image features based on the real-time fusion weights to obtain fused perception features.

5. The drone-based vehicle perception optimization method of claim 1, wherein, The step of performing dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between a vehicle and a UAV comprises: acquiring UAV calibration conversion data and vehicle calibration conversion data; performing dynamic pose matching based on the first environment data and the second environment data to obtain a relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system; generating the relative pose transformation information between the vehicle and the UAV based on the UAV calibration conversion data, the vehicle calibration conversion data and the relative pose transformation matrix.

6. The drone-based vehicle perception optimization method of claim 5, wherein, The step of performing dynamic pose matching based on the first environment data and the second environment data to obtain a relative pose transformation matrix of the UAV relative to the vehicle in a unified navigation coordinate system comprises: performing preset coordinate system conversion on the first environment data and the second environment data to obtain a first composite inverse rotation matrix and a second composite inverse rotation matrix; generate a relative attitude matrix of the UAV in a unified navigation coordinate system relative to the vehicle based on the first composite inverse rotation matrix and the second composite inverse rotation matrix; generate a relative translation vector of the UAV in the unified navigation coordinate system relative to the vehicle based on a navigation center coordinate of the vehicle; generate a relative pose transformation matrix according to the relative attitude matrix and the relative translation vector.

7. The UAV-based vehicle perception optimization method of claim 5, wherein, The step of obtaining the UAV calibration conversion data and the vehicle calibration conversion data comprises: performing intrinsic parameter calibration conversion on the UAV camera and the vehicle camera respectively to obtain UAV camera conversion data and vehicle camera conversion data; performing extrinsic parameter calibration on the UAV body and the UAV camera according to the UAV camera conversion data to obtain the UAV calibration conversion data; performing extrinsic parameter calibration on the vehicle body and the vehicle camera based on the vehicle camera conversion data to obtain the vehicle calibration conversion data.

8. An unmanned aerial vehicle based vehicle perception optimization apparatus, comprising: The UAV-based vehicle perception optimization device comprises: a data acquisition module configured to acquire first environment data collected by a UAV sensing module and second environment data collected by a vehicle-end sensing module; a pose conversion module configured to perform dynamic pose estimation based on the first environment data and the second environment data to obtain relative pose transformation information between the vehicle and the UAV; a feature analysis module configured to perform cross-view feature analysis according to the relative pose transformation information, the first environment data and the second environment data to obtain fused perception features; a perception module configured to generate a vehicle environment perception result based on the fused perception features.

9. A vehicle characterized by comprising: The vehicle comprises a memory, a processor, and a UAV-based vehicle perception optimization program stored on the memory and executable on the processor, and the UAV-based vehicle perception optimization program is configured to implement the steps of the UAV-based vehicle perception optimization method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium stores a UAV-based vehicle perception optimization program, and the UAV-based vehicle perception optimization program is executed by the processor to implement the steps of the UAV-based vehicle perception optimization method according to any one of claims 1 to 7.

Images