Unmanned aerial vehicle perception method, apparatus, and vehicle

Abstract

The embodiment of the application provides a kind of unmanned aerial vehicle sensing method, device and vehicle, it is related to image processing technical field, can realize low delay, high reliable low altitude unmanned aerial vehicle real-time sensing and early warning.The method comprises the following steps: firstly, according to the binocular image containing unmanned aerial vehicle, obtain the first feature map and the second feature map;Then, the multiple channels of the first feature map and the multiple channels of the second feature map are divided into multiple channel groups according to the corresponding relationship of channel;Further, for each channel group, based on multiple disparity hypotheses, determine the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map in the channel group;Subsequently, for each feature point in the first feature map, according to the feature similarity of the feature point, determine the depth distribution and geometric confidence of the feature point;Finally, according to the depth distribution and geometric confidence of each feature point in the first feature map, generate the bird's eye view feature map.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and vehicle for sensing unmanned aerial vehicles (UAVs). Background Technology

[0002] With the rapid development and widespread adoption of consumer and industrial drone technology, the number of aircraft in low-altitude airspace (typically below 120 meters) has increased dramatically. The extensive use of these drones in logistics delivery, aerial photography, agricultural plant protection, and security patrols has led to their frequent appearance in urban and suburban airspace, posing a new threat to the driving safety of intelligent connected vehicles.

[0003] To address the aforementioned issues, several technological attempts have been made. Existing solutions primarily focus on two directions: first, enhancing the detection capability of single-vehicle perception systems for drone targets; and second, achieving threat information sharing through inter-vehicle communication. The former still utilizes the existing perception framework, while the latter continues the traditional collaborative early warning technology approach.

[0004] However, existing technologies are often hampered by high latency and high computational demands, making it difficult for intelligent connected vehicles to achieve omnidirectional real-time perception and proactive prediction and early warning of low-altitude drones in complex scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a method, device and vehicle for drone perception, which aims to achieve low-latency and high-reliability real-time perception and early warning of low-altitude drones.

[0006] In a first aspect, this application provides a UAV perception method, the method comprising: acquiring a first feature map and a second feature map based on a binocular image containing the UAV, the binocular image including a first image and a second image, the first feature map being derived from the first image, the second feature map being derived from the second image, a feature map including multiple channels, each feature point on the feature map corresponding to a feature vector composed of the channels; dividing the multiple channels of the first feature map and the multiple channels of the second feature map into multiple channel groups according to the correspondence of the channels; for each channel group, determining the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map based on multiple disparity assumptions, the disparity assumptions being used to indicate the pixel offset of the corresponding feature point on the second feature map relative to each feature point on the first feature map; for each feature point in the first feature map, determining the depth distribution and geometric confidence of the feature point based on the feature similarity corresponding to the feature point under each channel group and each disparity assumption, the geometric confidence representing the reliability of the depth estimation result of the corresponding feature point; generating a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map, the bird's-eye view feature map being used to determine the three-dimensional spatial position of the UAV.

[0007] The UAV perception method provided in this application avoids the high computational cost of dense disparity search for all pixels in the entire image in traditional methods by dividing the channels of the first and second feature maps into multiple channel groups and calculating feature similarity based on multiple disparity assumptions. This reduces processing latency. Furthermore, each feature point simultaneously obtains its depth distribution and geometric confidence based on the feature similarity under each channel group and each disparity assumption, quantifying the reliability of depth estimation. This allows for the effective identification and elimination of unreliable depth information in complex scenes, avoiding perception errors caused by mismatches. Finally, a bird's-eye view feature map is generated based on the depth distribution and geometric confidence of each feature point, providing an efficient and robust feature foundation for determining the UAV's three-dimensional spatial position. This enables real-time perception and proactive prediction and early warning of low-altitude UAVs with low latency.

[0008] In conjunction with the first aspect above, in one possible implementation, for each channel group, based on multiple preset disparity assumptions, the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map within the channel group is determined, including: in the target channel group, for the target disparity assumption, determining the reference feature point in the second feature map corresponding to the target feature point in the first feature map, wherein the target channel group is any one of multiple channel groups, and the target disparity assumption is any one of multiple disparity assumptions; and based on the target feature point and the reference feature point, determining the grouping cost body of the target feature point under the target disparity assumption as the feature similarity between the two.

[0009] Based on the above technical means, this application determines the first feature point and the reference feature point in the second feature map in the target channel group according to the target disparity assumption and calculates the group cost body, so that the feature similarity can be calculated independently in the channel dimension, thereby reducing the complexity of a single similarity calculation and facilitating parallel processing.

[0010] In conjunction with the first aspect above, in one possible implementation, for each feature point in the first feature map, the depth distribution and geometric confidence of the feature point are determined based on the feature similarity corresponding to the feature point under each channel group and each disparity hypothesis. This includes: for each feature point in the first feature map, aggregating the grouped cost bodies corresponding to the feature point under each channel group and each disparity hypothesis to obtain the cost distribution of the feature point, wherein the cost distribution characterizes the matching quality of the feature point under different disparity hypotheses; and based on the cost distribution, determining the depth distribution and geometric confidence of the feature point, wherein the depth distribution is obtained by normalizing the cost distribution under multiple disparity hypotheses, and the geometric confidence is determined based on the sharpness or unimodality of the cost distribution.

[0011] Based on the above technical means, this application obtains the cost distribution by aggregating the grouped cost volumes of each feature point under each channel group and each disparity assumption, and determines the geometric confidence level based on the sharpness or unimodality of the cost distribution, so that the reliability of depth estimation can be directly obtained from the concentration of matching costs without additional calculation.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, generating a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map includes: for any feature point in the first feature map, determining the contribution of the feature point to the projection of the feature point onto the bird's-eye view space under each disparity assumption based on the depth distribution of the feature point; weighting the contribution of the feature point under each disparity assumption based on the geometric confidence of the feature point to obtain the bird's-eye view feature contribution; and generating a bird's-eye view feature map based on the bird's-eye view feature contribution corresponding to each feature point.

[0013] Based on the above technical means, this application determines the contribution of feature points to the bird's-eye view projection under each parallax assumption according to the depth distribution and weights it with geometric confidence, so that unreliable depth information is reduced and fused when generating bird's-eye view feature maps, thereby improving the accuracy of the representation of the UAV position in the bird's-eye view feature maps.

[0014] In conjunction with the first aspect above, in one possible implementation, the method further includes: determining an observation quality factor based on the image features of a local region in the first image, wherein the observation quality factor characterizes the observation quality of the local region; and weighting the contribution of the feature points under each disparity assumption based on the geometric confidence level corresponding to the feature points to obtain the bird's-eye view feature contribution, including: weighting the contribution of the feature points under each disparity assumption based on the geometric confidence level and the observation quality factor corresponding to the feature points to obtain the bird's-eye view feature contribution.

[0015] Based on the above technical means, this application introduces observation quality factor and geometric confidence to jointly weight the contribution, so that the features in the local areas of the image with poor observation quality (such as occlusion and weak texture) are further deweighted when generating the bird's-eye view feature map, thereby enhancing the robustness of the bird's-eye view feature map in complex environments.

[0016] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: detecting the UAV in the bird's-eye view feature map, obtaining the UAV's three-dimensional spatial state information and detection confidence, wherein the detection confidence characterizes the reliability of the UAV detection results; determining information credibility data based on the target vehicle's sensor state information, wherein the information credibility data is used by the receiving end to evaluate the credibility of the three-dimensional spatial state information; and generating a collaborative perception message based on the three-dimensional spatial state information, detection confidence, and information credibility data, wherein the collaborative perception message is used to assist the receiving end in perceiving the UAV.

[0017] Based on the aforementioned technical means, this application generates collaborative perception messages that include three-dimensional spatial state information, detection confidence, and information credibility data based on sensor state information. This enables the receiving end to simultaneously assess the reliability of the UAV detection results and the credibility of the information source itself, thereby providing a quantifiable fusion basis for multi-vehicle collaborative perception.

[0018] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: receiving collaborative perception messages from other traffic participants; determining the information credibility of other traffic participants based on the information credibility data in the collaborative perception messages; and fusing the three-dimensional spatial state information from multiple other traffic participants based on the information credibility to obtain the fused UAV state information.

[0019] Based on the aforementioned technical means, this application determines the information credibility of each traffic participant by means of the information credibility in the collaborative perception message, and fuses the three-dimensional spatial state information from multiple sources based on the credibility, so that the perception results of high credibility participants have a greater weight in the fusion, thereby improving the overall reliability and accuracy of the fused UAV state information.

[0020] In conjunction with the first aspect mentioned above, in one possible implementation, before fusing the three-dimensional spatial state information from multiple other traffic participants based on information credibility, the method further includes: determining the communication delay duration based on the timestamp carried in the collaborative sensing message and the current time of the target vehicle; and performing motion compensation on the three-dimensional spatial state information in the collaborative sensing message based on the communication delay duration to obtain the three-dimensional spatial state information at the current time.

[0021] Based on the aforementioned technical means, this application determines the communication delay duration by comparing the timestamp in the collaborative sensing message with the current time, and performs motion compensation on the three-dimensional spatial state information, so that the state information from different participants before fusion is unified to the same time, thereby eliminating the negative impact of spatiotemporal misalignment caused by communication delay on fusion accuracy.

[0022] In conjunction with the first aspect mentioned above, in one possible implementation, based on information credibility, the three-dimensional spatial state information from multiple other traffic participants is fused to obtain the fused UAV state information, including: determining the fusion weight corresponding to each other traffic participant based on information credibility; and weighting and fusing the current three-dimensional spatial state information of each other traffic participant according to each fusion weight to obtain the fused UAV state information.

[0023] Based on the aforementioned technical means, this application determines the fusion weights corresponding to each other traffic participant based on information credibility and performs weighted fusion, so that high credibility data contributes more and low credibility data contributes less in the fusion result, thereby achieving reliable fusion of multi-source heterogeneous perception information.

[0024] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: acquiring the motion trajectory sequence of the target drone when it disappears from the perception field of view; performing trajectory extrapolation and intent classification based on the motion trajectory sequence to determine the predicted trajectory of the target drone and its probability distribution on at least one behavioral intent; determining the probability of the target drone's presence within the current perception blind zone of the target vehicle based on the predicted trajectory and probability distribution; and generating a virtual predicted target within the perception blind zone when the probability of presence is higher than a preset threshold, wherein the state of the virtual predicted target is determined based on the predicted trajectory.

[0025] Based on the aforementioned technical means, this application acquires the motion trajectory sequence of the UAV after it disappears from the perception field of view and performs trajectory extrapolation and intent classification. Then, based on the predicted trajectory and the probability of behavioral intent, a virtual predicted target is generated in the perception blind zone. This enables the vehicle to maintain continuous estimation of the future state and position of the UAV even after visual loss, thereby providing a reliable virtual perception capability for active early warning in the blind zone.

[0026] Secondly, this application provides a drone sensing device, comprising: a feature extraction module, a feature grouping module, a feature matching module, a depth perception module, and a spatial mapping module. The feature extraction module is used to acquire a first feature map and a second feature map based on a binocular image containing the drone. The binocular image includes a first image and a second image. The first feature map is derived from the first image, and the second feature map is derived from the second image. Each feature map includes multiple channels, and each feature point on the feature map corresponds to a feature vector composed of the channels. The feature grouping module is used to divide the multiple channels contained in the first feature map and the multiple channels contained in the second feature map into multiple channel groups according to the correspondence between the channels. The feature matching module is used to, for each channel group, determine each feature on the first feature map within that channel group based on multiple disparity assumptions. The first feature map has a feature similarity between a point and its corresponding feature point on the second feature map. The disparity hypothesis is used to indicate the pixel offset of the corresponding feature point on the second feature map relative to each feature point on the first feature map. The second feature map has a depth perception module, which determines the depth distribution and geometric confidence of each feature point in the first feature map based on the feature similarity of the feature point under each channel group and each disparity hypothesis. The geometric confidence represents the reliability of the depth estimation result of the corresponding feature point. The third feature map has a spatial mapping module, which generates a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map. The bird's-eye view feature map is used to determine the three-dimensional spatial position of the UAV.

[0027] In conjunction with the second aspect above, in one possible implementation, the feature matching module is specifically used to: in the target channel group, for the target disparity hypothesis, determine the reference feature point in the second feature map corresponding to the target feature point in the first feature map, where the target channel group is any one of multiple channel groups and the target disparity hypothesis is any one of multiple disparity hypotheses; based on the target feature point and the reference feature point, determine the grouping cost body of the target feature point under the target disparity hypothesis, as the feature similarity between the two.

[0028] In conjunction with the second aspect above, in one possible implementation, the depth perception module is specifically used to: for each feature point in the first feature map, aggregate the grouped cost bodies corresponding to the feature point under each channel group and each disparity hypothesis to obtain the cost distribution of the feature point, the cost distribution characterizing the matching quality of the feature point under different disparity hypotheses; based on the cost distribution, determine the depth distribution and geometric confidence of the feature point, the depth distribution being obtained by normalizing the cost distribution under multiple disparity hypotheses, and the geometric confidence being determined according to the sharpness or unimodality of the cost distribution.

[0029] In conjunction with the second aspect above, in one possible implementation, the spatial mapping module is specifically used for: for any feature point in the first feature map, determining the contribution of the feature point to the bird's-eye view projection under each disparity assumption based on the depth distribution of the feature point; weighting the contribution of the feature point under each disparity assumption based on the geometric confidence corresponding to the feature point to obtain the bird's-eye view feature contribution; and generating a bird's-eye view feature map based on the bird's-eye view feature contribution corresponding to each feature point.

[0030] In conjunction with the second aspect above, in one possible implementation, the depth perception module is further configured to: determine an observation quality factor based on the image features of a local region in the first image, wherein the observation quality factor characterizes the observation quality of the local region; and weight the contribution of the feature points under each disparity assumption based on the geometric confidence and the observation quality factor corresponding to the feature points to obtain the bird's-eye view feature contribution.

[0031] In conjunction with the second aspect above, in one possible implementation, the device further includes a communication module for: detecting a drone in a bird's-eye view feature map; acquiring the drone's three-dimensional spatial state information and detection confidence level, wherein the detection confidence level characterizes the reliability of the drone detection results; determining information credibility data based on the target vehicle's sensor state information, wherein the information credibility data is used by the receiving end to evaluate the credibility level of the three-dimensional spatial state information; and generating a collaborative perception message based on the three-dimensional spatial state information, detection confidence level, and information credibility data, wherein the collaborative perception message is used to assist the receiving end in perceiving the drone.

[0032] In conjunction with the second aspect above, in one possible implementation, the communication module is further configured to: receive collaborative perception messages from other traffic participants; determine the information credibility of other traffic participants based on the information credibility data in the collaborative perception messages; and fuse the three-dimensional spatial state information from multiple other traffic participants based on the information credibility to obtain the fused UAV state information.

[0033] In conjunction with the second aspect above, in one possible implementation, the device further includes a cooperative sensing module, used to: determine the communication delay duration based on the timestamp carried in the cooperative sensing message and the current time of the target vehicle; and perform motion compensation on the three-dimensional spatial state information in the cooperative sensing message based on the communication delay duration to obtain the three-dimensional spatial state information at the current time.

[0034] In conjunction with the second aspect above, in one possible implementation, the collaborative perception module is specifically used to: determine the fusion weights corresponding to each other traffic participant based on information credibility; and perform weighted fusion of the current three-dimensional spatial state information of each other traffic participant according to each fusion weight to obtain the fused UAV state information.

[0035] In conjunction with the second aspect above, in one possible implementation, the device further includes a blind spot prediction module, configured to: acquire a motion trajectory sequence of the target drone when it disappears from the perception field of view; perform trajectory extrapolation and intent classification based on the motion trajectory sequence to determine the predicted trajectory of the target drone and its probability distribution on at least one behavioral intent; determine the probability of the target drone's presence within the current perception blind spot of the target vehicle based on the predicted trajectory and probability distribution; and generate a virtual predicted target within the perception blind spot if the probability of presence is higher than a preset threshold, wherein the state of the virtual predicted target is determined based on the predicted trajectory.

[0036] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, the electronic device implements the method of the first aspect described above.

[0037] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to implement the method described in the first aspect.

[0038] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the steps of the relevant method described in the first aspect above, so as to implement the method of the first aspect above.

[0039] In a sixth aspect, this application also provides a vehicle comprising: a binocular camera and a data processing unit; when the binocular camera acquires binocular image data of a drone surrounding the vehicle, the data processing unit may execute the method described in the first aspect.

[0040] The beneficial effects of the second to sixth aspects mentioned above can be referred to the corresponding descriptions in the first aspect, and will not be repeated here.

[0041] It should be noted that any of the possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a drone perception method provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for determining a bird's-eye view feature map provided in an embodiment of this application; Figure 3 A schematic diagram of an encapsulation structure for a collaborative sensing message provided in an embodiment of this application; Figure 4 A timing diagram illustrating the interaction between different traffic participants, provided as an embodiment of this application; Figure 5 A flowchart illustrating a method for enhancing bird's-eye view feature maps provided in an embodiment of this application; Figure 6 A flowchart illustrating a virtual target generation method provided in an embodiment of this application; Figure 7 A detailed flowchart illustrating a drone perception method provided in this application embodiment; Figure 8 This is a schematic diagram illustrating the composition of a drone sensing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a drone sensing device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0046] In the embodiments of this application, the terms "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," "fourth," "fifth," and "sixth" may explicitly or implicitly include one or more of that feature.

[0047] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0048] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0049] Currently, vehicle-mounted environmental perception systems are primarily optimized for traditional road targets such as ground vehicles, pedestrians, and traffic signs. However, they suffer from a series of shortcomings when dealing with "low-altitude, slow-moving, and small-sized" drone targets. At the single-vehicle perception level, millimeter-wave radar struggles to provide accurate three-dimensional orientation information for non-metallic, irregularly shaped targets with small radar cross-sections. Monocular vision cameras lack direct depth perception capabilities, making it difficult to determine the true distance between the drone and the vehicle. While lidar offers high accuracy, its high cost limits its widespread application in mass-produced vehicles. Furthermore, at the field-of-view level, existing vehicle-mounted environmental perception systems are mostly forward-facing designs, exhibiting blind spots in both vertical and horizontal directions, making them ill-suited for dealing with omnidirectional drones.

[0050] Some solutions that improve drone detection performance through multi-sensor fusion, vehicle-to-everything (V2X) wireless communication technology, or feature enhancement may suffer from problems such as environmental sensitivity, excessive computational complexity, or the inability to detect non-connected third-party drones while only supporting state exchange between connected vehicles. Others may rely on signal quality and struggle to operate stably in complex electromagnetic environments. Furthermore, traditional binocular stereo vision-based bird's-eye view (BEV) spatial perception methods incur significant computational and memory overhead, often resulting in high latency in practical applications.

[0051] Based on this, this application provides a UAV perception method. By dividing the channels of the first and second feature maps into multiple channel groups and calculating feature similarity based on multiple disparity assumptions, it avoids the high computational cost of dense disparity search for all pixels in the entire image in traditional methods, thereby reducing processing latency. Furthermore, each feature point simultaneously obtains its depth distribution and geometric confidence based on the feature similarity under each channel group and each disparity assumption, quantifying the reliability of depth estimation. This allows for the effective identification and elimination of unreliable depth information in complex scenes, avoiding perception errors caused by mismatches. Finally, a bird's-eye view feature map is generated based on the depth distribution and geometric confidence of each feature point, providing an efficient and robust feature foundation for determining the three-dimensional spatial position of the UAV. This enables real-time perception and proactive prediction and early warning of low-altitude UAVs with low latency.

[0052] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0053] The drone perception method provided in this application can be applied to vehicles equipped with binocular cameras. When the binocular camera acquires binocular images of the drone around the vehicle, the data processing unit inside the vehicle can gradually implement the drone perception method provided in this application to perceive the three-dimensional spatial position of the drone.

[0054] In one possible implementation, the vehicle may be an intelligent connected vehicle, an autonomous vehicle, an assisted driving vehicle, or other types of motor vehicles with onboard computing capabilities and binocular camera installation conditions. This application does not impose specific restrictions on the specific vehicle model, power type, autonomous driving level, hardware configuration, etc.

[0055] In one possible implementation, the data processing unit can be an on-board computing platform integrated in the vehicle, such as an embedded system based on a graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or dedicated neural network acceleration chip, or a general-purpose processor system with sufficient computing power. This application does not impose specific limitations on the specific form and configuration of the data processing unit.

[0056] In some embodiments, the data processing unit can be communicatively connected to a binocular camera to receive binocular image data and execute the UAV perception method provided in this application. The data processing unit can also be connected to a vehicle's positioning module, V2X communication module, etc., to obtain vehicle pose information and transmit / receive cooperative perception messages.

[0057] In one possible implementation, the binocular camera consists of two cameras fixed in relative positions with a known baseline distance between them, which can acquire left and right eye image pairs of the vehicle's surrounding environment.

[0058] In some embodiments, the binocular camera can be mounted inside the windshield, on the roof, near the rearview mirror, or other suitable locations for acquiring forward and upper-side field of view. Its mounting angle can be adjusted according to actual sensing needs to cover the detection range of low-altitude unmanned aerial vehicle targets. The image data acquired by the binocular camera is transmitted to a data processing unit for subsequent processing such as feature extraction, depth estimation, and bird's-eye view feature map generation.

[0059] The UAV perception method provided in this application can use both the left and right eye images captured by the binocular camera as a reference for generating a bird's-eye view feature map, and this application does not impose any limitations. In the following embodiments, the left eye image (hereinafter referred to as the first image) is used as an example to generate the bird's-eye view feature map.

[0060] For example, such as Figure 1 As shown, the UAV perception method provided in this application embodiment includes the following steps S101-S105: S101. Based on the binocular image containing the drone, obtain the first feature map and the second feature map.

[0061] The binocular image includes a first image and a second image. The first feature map comes from the first image, and the second feature map comes from the second image. A feature map includes multiple channels, and each feature point on the feature map corresponds to a feature vector composed of the channels.

[0062] In this embodiment, a feature map refers to a multidimensional tensor obtained after feature processing of an input image. Its spatial dimension corresponds to the pixel position of the original image, and its depth dimension consists of multiple channels. Each channel can be understood as a response map to a certain type of abstract feature of the image. A channel refers to an independent slice in the depth dimension of the feature map. Different channels can respond to different visual patterns, such as edge direction, texture frequency, or local shape. A feature point refers to a coordinate position in the spatial dimension of the feature map. The values ​​of this position in all channels together constitute a feature vector, which can be used to describe the visual features of the corresponding local region in the original image.

[0063] In some embodiments, the first feature map and the second feature map can be obtained by processing the first image and the second image respectively through a feature extraction network with shared weights. This feature extraction network can be a backbone neural network based on a convolutional neural network or a Transformer model architecture, with its input being the raw image data acquired by a stereo camera and its output being a feature map with multiple channels. The first image and the second image constitute a stereo image pair, and there is disparity between them in the horizontal direction, which can provide geometric constraints for subsequent depth estimation.

[0064] In some embodiments, the data processing unit can communicate via an onboard binocular camera to synchronously acquire left and right eye image pairs containing the UAV. After completing image timestamp alignment and epipolar correction, the first image and the second image are respectively input into a backbone neural network with shared weights to obtain a first feature map and a second feature map.

[0065] In some embodiments, the data processing unit may further: acquire multi-scale first and second feature maps through a backbone neural network with shared weights, based on user pre-configuration. The feature maps at different scales have different spatial resolutions, which can adapt to the perception needs of UAV targets of different sizes and distances.

[0066] In one possible implementation, the data processing unit can input a first image and a second image containing the UAV into a residual network (ResNet) with shared weights and identical structure. The ResNet network can perform stepwise downsampling and feature abstraction on the input images through multiple residual blocks, and finally output a multi-channel feature map with a spatial resolution of one-sixteenth of the original image from a specific level, which serves as the first feature map. and the second feature map ,in Represents the scale.

[0067] As can be seen from step S101, the first feature map and the second feature map are obtained synchronously through the feature extraction network with shared weights, so that the features of the left and right eye images can be expressed in the same semantic space, thereby providing a consistent visual description for subsequent depth estimation based on feature similarity.

[0068] S102. Divide the multiple channels of the first feature map and the multiple channels of the second feature map into multiple channel groups according to the correspondence between the channels.

[0069] In this embodiment of the application, a channel group refers to a subset of channels obtained by dividing the channels with corresponding relationships in the first feature map and the second feature map. Each channel group contains a portion of the channels in the first feature map and the same number of channels corresponding to the positions in the second feature map. The channels in the same channel group are at the same level in the feature extraction network and have a semantic correspondence. Different channel groups do not overlap in the channel dimension.

[0070] In some embodiments, all channels contained in the first feature map and the second feature map are evenly divided into multiple channel groups, and each channel group contains the same number of channels. The i-th group of channels in the first feature map and the i-th group of channels in the second feature map form the i-th channel group, and the index positions of the channels in the group correspond one-to-one with those in their respective feature maps.

[0071] For example, the 256 channels of the first feature map are evenly divided into 8 groups along the channel dimension, each group containing 32 channels. The 256 channels of the second feature map are divided into 8 groups in the same way. The first group of channels of the first feature map and the first group of channels of the second feature map together form the first channel group, and so on to form all 8 channel groups.

[0072] In some embodiments, after acquiring the first feature map and the second feature map, the data processing unit may perform a uniform division operation on the channel dimension of the two groups of feature maps according to a preset number of groups G, and classify the channel segments with the same channel index in the first feature map and the second feature map into the same channel group, thereby forming G channel groups.

[0073] In some embodiments, the data processing unit may also dynamically adjust the number of channel groups based on its real-time load status. For example, when computing resources are abundant, fewer groups may be used to obtain more refined feature matching, while more groups may be used when computing resources are scarce to reduce computational overhead. Alternatively, the grouping strategy may be adaptively adjusted based on the density of drone targets in the scene.

[0074] In one possible implementation, the data processing unit can evenly divide the channel dimensions of the first feature map and the second feature map into 8 groups, with the number of channels in each group being one-eighth of the original number of channels. Channel groups in the first feature map and the second feature map that are in the same index interval are paired into the same channel group. Subsequently, feature similarity is calculated independently only within each channel group, and no cross-group correlation calculation is performed between different channel groups.

[0075] As can be seen from step S102, by dividing the first feature map and the second feature map into multiple channel groups according to the channel correspondence, the subsequent feature similarity calculation can be performed independently in each channel group. The number of channels in each group is much smaller than the total number of channels, thereby reducing the matrix operation scale and memory access overhead of a single calculation. At the same time, parallel calculation can be performed between each group to improve calculation efficiency and reduce calculation time.

[0076] S103. For each channel group, based on multiple disparity assumptions, determine the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map within the channel group.

[0077] The disparity hypothesis is used to indicate the pixel offset of a corresponding feature point on the second feature map relative to each feature point on the first feature map.

[0078] In this embodiment, the disparity hypothesis refers to a discretized preset estimate of the possible horizontal pixel offset of the corresponding point of a pixel in the first image in the second image during the stereo matching process. Each disparity hypothesis value corresponds to a possible target depth level. The larger the disparity hypothesis value, the closer the target is to the camera, and the smaller the disparity hypothesis value, the farther the target is from the camera.

[0079] In some embodiments, multiple disparity assumptions constitute a discrete integer sequence from zero to the maximum disparity search value. The maximum disparity search value can be preset based on the baseline length and focal length of the binocular camera, as well as the shortest distance to the UAV target to be detected. When the disparity assumption is zero, the corresponding target is at infinity. Operationally on the feature maps involves matching feature points on the first feature map with feature points at the same horizontal coordinates on the second feature map. As the disparity assumption value increases, the number of pixels the reference feature point on the second feature map is offset to the left relative to the feature point on the first feature map increases accordingly.

[0080] In some embodiments, the data processing unit can obtain multiple parallax assumption values ​​by reading preset parallax search range parameters. The parallax search range parameters can be calculated offline by the system based on the calibration parameters of the binocular camera and the maximum distance requirement for UAV detection and stored in a configuration file. The data processing unit can load the parameters from the configuration file when performing this step.

[0081] In some embodiments, the data processing unit may also: dynamically adjust the search range and granularity of the parallax hypothesis according to the size and distance distribution of the UAV target in the current scene, using a larger search range and a coarser parallax step size in long-range detection scenarios, and a smaller search range and a finer parallax step size in short-range detection scenarios, in order to balance computational overhead and depth estimation accuracy.

[0082] In one possible implementation, for a target channel group and a target disparity hypothesis, the data processing unit can determine the reference feature point in the second feature map corresponding to the target feature point on the first feature map as the position of the target feature point after offsetting its horizontal coordinate from the target disparity hypothesis value. Then, it calculates the inner product between the feature vector of the target feature point and the feature vector of the reference feature point, and uses the result of this inner product as the feature similarity of the target feature point under the target disparity hypothesis. This feature similarity reflects the probability that the two feature points match each other under the current disparity hypothesis.

[0083] As can be seen from step S103, by introducing multiple discretized disparity assumptions and independently calculating feature similarity within each channel group, the matching degree of each feature point at different depth probabilities can be evaluated in parallel and efficiently. At the same time, the calculation within the group avoids redundant calculations in the entire channel dimension, reduces processing latency, and provides quantitative basic data for subsequent parsing of depth information and evaluation of depth reliability.

[0084] S104. For each feature point in the first feature map, determine the depth distribution and geometric confidence of the feature point based on the feature similarity of the feature point under each channel group and each disparity assumption.

[0085] Among them, geometric confidence represents the reliability of the depth estimation result of the corresponding feature point.

[0086] In this embodiment, the depth distribution refers to the probability distribution of a single feature point in the first feature map under various discrete disparity assumptions, representing the likelihood of the feature point being located at different depth levels. Geometric confidence is a scalar value decoded from the aggregation results of feature similarity, used to characterize the reliability of the depth estimation result for the corresponding feature point; a higher geometric confidence indicates a more reliable depth estimation for the feature point.

[0087] In some embodiments, the depth distribution can be obtained by normalizing the feature similarity of feature points under each disparity hypothesis. The depth distribution can be represented as a probability vector in which the sum of the values ​​in the disparity dimension is one, and the disparity hypothesis with the largest probability value is the optimal disparity estimate of the feature point.

[0088] In some embodiments, at each disparity hypothesis level in the depth distribution, a feature point has an independent geometric confidence scalar that reflects the sharpness of the matching result of the feature point under the current disparity hypothesis or the degree of distinction from other disparity hypotheses.

[0089] In some embodiments, the data processing unit can aggregate the feature similarity of feature points under the same disparity assumption in each channel group to obtain the comprehensive similarity of the feature point under the disparity assumption, then traverse all disparity assumptions to form the comprehensive similarity vector of the feature point, and calculate the depth distribution and geometric confidence based on the comprehensive similarity vector.

[0090] In some embodiments, the data processing unit may further: mark feature points with geometric confidence levels below a preset threshold as unreliable points and record their spatial locations, so as to suppress or exclude the contributions of these feature points in subsequent processes, and generate a spatial distribution map of unreliable areas in the current scene to provide a perception quality reference for the vehicle's decision planning module.

[0091] In one possible implementation, the data processing unit, for a feature point on the first feature map, first sums the feature similarities of that feature point under the same disparity assumption in each channel group to obtain the total similarity under that disparity assumption. The total similarity of all disparity assumptions is then normalized to obtain the depth distribution. Then, for each disparity assumption level, the difference between the total similarity of that level and the maximum total similarity of the other levels is calculated. This difference is used as the geometric confidence level corresponding to that disparity assumption level. The larger the difference, the more distinct the level is from other levels, and the more reliable the depth estimation.

[0092] As can be seen from step S104, by simultaneously determining the depth distribution of each feature point and the geometric confidence sequence corresponding to the depth distribution, the estimation reliability of each depth hypothesis level is independently quantified, thus providing a layer-by-layer confidence basis for weighting each projection contribution along the depth direction in the subsequent bird's-eye view feature map generation, so that the feature contribution of reliable depth levels is retained and the feature contribution of unreliable depth levels is effectively suppressed.

[0093] S105. Generate a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map.

[0094] Among them, the bird's-eye view feature map is used to determine the three-dimensional spatial position of the drone.

[0095] In this embodiment of the application, the bird's-eye view feature map refers to the feature representation that describes the surrounding environment information from a top-down perspective with the vehicle as the center. Its spatial grid corresponds to the spatial region in the actual physical space. Each grid stores a high-dimensional feature vector that is projected and aggregated from image features, which can be used to directly detect the three-dimensional spatial position of targets such as drones.

[0096] In some embodiments, the coverage area of ​​the bird's-eye view feature map can be pre-set according to the vehicle perception requirements, for example, extending tens of meters in front and behind the vehicle and extending to the left and right by several lane widths. The number of channels in the bird's-eye view feature map is the same as the number of channels in the first feature map, so that the projected features can retain the semantic expressive power of the original image, which facilitates the subsequent direct 3D target detection on the bird's-eye view feature map.

[0097] In some embodiments, the data processing unit can obtain the bird's-eye view feature map by traversing each feature point on the first feature map, determining its optimal disparity hypothesis from the depth distribution based on the geometric confidence of each feature point, projecting the feature point onto the corresponding grid position in the bird's-eye view space using the optimal disparity hypothesis and the camera projection model, and filling the feature vector of the feature point into the corresponding grid of the bird's-eye view feature map.

[0098] In some embodiments, the data processing unit may further: when projecting feature points onto a bird's-eye view feature map, if multiple feature points fall into the same grid, perform a summation or averaging operation on the feature vectors within that grid to avoid loss of effective information, and simultaneously record the number of feature points received by each grid for subsequent normalization processing of the feature density.

[0099] In one possible implementation, for each feature point on the first feature map, the data processing unit can select the disparity hypothesis with the highest confidence value from the geometric confidence sequence of that feature point as the optimal disparity hypothesis. Then, it obtains the depth probability value corresponding to the optimal disparity hypothesis from the depth distribution of that feature point, and projects the feature vector of that feature point onto the corresponding grid of the bird's-eye view feature map using the depth level corresponding to the optimal disparity hypothesis. If the depth probability value is lower than a preset threshold, the feature point is discarded and not projected.

[0100] As can be seen from step S105, by generating a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point, the feature information in the two-dimensional image space is elevated to the representation in the three-dimensional bird's-eye view space, so that the spatial position of the UAV target can be directly described from the bird's-eye view. At the same time, by using geometric confidence to guide the selection of projection depth, the spatial position deviation caused by depth estimation error can be effectively suppressed, providing a more reliable spatial feature basis for subsequent three-dimensional target detection.

[0101] In this embodiment of the application, in step S103 above, the data processing unit may further form multiple grouped related volumes in each channel group to serve as the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map in the channel group.

[0102] For example, the data processing unit can, within the target channel group, determine a reference feature point in the second feature map corresponding to the target feature point in the first feature map, based on the target disparity assumption. Then, based on the target feature point and the reference feature point, it determines the grouping cost volume (i.e., grouping correlation volume) of the target feature point under the target disparity assumption, as the feature similarity between the two.

[0103] The target channel group is any one of multiple channel groups, and the target disparity hypothesis is any one of multiple disparity hypotheses.

[0104] In some embodiments, the data processing unit can select any channel group as the target channel group from the multiple channel groups divided in step S102, and select any disparity hypothesis value as the target disparity hypothesis from the preset disparity search range. By traversing all channel groups and all combinations of disparity hypotheses, the data processing unit can obtain the grouping cost body of the corresponding feature points between the first feature map and the second feature map under each combination.

[0105] In some embodiments, the data processing unit may further: when calculating the group cost body, for all feature points in the same channel group and the same disparity hypothesis, use matrix operations to complete the feature similarity calculation of all feature point pairs at once, and use the tensor structure of the feature map to perform batch inner product operations, so as to reduce the number of point-by-point loop calculations and make full use of the hardware parallel acceleration capability.

[0106] In one possible implementation, the data processing unit determines the grouping cost body for the target channel group under the target disparity hypothesis as follows: First, extract the feature vector of the target feature point within the target channel group from the first feature map. The dimension of this feature vector is equal to the number of channels in the target channel group. Then, determine the horizontal coordinate of the reference feature point corresponding to the feature point in the second feature map based on the target disparity hypothesis. The horizontal coordinate of the reference feature point is the horizontal coordinate of the feature point in the first feature map minus the target disparity hypothesis value, while the vertical coordinate remains unchanged. Next, extract the feature vector of the reference feature point within the target channel group from the second feature map. Finally, calculate the inner product between these two feature vectors, and use the result of the inner product as the grouping cost body of the target feature point under the target disparity hypothesis.

[0107] For example, the grouping cost volume of target feature points under the target disparity assumption. One way to determine this is as follows: ; in, and The left and right eye images are respectively in the first... Scale, g-th channel group, and feature vectors of corresponding spatial locations (i.e., target feature points and reference feature points). This represents the inner product operation, where h is the vertical pixel coordinate of the current point in the left eye image, and w is the horizontal pixel coordinate of the current point in the left eye image. It is the target parallax hypothesis. It represents the total number of channels in each channel group.

[0108] Understandably, in binocular images, depth information is typically represented as disparity (i.e., the difference in horizontal position of the same point in the left and right images). `wd` is the theoretical horizontal coordinate of that point in the right-eye image, assuming a disparity of `d` pixels. The method for determining the grouped cost volume can be understood as follows: to determine if the depth of a feature point (h, w) is `d`, the features of that feature point in the left eye are extracted and compared with the features of the theoretically corresponding feature point (h, wd) in the right eye (calculating the inner product). By traversing all possible `d` values, the feature with the highest correlation score is found, thus determining the optimal disparity of that feature point and consequently its depth. This operation only calculates correlation between features in the same group, avoiding the construction of a large cost volume across all channels, reducing memory usage and computational load.

[0109] In this embodiment, the traditional binocular matching method calculates the inner product of all feature channel pairs in the left and right feature maps, generating a four-dimensional cost volume. The dimensions of this cost volume are: feature channel pair × disparity search range × feature map height × feature map width. Let the number of feature channels be C, the disparity search range be D, and the feature map spatial size be... ,but: Computational complexity: Requires computation The inner product of feature channel pairs, where each inner product corresponds to a spatial location under the disparity assumption, results in a computational complexity of: .

[0110] Memory usage: Since it needs to store the correlation values ​​(scalars) for each feature channel pair, each disparity, and each location, the memory usage of traditional methods is: .

[0111] This application employs a grouped cost body to divide the channels into G groups, with each group containing a certain number of channels. Furthermore, the inner product of all feature channel pairs is calculated only within the same group (the number of feature channel pairs in each group is...). Then, the results of each group are either concatenated or stored independently. This ultimately generates G independent 3D cost meshes (all of which are of different sizes). Each grid element corresponds to a relevant value within a group.

[0112] Computational complexity: The total number of inner product iterations is Therefore, the computational complexity of this application is: .

[0113] Memory usage: Each group is one There are G scalar grids in total, therefore the total memory usage of this application is: .

[0114] Compared to traditional binocular matching full-channel methods, the computational complexity of this application is reduced to [a fraction of the original value]. Memory usage has been reduced to its original level. .

[0115] Assuming the number of feature channels C=256, the number of groups G=8, the disparity search range D=128, and the feature map space size H×W=80×45 (corresponding to a downsampled 1280x720 image from the original image),... 16, and each element is float32 (occupies 4 bytes): Traditional omnichannel memory: ; The grouped memory in this application: ; The computational complexity is reduced to the original amount. That is, a reduction of 87.5%. Memory usage has been reduced to [a certain percentage] of its original value. .

[0116] In this embodiment of the application, in step S104 above, the data processing unit can also aggregate the group cost body corresponding to each feature point under each channel group and each disparity assumption to obtain the depth distribution and geometric confidence of each feature point.

[0117] For example, the data processing unit can aggregate the grouped cost volumes corresponding to each feature point in the first feature map under each channel group and each disparity assumption to obtain the cost distribution of the feature points. Then, based on the cost distribution, the depth distribution and geometric confidence of the feature points are determined.

[0118] Among them, the cost distribution characterizes the matching quality of feature points under different disparity assumptions, the depth distribution is obtained by normalizing the cost distribution under multiple disparity assumptions, and the geometric confidence is determined according to the sharpness or unimodality of the cost distribution.

[0119] In some embodiments, the data processing unit can obtain the cost distribution of a feature point by traversing each feature point on the first feature map and aggregating the cost volumes of each channel group and each group under each disparity hypothesis for that feature point. Specifically, for each disparity hypothesis, the cost volumes of each group under that disparity hypothesis for that feature point in all channel groups are summed or concatenated to form a comprehensive cost vector. The values ​​of each element of this cost vector in the disparity dimension constitute the cost distribution of that feature point.

[0120] In some embodiments, the data processing unit may further: after obtaining the cost distribution by aggregating and grouping cost volumes, input the cost distribution into a lightweight three-dimensional convolutional neural network for cost aggregation and regularization processing, and fuse cost information within the neighborhood of feature points through convolution operations to smooth noise fluctuations in the cost distribution, so as to make the depth distribution and geometric confidence subsequently parsed from the cost distribution more stable and reliable.

[0121] In one possible implementation, the data processing unit can first sum the grouped cost volumes of feature points under the same disparity assumption in each channel group element-wise to obtain the aggregated cost of the feature point under each disparity assumption. The aggregated costs under all disparity assumptions together constitute the cost distribution of the feature point, and each element in the cost distribution directly reflects the matching quality of the feature point under that disparity assumption. Then, a softmax normalization operation is performed on the cost distribution across all disparity assumption dimensions to convert the original matching quality values ​​into probability values ​​for each disparity assumption, thus obtaining the depth distribution of the feature point. The disparity assumption with the highest probability in the depth distribution is the optimal depth estimate for the feature point. Simultaneously, the morphological characteristics of the cost distribution are analyzed, and the contrast between the maximum value and the other values ​​in the cost distribution is calculated, or the information entropy of the cost distribution is calculated. The magnitude of the contrast or the level of the information entropy are used as the metric for geometric confidence. When the cost distribution exhibits a clear unimodal shape and a prominent peak, the corresponding geometric confidence is higher.

[0122] For example, the data processing unit can group the cost body Input a 3D convolutional neural network for cost aggregation and regularization. This 3D convolutional neural network can output two parallel results: disparity probability distribution (i.e., depth distribution): This is achieved through a softmax operation for each feature point in the image. The regression yields a probability distribution based on the discrete disparity assumption, representing the probability that the point lies in different disparities.

[0123] Geometric confidence Confidence level It does not originate from the target classification score, but rather from the geometric characteristics of the relevant volume itself after decoding self-regularization, which can objectively and directly reflect the reliability of the feature point depth estimation results.

[0124] In this embodiment, a unified cost distribution is formed by aggregating the cost volumes of each channel group and each group under each disparity assumption. Based on the cost distribution, the depth distribution and geometric confidence are determined simultaneously, thus providing a quantitative assessment of the reliability of the depth estimation results. The geometric confidence is directly derived from the morphological features of the cost distribution without relying on external supervision signals or additional computational branches. This reduces computational overhead while objectively reflecting the certainty of the depth estimation for each feature point, thereby providing a reliable basis for the confidence weighting in the subsequent bird's-eye view feature map generation stage.

[0125] In this embodiment of the application, in step S105 above, the data processing unit can also fuse the feature contributions of all feature points according to the depth distribution of all feature points to obtain a bird's-eye view feature map.

[0126] For example, the data processing unit can, for any feature point in the first feature map, determine the contribution of the feature point to the projection of the bird's-eye view space under each disparity assumption based on the depth distribution of the feature point. Then, based on the geometric confidence level corresponding to the feature point, the contribution of the feature point under each disparity assumption is weighted to obtain the bird's-eye view feature contribution. Finally, a bird's-eye view feature map is generated based on the bird's-eye view feature contribution corresponding to each feature point.

[0127] In some embodiments, the data processing unit can extract the data corresponding to each feature point from the depth distribution and geometric confidence level determined in step S104 by traversing each feature point on the first feature map. For each feature point, its depth distribution gives the probability value under each disparity assumption, and the geometric confidence level gives the confidence value corresponding to each disparity assumption level in the depth distribution. The data processing unit can calculate the contribution of the feature point to the bird's-eye view projection based on these two sets of data.

[0128] In some embodiments, the data processing unit may further: when projecting feature points onto the bird's-eye view space, introduce camera intrinsic and extrinsic parameter calibration data, convert the image coordinates and disparity assumption values ​​of the feature points into three-dimensional spatial coordinates, and then map the three-dimensional spatial coordinates onto the grid coordinate system of the bird's-eye view feature map, thereby determining the grid position of the bird's-eye view feature map corresponding to the feature point under a specific disparity assumption.

[0129] In one possible implementation, the data processing unit generates the bird's-eye view feature map as follows: First, a blank bird's-eye view feature map is initialized. The spatial extent of this feature map covers a preset perception area around the vehicle, and the feature dimension of each grid is the same as the dimension of the feature vector in the first feature map. Then, for each feature point on the first feature map, based on the probability value corresponding to each disparity hypothesis in the depth distribution of that feature point, the contribution of that feature point to the projection into the bird's-eye view space under each disparity hypothesis is determined. The contribution is the probability value of that disparity hypothesis in the depth distribution. Next, the contribution under each disparity hypothesis is multiplied by the geometric confidence value corresponding to the disparity hypothesis level to obtain the weighted bird's-eye view feature contribution of that feature point under that disparity hypothesis. Subsequently, based on the camera projection model and the depth values ​​corresponding to each disparity hypothesis, the feature point is projected onto the corresponding grid position in the bird's-eye view feature map, and the feature vector of that feature point is multiplied by the weighted bird's-eye view feature contribution and then accumulated into the feature of that grid. After completing the above operations for all feature points, the final bird's-eye view feature map is obtained.

[0130] For example, such as Figure 2 As shown, the data processing unit first extracts features from the left and right eye images, then constructs grouped cost volumes, which are then aggregated through a 3D convolutional neural network to output depth distribution and geometric confidence scores in parallel. After obtaining the depth distribution and geometric confidence scores, the data processing unit performs a geometric confidence-weighted depth distribution operation to obtain a bird's-eye view feature map. Subsequently, the data processing unit can use the bird's-eye view feature map as input to regress the 3D bounding box parameters of the UAV through a 3D object detection head.

[0131] Understandably, in the process of acquiring the bird's-eye view feature map, for each feature point in the left-eye image, the data processing unit can use the probability value of each disparity hypothesis level in its depth distribution as the initial contribution of that level to the bird's-eye view feature grid. Then, this initial contribution is multiplied element-wise by the geometric confidence level corresponding to that disparity hypothesis level to obtain the final bird's-eye view feature contribution of that feature point under that disparity hypothesis. Finally, the weighted feature contribution is splashed along the camera ray direction to fill the corresponding grid in the bird's-eye view feature map space. This weighting mechanism ensures that feature contributions from high-reliability depth estimation levels are fully preserved in the bird's-eye view feature map, while feature contributions from vague or unreliable depth estimation levels are effectively suppressed.

[0132] In the step of regressing the 3D bounding box parameters of the UAV, the data processing unit can generate a bird's-eye view feature map. Input a BEV-based 3D target detection head (an anchor box center point-based detector). This detector can directly predict the 3D bounding box parameters of targets such as UAVs in the BEV space, including the center point. ,size Heading angle and speed And tracking identifiers. Because the input features themselves have undergone confidence cleaning, the detection results have high positioning accuracy and stability.

[0133] In this embodiment, the projection contribution under each disparity assumption is determined based on the depth distribution of feature points, and the contribution is weighted using geometric confidence. This ensures that feature points with reliable depth estimation dominate the projection during the generation of the bird's-eye view feature map, while the projection contribution of feature points with ambiguous depth estimation is automatically weakened. This feature projection method effectively suppresses spatial noise caused by stereo matching errors, resulting in a cleaner and more reliable bird's-eye view feature map. This provides a high-quality spatial feature foundation for subsequent direct 3D target detection on the bird's-eye view feature map.

[0134] In this embodiment of the application, in step S105 above, the data processing unit can further weight the contribution of the bird's-eye view features according to the quality of the acquired image, so as to obtain a more reliable bird's-eye view feature map that can adapt to different observation conditions.

[0135] For example, the data processing unit can determine the observation quality factor based on the image features of local regions in the first image. Then, based on the geometric confidence level and observation quality factor corresponding to the feature points, the contribution of the feature points under each disparity assumption is weighted to obtain the bird's-eye view feature contribution.

[0136] Among them, the observation quality factor characterizes the observation quality of local regions in the image.

[0137] In this embodiment, the observation quality factor is a scalar value obtained by comprehensively evaluating a local region in the first image based on the image features and observation geometry of that region. It is used to characterize the observation quality of that local region under the current binocular system observation conditions. A higher observation quality factor indicates that the region is more suitable for stereo matching and depth estimation, while a lower observation quality factor indicates that the region is more difficult to match due to reasons such as missing texture, poor observation angle, or excessive parallax.

[0138] In some embodiments, the observation quality factor can be a weighted combination of three components: observation angle score, effective stereo baseline ratio, and region texture richness. The observation angle score reflects the angle between the target region and the optical axis of the stereo camera; a smaller angle indicates stronger geometric constraints for stereo matching, resulting in a higher score. The effective stereo baseline ratio reflects the proportion of the target region's matching disparity in the left and right images relative to the system's maximum effective matching disparity; a moderate disparity indicates the most reliable matching. Region texture richness is obtained by calculating the magnitude and direction variance of the gradient within a local image patch; richer texture indicates more reliable feature matching. These three components collectively characterize the observation conditions of the local region from three dimensions: observation geometry, disparity range, and image content.

[0139] In some embodiments, while determining the depth distribution and geometric confidence of any feature point, the data processing unit can also determine the observation quality factor by performing image analysis on the local region where the feature point is located in the first image. Specifically, for a local image patch centered on the feature point in the first image, the data processing unit calculates the observation angle score of the center position of the image patch relative to the optical axis of the binocular camera, calculates the effective stereo baseline ratio based on the preliminary disparity estimate of the feature point in the first feature map, and performs gradient analysis on the local image patch to calculate the regional texture richness. Then, these three components are weighted and summed according to preset weights to obtain the observation quality factor corresponding to the feature point.

[0140] For example, one way to determine the observation quality factor is as follows: Let the image coordinates of the target feature point be... The observation quality factor corresponding to this feature point is denoted as The observation quality factor is scored based on the observation angle. Effective three-dimensional baseline ratio and regional texture richness The weighted sum of the three components is obtained as follows: In the formula , , For normalized weight coefficients, satisfying From a set of balanced preset values, the following can be selected: , , =0.3. .

[0141] Among them, the observation angle score This can reflect the angle between the drone and the camera's optical axis; the smaller the angle, the stronger the geometric constraint of binocular stereo matching. Assuming the approximate center of the drone is located at... Its observation angle ,in Let the target direction vector be... is the camera's optical axis vector.

[0142] Effective three-dimensional baseline ratio This reflects the ratio of the drone's parallax to the maximum effective parallax in the left and right images; matching is most reliable when the parallax is moderate. Assume the horizontal pixel coordinates of the drone in the left and right views are respectively... Then match disparity , ,in, In some embodiments, the preset maximum effective matching disparity is used. The value can be determined based on the focal length of the binocular camera. baseline and the preset nearest effective detection distance Determined, that is The nearest effective detection distance characterizes the closest distance threshold required for reliable depth estimation in the UAV perception method of this application. This threshold can be preset according to the safety requirements of the vehicle driving scenario, for example, preset to 3 meters or 5 meters. Simultaneously, to ensure the reliability of stereo matching, a preset distance is also used. The value should not exceed one-third of the image's horizontal resolution.

[0143] Region texture richness It can be obtained by calculating the magnitude and direction variance of the gradient within a local image patch. The richer the texture, the more reliable the feature matching.

[0144] In some embodiments, the data processing unit may further: multiply the calculated observation quality factor element-wise with the geometric confidence of the feature point to obtain a comprehensive confidence weight, and then use this comprehensive confidence weight to weight the contribution of the feature point under each disparity hypothesis. Alternatively, the data processing unit may use the observation quality factor and geometric confidence as independent weighting dimensions, first weighting the contribution of each disparity hypothesis level by the geometric confidence, and then modulating the weighted contribution as a whole by the observation quality factor. Both methods can achieve the effect of dual weighting to suppress unreliable feature contributions.

[0145] In one possible implementation, the data processing unit can first calculate the observation quality factor of the local region where each feature point is located in the first image. Then, for each feature point, its contribution under each disparity hypothesis is multiplied and weighted by the geometric confidence value of the corresponding disparity hypothesis level and the observation quality factor to obtain the final bird's-eye view feature contribution of the feature point under that disparity hypothesis. Specifically, the initial contribution of a feature point under each disparity hypothesis is determined by the probability value of that disparity hypothesis in its depth distribution. The weighted bird's-eye view feature contribution is equal to the initial contribution multiplied by the geometric confidence value of the disparity hypothesis level and then multiplied by the observation quality factor corresponding to the feature point. Since the observation quality factor remains the same across all disparity hypothesis levels for the same feature point, it serves as the overall observation condition evaluation value for the feature point to globally modulate the projection contribution, while the geometric confidence value modulates the reliability at different depth levels differently.

[0146] For example, feature points Coordinates in the bird's-eye view feature map The weighted feature contribution at point is: ; in, For discrete disparity assumption values, Represents the overall depth distribution Feature points after softmax normalization Located at depth The depth probability. It is based on the camera model to extract feature points At the parallax level The function that projects downwards onto the coordinate system of the bird's-eye view feature map. It is an indicator function.

[0147] Furthermore, the data processing unit can accumulate the features of all feature points in the left-eye image to the contributing features of the bird's-eye feature map grid, ultimately forming the bird's-eye feature map. The method for determining it is as follows: ; It is understandable that, during the feature enhancement stage, the contribution of each feature point is determined by the depth distribution probability. Geometric confidence level and observation quality factor Jointly weighted. Among them, It can effectively suppress the contribution of features from areas with poor observation geometry (such as side views, low texture, excessive parallax due to close proximity, etc.), thus embedding a self-evaluation of perception ability at the source of generating bird's-eye view feature maps, resulting in more reliable bird's-eye view feature maps that are more adaptable to different observation conditions.

[0148] In this embodiment, by introducing both an observation quality factor and a geometric confidence level to doubly weight the contribution of feature points under each disparity assumption, the contribution of feature points with reliable depth estimation and good observation conditions is fully preserved in the bird's-eye view feature map, while the contribution of feature points with ambiguous depth estimation or poor observation conditions is suppressed. This dual quality assessment mechanism, embedded from the source of bird's-eye view feature map generation, can effectively solve the problem of large depth estimation errors in weak texture regions using traditional methods.

[0149] For example, the data processing unit can customize and annotate a UAV target dataset based on the publicly available dataset (Karlsruhe Institute of Technology and Toyota Technological Institute Raw Dataset, KITTI Raw) jointly created by the Karlsruhe Institute of Technology and Toyota Technological Institute in the USA. This dataset contains 1200 frames (1242x375 resolution), with weak texture areas (road reflections, sky, white walls) accounting for 35% and normal texture areas (trees, buildings) accounting for 65%. Furthermore, the data processing unit can split 200 weak texture images from the UAV target dataset (100 frames for road reflections, 70 frames for the sky, and 30 frames for white walls) to focus on depth error and false alarm rate testing. A comparison is made between the binocular BEV perception model based on classification confidence (baseline architecture: ResNet + full-channel cost volume + classification confidence weighted bird's-eye view features) and the method provided in this application using dual-weighted bird's-eye view features of "geometric confidence + observation quality factor".

[0150] Verification showed that the depth estimation error in weakly textured areas (such as road surface reflections and the sky) was reduced from 2.1m to 0.5m compared to traditional methods, and the false alarm rate of 3D detection was reduced from 12% to 3%. On the KITTI drone dataset, the mean average precision at intersection over union threshold of 0.5 (mAP@0.5) reached 88%, which is 23 percentage points higher than the method that only uses classification confidence.

[0151] Specifically, the average depth error of UAV targets was statistically analyzed on a 200-frame weak-texture test set. Traditional methods (classification confidence only): average error 2.1m (standard deviation 0.8m), with 2.3m for road surface reflections, 1.9m for the sky, and 2.0m for white walls. This application (weighted by geometric confidence and observation quality factor): average error 0.4m (standard deviation 0.2m), with errors in all weak-texture areas reduced to within 0.5m (0.5m for road surface reflections, 0.3m for the sky, and 0.4m for white walls).

[0152] On a 200-frame weak texture-specific test set, the percentage of false alarm targets was calculated: False alarm rate = ,in, The number of false alarm targets, To accurately detect the target number, This represents the number of missed detections. Traditional method: Number of false alarms. =24 (total targets 200), false alarm rate 12% (due to false detections of "ghosting" in weak texture areas). This application: Number of false alarm targets. =6, false alarm rate 3% (geometric confidence filters out 80% of low-quality deep features).

[0153] On a subset of 1200 KITTI UAVs, the average accuracy was calculated at IoU=0.5. Traditional method (classification confidence only): mAP@0.5=65%. This application (geometric confidence + observation quality factor weighting): mAP@0.5=88% (precision 92%, recall 85%), a performance improvement of 23 percentage points.

[0154] In this embodiment of the application, the data processing unit can also share the drone location results detected by the vehicle with surrounding vehicles, so that surrounding vehicles can perceive the drone's location based on the shared information.

[0155] For example, the data processing unit can first detect the drone in the bird's-eye view feature map, obtaining the drone's three-dimensional spatial state information and detection confidence level. Then, based on the target vehicle's sensor state information, it determines information confidence data. Finally, based on the three-dimensional spatial state information, detection confidence level, and information confidence data, a collaborative perception message is generated.

[0156] Among them, detection confidence represents the reliability of the UAV detection results, information credibility data is used for the receiver to evaluate the credibility of the three-dimensional spatial state information, and collaborative perception messages are used to assist the receiver in perceiving the UAV.

[0157] In this embodiment, detection confidence refers to a scalar value output by the detection head when a UAV target is detected in the bird's-eye view feature map. It characterizes the overall reliability of the UAV detection result and integrates information such as the geometric confidence of each feature point within the target area, the observation quality factor, and the classification probability. Information credibility data refers to a set of quantitative indicators generated by the sending vehicle based on its own sensor status and observation conditions. This data is used by the receiving vehicle to evaluate the credibility of the three-dimensional spatial state information it sends. This data is independent of the target detection result itself and reflects the state and observation capabilities of the sending vehicle's own perception system. Cooperative perception messages refer to message packets encapsulated according to a predetermined protocol format and broadcast to surrounding traffic participants via a V2X communication link. These messages simultaneously contain the detected UAV three-dimensional spatial state information, a reliability assessment of the detection result, and a credibility assessment of the sending vehicle's own perception system, used to assist the receiving vehicle in perceiving the UAV target.

[0158] In some embodiments, the collaborative perception message can be encapsulated in the extended basic safety messages (BSM) format. In addition to the message header (such as message ID, timestamp, sending vehicle ID, message type and version, etc.), the message body can define UAV target data elements, including standard fields such as target ID, timestamp, three-dimensional position, three-dimensional size, heading angle, three-dimensional velocity and detection confidence.

[0159] For example, such as Figure 3 As shown, the BSM-formatted cooperative perception message includes a message header and a target drone message. The message header includes the message ID, timestamp, vehicle ID, message type, and protocol version. The target drone message includes speed (including V). X V Y V Z The data includes: dimensions (length, width, and height), location (longitude, latitude, and altitude), target UAV ID, heading angle, and detection confidence level.

[0160] In some embodiments, the information reliability data may further include the following key fields: observability value, sensor self-test status code, and vehicle pose covariance trace, among other evaluation dimensions. The observability value is the geometric quality factor or geometric confidence level of the observation of the target calculated by the transmitting vehicle based on its own binocular images, reflecting the reliability of the observation information provided by the vehicle. The sensor self-test status code is the sensor self-test status code of the transmitting vehicle's binocular camera, reflecting health information such as the cleanliness of the optical lens, the validity of calibration parameters, and internal fault conditions. The vehicle pose covariance trace is the trace of the position estimation covariance matrix output by the transmitting vehicle's own positioning system, serving as a measure of its own positioning uncertainty.

[0161] For example, one way to determine observability value is as follows: ; Understandably, the higher the observability value, the more reliable and unique the vehicle's observation information of the UAV target is, and the higher the weight should be given to it when the receiver performs multi-source fusion.

[0162] In some embodiments, after generating the bird's-eye view feature map, the data processing unit can also detect the UAV and obtain its 3D bounding box parameters, including the 3D coordinates of the center point, length, width, height, and heading angle, by running a 3D target detection head based on the BEV space on the bird's-eye view feature map. It can also output the target's velocity vector and tracking identifier. The detection head also synchronously outputs a detection confidence score for each detected target. This score is calculated by combining the geometric confidence mean, the observation quality factor mean, and the category confidence score output by the classification branch within the target area. Simultaneously, the data processing unit can obtain the current sensor self-test status code of the binocular camera from the vehicle's sensor health monitoring module, obtain the position estimation covariance matrix fused from the global navigation satellite system (GNSS) / inertial measurement unit (IMU) from the vehicle's positioning system, and combine this with the calculated target area observation quality factor to construct information confidence data.

[0163] In some embodiments, the data processing unit may further: before generating the cooperative sensing message, perform timestamp alignment and tracking ID association on the detected UAV targets, ensuring that the same UAV target maintains a consistent identifier across consecutive frames, and include the target's motion trajectory information in the cooperative sensing message so that the receiving end can better understand the target's motion state and historical trajectory. Furthermore, the data processing unit may dynamically adjust the broadcast frequency and message content detail of the cooperative sensing message based on the current communication channel load, prioritizing the transmission of target information with high detection confidence and high information credibility when the channel is congested.

[0164] In one possible implementation, the data processing unit can first generate information credibility data, which includes observability value data blocks. The observability value is defined as the minimum of the observation geometric quality factor and geometric confidence of the target in the BEV space within the target area. Then, the UAV's 3D spatial state information, detection confidence, and information credibility data are encoded and encapsulated according to a predefined message protocol to generate a collaborative perception message. This message is then broadcast to the surrounding area via the vehicle's V2X communication module at a preset frequency. Upon receiving the collaborative perception message, the receiving vehicle can parse the target state information and simultaneously obtain the sending vehicle's self-assessment of the information's credibility. This allows for informed weighting of information from different sources in subsequent collaborative fusion stages.

[0165] For example, such as Figure 4 As shown, the interaction between different traffic participants is as follows: The drone reflects a signal; the vehicle's perception module performs drone perception (approximately 80ms) to generate local cooperative perception data; subsequently, the vehicle's communication module broadcasts the cooperative perception message via the V2X channel. Other vehicle systems receive the cooperative perception message via the V2X channel, perform a fusion decision (approximately 20ms), and then output a warning or control command. Through this process, the end-to-end latency T is less than 150ms throughout the entire interaction.

[0166] Understandably, the vehicle's communication module can also receive collaborative perception messages broadcast by other vehicles' systems via the V2X channel. The vehicle's perception module performs fusion decisions based on the received collaborative perception messages and local perception data, and outputs warnings or control commands.

[0167] In this embodiment of the application, by generating a collaborative perception message that simultaneously contains three-dimensional spatial state information, detection confidence, and information credibility data, the receiving end can not only obtain the position and motion state of the UAV target, but also obtain a quantitative assessment of the reliability of the information sent by the sending end, thereby providing a measurable fusion basis for multi-vehicle collaborative perception.

[0168] In this embodiment, the data processing unit can also receive collaborative perception messages shared by other vehicles to enhance the environmental situation of the bird's-eye view feature map generated by the vehicle.

[0169] For example, the data processing unit may first receive collaborative perception messages from other traffic participants. Then, based on the information credibility data in the collaborative perception messages, the credibility of the information from other traffic participants is determined. Finally, based on the information credibility, the three-dimensional spatial state information from multiple other traffic participants is fused to obtain the fused UAV state information.

[0170] In some embodiments, the data processing unit can receive cooperative perception messages broadcast by other traffic participants in the vicinity via the vehicle's V2X communication module from a wireless channel.

[0171] In some embodiments, the data processing unit may further perform motion compensation before fusing the three-dimensional spatial state information from multiple other traffic participants based on information credibility. Specifically, the data processing unit determines the communication delay duration based on the timestamp carried in the cooperative sensing message and the current time of the target vehicle, and then performs motion compensation on the three-dimensional spatial state information in the cooperative sensing message based on the communication delay duration, predicting the target state at the time of transmission to the current time, thereby obtaining time-aligned three-dimensional spatial state information, thus eliminating the adverse effects of spatiotemporal misalignment caused by communication delay on the fusion accuracy.

[0172] For example, one way to determine motion compensation is as follows: assuming other traffic participants are... The target drone status is included in the collaborative sensing messages sent in real time. (i.e., three-dimensional spatial state information) and covariance The target vehicle is This message is received at all times. Due to communication delay, the vehicle's data processing unit can use a Kalman filter prediction step to predict the target UAV's state to the current moment. The predicted state is... The predicted covariance is Where F is the state transition matrix, Q is the process noise covariance, and the state vector is... This includes the target UAV's three-dimensional position and three-dimensional velocity. After motion compensation, the target state in the cooperative sensing messages from different transmission times is aligned to the target vehicle's current state. .

[0173] In some embodiments, the data processing unit may further: determine the fusion weights corresponding to each other traffic participant based on information credibility, and perform weighted fusion of the current three-dimensional spatial state information of each other traffic participant according to each fusion weight to obtain the fused UAV state information. The higher the information credibility of a traffic participant, the greater its corresponding fusion weight, and the more prominent its contribution to the fusion result, thereby ensuring that the fusion result makes more use of the information provided by vehicles with good observation conditions, healthy sensors, and accurate self-positioning.

[0174] For example, for observations of the same UAV target from N different sources, the dynamic fusion weights corresponding to source i... One way to determine this is as follows: ; in, It is derived from the target overall confidence level of the i report. It is the observed geometric quality factor from the i-report. The source of normalization is the uncertainty in vehicle pose. It is the trace of covariance from the i report. This is a preset positioning uncertainty threshold (this threshold can be determined based on the trace of the typical covariance matrix of the vehicle's positioning system under normal operating conditions. For example, it can be...). The default value is twice the typical trace value. Alternatively, it can be set according to the tolerance of the cooperative sensing system for the position error of the participating vehicles. For example, when the standard deviation of the position error of the participating vehicles is required to be no more than 0.5 meters, it can be set to... The default value is 1.0. When the trace of the covariance matrix reported by a vehicle from a certain source... When this threshold is exceeded, its normalized pose uncertainty increases. A value of 1 indicates poor positioning accuracy. It is a quantitative value of the sensor's health status (e.g., normal = 1.0, slight degradation = 0.7, abnormal = 0.3). For adjustable non-negative weighting coefficients, satisfying This is used to balance the contributions of different evaluation dimensions. Among a set of balanced preset values, one can take... .

[0175] Furthermore, the data processing unit can perform weighted fusion of multiple three-dimensional spatial state information of the same UAV target based on dynamic fusion weights, resulting in the fused target position observation value. One way to determine this is: ; in, This represents the state observation value given by the i-th source (which can be the perception result of the vehicle's own sensors or the perception result of surrounding vehicles received by V2X) for the same UAV target, specifically the position of the UAV target (usually a three-dimensional coordinate vector).

[0176] In some embodiments, the data processing unit may also use the standard Kalman filter update formula, based on the target location observations. By fusing the state estimates and covariance of the vehicle itself and other vehicles, a precise and stable global list of UAV target trajectories is obtained. This mechanism ensures that information from vehicles with better observation geometry, healthier sensors, and more accurate self-localization gains greater influence in global fusion, thereby improving the accuracy and robustness of collaborative perception.

[0177] In some embodiments, the data processing unit may further: after generating the bird's-eye view feature map, for a target drone that disappears from the perception field of view, use its motion trajectory sequence before disappearing to predict the possible location of the target drone in the blind zone, and generate a virtual predicted target with an existence probability in the blind zone.

[0178] In some embodiments, the data processing unit may further: before fusing multi-source information, uniformly transform the state information of all targets to the same BEV coordinate system centered on the target vehicle, and then use a data association algorithm in this unified space to associate and match observations from different sources to determine which observations point to the same UAV target. For multiple observations that are successfully associated, the aforementioned weighted fusion method is used for state fusion. For new targets that fail to be associated, they are added as new tracking targets to the vehicle's target tracking list.

[0179] In one possible implementation, the data processing unit performs the following complete multi-source information fusion process: First, it receives and parses collaborative perception messages from multiple other traffic participants, extracting the UAV's 3D spatial state information, detection confidence, and information credibility data from each message. Simultaneously, it predicts virtual UAV targets in the vehicle's blind spots from the bird's-eye view feature map. Next, based on the difference between the timestamp of each message and the current time, it performs a Kalman filter prediction step on the target state in each message to complete motion compensation. Then, it uniformly transforms the UAV target state detected by the vehicle and the compensated target states of other vehicles into a BEV coordinate system centered on the vehicle, and uses the Hungarian algorithm for data association. For successfully associated target observations from multiple sources, it calculates the dynamic fusion weights of each source based on the information credibility data, calculates the fused target position according to the weighted fusion formula, and then uses a standard Kalman filter to update the fusion state estimate and covariance to obtain the fused UAV state information.

[0180] For example, such as Figure 5As shown, after generating the bird's-eye view feature map and detecting 3D targets, the data processing unit can simultaneously obtain a list of drones perceived by the vehicle and a list of virtual drones in the blind spot (including the probability of drone presence). Simultaneously, the data processing unit can receive cooperative perception messages sent by other vehicles through the V2X communication interface and parse the 3D spatial state information of the drones from them, then use Kalman filtering for motion compensation and temporal alignment. Subsequently, the data processing unit can unify the drone list, the list of virtual drones in the blind spot, and the received information from other vehicles into the BEV space, and then perform data association on each drone state information (using the Hungarian algorithm). Then, it determines whether the data association is successful. If the association is successful, the data processing unit will perform state fusion of drone information from different data sources (using Kalman update). If the association fails, the data processing unit will introduce the drone as a new target into the tracking list. Finally, the data processing unit will obtain a globally fused drone tracking list.

[0181] In this embodiment of the application, the information credibility of each traffic participant is determined by the information credibility data in the collaborative perception message, and the multi-source three-dimensional spatial state information is fused based on the credibility, so that the information provided by the participants with high observation quality and high positioning accuracy occupies a greater weight in the fusion result, thereby effectively improving the overall reliability of the fused UAV state information.

[0182] For example, compared with existing solutions that simply average or rely solely on target detection confidence, the information credibility assessment mechanism of this application can distinguish the quality of different information sources from the source. By adopting the motion compensation and information credibility weighted fusion mechanism of this application, the fusion latency is reduced from 100ms to 20ms, the multi-source positioning error is reduced from 1.2m to 0.3m, the fusion accuracy can be maintained above 85% under the condition of a 20% communication packet loss rate, and the effective perception range of the vehicle group can be extended from 200m for a single vehicle to 500m.

[0183] In this embodiment, the data processing unit can also predict virtual drone information in the blind spot of the binocular camera image acquisition based on the bird's-eye feature map, so as to actively avoid blind spot risks.

[0184] For example, the data processing unit can first acquire the motion trajectory sequence of the target drone after it disappears from the perception field of view. Then, based on the motion trajectory sequence, it performs trajectory extrapolation and intent classification to determine the predicted trajectory of the target drone and its probability distribution for at least one behavioral intent. Subsequently, based on the predicted trajectory and probability distribution, it determines the probability of the target drone's presence within the current perception blind zone of the target vehicle. If the probability of presence is higher than a preset threshold, a virtual predicted target is generated within the perception blind zone.

[0185] The state of the virtual predicted target is determined based on the predicted trajectory.

[0186] In this embodiment, the target drone refers to a drone target that has been detected and tracked by the vehicle in the current frame or several previous frames, but disappears from the field of view of the vehicle's binocular camera in the current frame. The reasons for its disappearance may include entering the sensor's vertical blind zone, being obstructed by a vehicle in front or other obstacles, or moving out of the edge of the camera's field of view. The motion trajectory sequence refers to the time-series data consisting of the three-dimensional spatial position and corresponding timestamps recorded in consecutive frames within a certain period before the target drone disappears. Each element in the sequence contains the target's three-dimensional coordinates and velocity information at a certain moment, used to characterize the target drone's motion state and trend before disappearing.

[0187] In some embodiments, the duration of the motion trajectory sequence can be preset according to the target's motion characteristics and scenario requirements, such as taking all tracking records within the last five seconds before the target drone disappears. In addition to the target drone's three-dimensional position at each moment, the motion trajectory sequence can also include motion state information such as the target drone's velocity, acceleration, and heading angle at each moment to more completely describe the target drone's motion pattern. When the duration of stable tracking of the target drone before disappearing is less than the preset minimum sequence length, the sequence length can be appropriately shortened or the target drone can be marked as a drone with low prediction reliability.

[0188] In some embodiments, the data processing unit may maintain a rolling spatiotemporal scene memory unit, continuously recording a sequence of bird's-eye view feature maps from the past several frames. When it is detected that a tracked target drone is not successfully associated with any detection result in the current frame, and its last appearance location is close to the edge of the sensor's field of view or an occluded area, the data processing unit determines that the target drone has disappeared from the perception field of view. Subsequently, it extracts all historical state records of the target drone within a preset time period before its disappearance from the spatiotemporal scene memory unit, forming a motion trajectory sequence of the target drone.

[0189] In some embodiments, the data processing unit can also: calculate and update the vehicle's perception area and various types of blind spots in real time within the BEV space based on the precise calibration parameters of the vehicle's binocular cameras and the current three-dimensional environment model surrounding the vehicle. Blind spot types may include blind spots directly above the vehicle roof, side and rear blind spots, and shadow blind spots created by obstructions from vehicles in front. When calculating the probability of a target drone entering each blind spot, the data processing unit cross-compares the predicted trajectory with the spatial range of each blind spot to determine the likelihood of the target entering each blind spot.

[0190] In one possible implementation, the data processing unit inputs the motion trajectory sequence of the target UAV into a pre-trained recurrent neural network, which can specifically adopt a long short-term memory (LSTM) network structure. This recurrent neural network performs two tasks in parallel: first, it extrapolates the motion trajectory sequence to generate a predicted trajectory of the target UAV within a preset future time step; second, it classifies the target UAV's behavioral intentions at the moment of disappearance, outputting the probability distribution of the target's behavior across several preset behavioral intentions. These preset behavioral intentions can include categories such as approaching the vehicle, moving away from the vehicle, hovering, and crossing roads. Furthermore, the data processing unit spatially matches the predicted trajectory with blind spot maps in the current BEV space, and calculates the probability of the target UAV's presence in each blind spot based on the intention classification probability. When the presence probability exceeds a preset threshold, the data processing unit generates a virtual predicted target at the corresponding blind spot location. The three-dimensional position and velocity of this virtual predicted target are determined by the values ​​of the predicted trajectory at the current moment, and the presence probability value is attached as its credibility attribute.

[0191] For example, one way to determine the probability of the existence of a target drone is as follows: Let the sequence of the target drone's motion trajectory in the period before it disappears be... The sequence is then input into a pre-trained Long Short-Term Memory (LSTM) network, which will output predicted trajectories in parallel. The intention classification probability distribution is used. For a given intention, the corresponding predicted trajectory gives the position of the target UAV in the next K time steps. Combining the intention classification probability distribution and the spatial range of each blind zone, the probability of existence (PoE) of the target UAV currently existing in a certain blind zone can be calculated as the weighted sum of the probabilities of the predicted trajectories entering that blind zone under each intention. After the virtual predicted target is generated, its state information is given by the value of the extrapolated trajectory at the current time, and its existence probability (PoE) is attached as an independent attribute to the data structure of the virtual target.

[0192] It is understandable that, such as Figure 6As shown, one generation process for virtual predicted targets is as follows: First, the data processing unit continuously records the historical sequence of generated bird's-eye view feature maps. Simultaneously, the data processing unit calculates and updates the blind spot map of the vehicle's BEV space in real time based on sensor parameters. Next, the data processing unit senses drones in the bird's-eye view feature maps. For target drones marked as disappeared in the historical bird's-eye view feature maps, it extracts their trajectory and motion state before disappearing and inputs them into a pre-trained Long Short-Term Memory (LSTM) network. This LSM network can output the most likely future trajectory of the target drone and its behavioral intent probability in parallel. Then, the data processing unit combines the future trajectory, behavioral intent probability, and BEV space blind spot map to determine the probability that the target drone currently exists in each blind spot, and generates virtual predicted targets with existence probabilities at the corresponding blind spot locations. Finally, the data processing unit overlays and fuses these virtual predicted targets onto the bird's-eye view feature maps as special layers.

[0193] In this embodiment of the application, by acquiring the motion trajectory sequence of the target UAV after it disappears from the perception field of view, performing trajectory extrapolation and intent classification, and then combining the blind spot map to calculate the existence probability and generate a virtual predicted target with the existence probability, the vehicle can still maintain a continuous estimate of the future state and position of the target even after losing direct observation of the target.

[0194] In summary, such as Figure 7 As shown, the detailed process of the UAV perception method provided in this application is as follows: After the binocular camera data is input, the data processing unit performs left-eye feature extraction and right-eye feature extraction respectively, and the outputs of the two are used to construct a grouped cost body. Subsequently, the depth distribution and geometric confidence are calculated based on the grouped cost body to generate a bird's-eye view feature map. Based on the bird's-eye view feature map, the detection head can regress the UAV to obtain the UAV perception result of the vehicle. Then, the vehicle state and the UAV perception result of the vehicle can be input into a message encapsulator, which processes the message and broadcasts a collaborative perception message. At the same time, the data processing unit also receives collaborative perception messages from other vehicles and parses the UAV information. Then, the data processing unit performs asynchronous temporal alignment and fusion of the UAV perception result of the vehicle and the UAV information parsed from the collaborative perception messages of other vehicles to obtain an enhanced bird's-eye view feature map. Finally, the data processing unit obtains the historical bird's-eye view feature map sequence based on the enhanced bird's-eye view feature map and performs trajectory extrapolation and intent prediction to generate a virtual UAV to assist in vehicle decision-making.

[0195] The UAV perception method provided in this application embodiment can be applied in the following scenarios: In the autonomous forward collision avoidance scenario, when the vehicle is driving on urban or suburban roads, the data processing unit can obtain the drone's position, speed, size, and heading angle through binocular BEV perception and blind spot proactive prediction, and predict its intention and the probability of its presence in the blind spot. Using enhanced bird's-eye view feature maps, it calculates the collision time and risk probability, triggering a tiered warning system from alert to warning to emergency braking, while simultaneously optimizing the driving path to avoid the drone's trajectory. When the vehicle is driving in areas with weak cooperative coverage, such as tunnel blind spots, if the drone disappears from the side or front, the data processing unit can predict its entry onto the roof through trajectory extrapolation, generating a high-probability virtual predicted target, triggering an overhead warning and automatically slowing down.

[0196] In the scenario of reporting to the flight control zone, after a vehicle detects a drone, the data processing unit can encapsulate its three-dimensional spatial state information, intent classification results, and observation quality factors into a collaborative perception message and report it to the Civil Aviation Administration or the drone supervision platform. The control zone can use this information to identify drones that illegally intrude into the no-fly zone or approach the road, and take control measures such as notifying the operator to adjust the route or driving them away to avoid collisions from the source.

[0197] In scenarios where low-altitude activities coexist, when the trajectory of a logistics delivery drone intersects with that of a vehicle, the data processing unit triggers a warning indicating drone operation ahead. The vehicle then slows down and avoids direct interception, ensuring a balance between logistics efficiency and driving safety. When a drone is hovering above the roadside, the data processing unit issues a warning about overhead filming equipment, assisting the vehicle in passing smoothly and preventing airflow from interfering with vehicle stability.

[0198] In sensor and communication protection scenarios, when electromagnetic signals emitted by a drone are detected interfering with the vehicle's millimeter-wave radar, the vehicle switches to a binocular vision-dominated perception mode. It utilizes grouped correlation volume to reduce computational complexity and, in conjunction with dual-weighted BEV features, maintains reliable perception of the surrounding environment. When airflow generated by the drone's flight affects vehicle stability, the vehicle can adjust its speed based on the drone's position, moving away from the area directly beneath the drone to avoid rollover.

[0199] In extreme environment scenarios, when the vehicle is driving in fog, the geometric confidence decoding of the autocorrelation volume of binocular vision effectively filters ghosting false detections in weak texture areas and, combined with trajectory extrapolation, predicts the location of drones in blind spots. Under strong backlight conditions, the observation quality factor weights the BEV features to suppress false detections in the sky area, ensuring the vehicle's continuous and stable operation under harsh conditions.

[0200] In summary, the UAV perception method provided in this application reduces processing latency by dividing the channels of the first and second feature maps into multiple channel groups and calculating feature similarity based on multiple disparity assumptions. This avoids the high computational cost of dense disparity search across all pixels in the entire image, as is common in traditional methods. Furthermore, each feature point simultaneously obtains its depth distribution and geometric confidence based on the feature similarity under each channel group and each disparity assumption, quantifying the reliability of depth estimation. This allows for the effective identification and elimination of unreliable depth information in complex scenes, preventing perception errors caused by mismatches. Finally, a bird's-eye view feature map is generated based on the depth distribution and geometric confidence of each feature point, providing an efficient and robust feature foundation for determining the UAV's three-dimensional spatial position. This enables real-time perception and proactive prediction and early warning of low-altitude UAVs with low latency.

[0201] In an exemplary embodiment, such as Figure 8 As shown, the UAV sensing device includes: a feature extraction module 801, a feature grouping module 802, a feature matching module 803, a depth perception module 804, and a spatial mapping module 805. The feature extraction module 801 is used to acquire a first feature map and a second feature map based on a stereo image containing the UAV. The stereo image includes a first image and a second image. The first feature map comes from the first image, and the second feature map comes from the second image. A feature map includes multiple channels, and each feature point on the feature map corresponds to a feature vector composed of the channels. The feature grouping module 802 is used to divide the multiple channels contained in the first feature map and the multiple channels contained in the second feature map into multiple channel groups according to the correspondence between the channels. The feature matching module 803 is used to, for each channel group, determine the features on the first feature map within that channel group based on multiple disparity assumptions. The feature similarity between the feature point and the corresponding feature point on the second feature map, and the disparity hypothesis are used to indicate the pixel offset of the corresponding feature point on the second feature map relative to each feature point on the first feature map; the depth perception module 804 is used to determine the depth distribution and geometric confidence of each feature point in the first feature map based on the feature similarity of the feature point under each channel group and each disparity hypothesis, and the geometric confidence characterizes the reliability of the depth estimation result of the corresponding feature point; the spatial mapping module 805 is used to generate a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map, and the bird's-eye view feature map is used to determine the three-dimensional spatial position of the UAV.

[0202] In this embodiment of the application, the feature matching module 803 is specifically used to: in the target channel group, for the target disparity hypothesis, determine the reference feature point in the second feature map corresponding to the target feature point in the first feature map, wherein the target channel group is any channel group among multiple channel groups, and the target disparity hypothesis is any disparity hypothesis among multiple disparity hypotheses; and based on the target feature point and the reference feature point, determine the grouping cost body of the target feature point under the target disparity hypothesis as the feature similarity between the two.

[0203] In this embodiment, the depth perception module 804 is specifically used to: for each feature point in the first feature map, aggregate the grouped cost bodies corresponding to the feature point under each channel group and each disparity hypothesis to obtain the cost distribution of the feature point, the cost distribution characterizing the matching quality of the feature point under different disparity hypotheses; based on the cost distribution, determine the depth distribution and geometric confidence of the feature point, the depth distribution being obtained by normalizing the cost distribution under multiple disparity hypotheses, and the geometric confidence being determined according to the sharpness or unimodality of the cost distribution.

[0204] In this embodiment of the application, the spatial mapping module 805 is specifically used for: for any feature point in the first feature map, determining the contribution of the feature point to the bird's-eye view projection under each disparity assumption based on the depth distribution of the feature point; weighting the contribution of the feature point under each disparity assumption based on the geometric confidence corresponding to the feature point to obtain the bird's-eye view feature contribution; and generating a bird's-eye view feature map based on the bird's-eye view feature contribution corresponding to each feature point.

[0205] In this embodiment of the application, the depth perception module 804 is further configured to: determine the observation quality factor based on the image features of a local region in the first image, wherein the observation quality factor characterizes the observation quality of the local region; and weight the contribution of the feature points under each disparity assumption based on the geometric confidence and the observation quality factor corresponding to the feature points to obtain the bird's-eye view feature contribution.

[0206] In this embodiment, the device further includes a communication module, configured to: detect a drone in a bird's-eye view feature map; acquire the drone's three-dimensional spatial state information and detection confidence level, wherein the detection confidence level characterizes the reliability of the drone detection results; determine information credibility data based on the sensor state information of the target vehicle, wherein the information credibility data is used by the receiving end to evaluate the credibility of the three-dimensional spatial state information; and generate a collaborative perception message based on the three-dimensional spatial state information, detection confidence level, and information credibility data, wherein the collaborative perception message is used to assist the receiving end in perceiving the drone.

[0207] In this embodiment of the application, the communication module is further configured to: receive collaborative perception messages from other traffic participants; determine the information credibility of other traffic participants based on the information credibility data in the collaborative perception messages; and fuse the three-dimensional spatial state information from multiple other traffic participants based on the information credibility to obtain the fused UAV state information.

[0208] In this embodiment of the application, the device further includes a cooperative sensing module, used to: determine the communication delay duration based on the timestamp carried in the cooperative sensing message and the current time of the target vehicle; and perform motion compensation on the three-dimensional spatial state information in the cooperative sensing message based on the communication delay duration to obtain the three-dimensional spatial state information at the current time.

[0209] In this embodiment of the application, the collaborative perception module is specifically used to: determine the fusion weights corresponding to each other traffic participant based on the information credibility; and perform weighted fusion of the current three-dimensional spatial state information of each other traffic participant according to each fusion weight to obtain the fused UAV state information.

[0210] In this embodiment of the application, the device further includes a blind spot prediction module, configured to: acquire a motion trajectory sequence of the target drone when the target drone disappears from the perception field of view; perform trajectory extrapolation and intent classification based on the motion trajectory sequence to determine the predicted trajectory of the target drone and the probability distribution of at least one behavioral intent; determine the probability of the presence of the target drone within the current perception blind spot of the target vehicle according to the predicted trajectory and the probability distribution; and generate a virtual predicted target within the perception blind spot if the probability of presence is higher than a preset threshold, wherein the state of the virtual predicted target is determined according to the predicted trajectory.

[0211] In an exemplary embodiment, this application also provides an electronic device, which may be the drone sensing device in the above method embodiments. For example... Figure 9 As shown, the UAV sensing device may include a processor 901 and a memory 902. The memory 902 stores instructions executable by the processor 901. When the processor 901 is configured to execute the instructions, it causes an electronic device, network device, or manager to perform the system functions described in the foregoing method embodiments.

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for sensing unmanned aerial vehicles (UAVs), characterized in that, The method includes: Based on the binocular images containing the drone, a first feature map and a second feature map are obtained. The binocular images include a first image and a second image. The first feature map is obtained from the first image, and the second feature map is obtained from the second image. A feature map includes multiple channels, and each feature point on the feature map corresponds to a feature vector composed of the channels. The multiple channels of the first feature map and the multiple channels of the second feature map are divided into multiple channel groups according to the correspondence between the channels; For each channel group, based on multiple disparity assumptions, the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map within the channel group is determined. The disparity assumptions are used to indicate the pixel offset of the corresponding feature point on the second feature map relative to each feature point on the first feature map. For each feature point in the first feature map, the depth distribution and geometric confidence of the feature point are determined based on the feature similarity corresponding to the feature point under each channel group and each disparity assumption. The geometric confidence characterizes the reliability of the depth estimation result of the corresponding feature point. Based on the depth distribution and geometric confidence of each feature point in the first feature map, a bird's-eye view feature map is generated, which is used to determine the three-dimensional spatial position of the UAV.

2. The UAV sensing method according to claim 1, characterized in that, For each channel group, based on multiple preset disparity assumptions, the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map within that channel group is determined, including: In the target channel group, for the target disparity hypothesis, the reference feature point in the second feature map corresponding to the target feature point in the first feature map is determined. The target channel group is any one of the multiple channel groups, and the target disparity hypothesis is any one of the multiple disparity hypotheses. Based on the target feature points and the reference feature points, the group cost body of the target feature points under the target disparity assumption is determined as the feature similarity between the two.

3. The UAV sensing method according to claim 2, characterized in that, For each feature point in the first feature map, determining the depth distribution and geometric confidence of the feature point based on the feature similarity corresponding to the feature point under each channel group and each disparity assumption includes: For each feature point in the first feature map, the grouped cost bodies corresponding to the feature point under each of the channel groups and each of the disparity assumptions are aggregated to obtain the cost distribution of the feature point. The cost distribution characterizes the matching quality of the feature point under different disparity assumptions. Based on the cost distribution, the depth distribution and geometric confidence of the feature points are determined. The depth distribution is obtained by normalizing the cost distribution over the multiple disparity hypotheses, and the geometric confidence is determined based on the sharpness or unimodality of the cost distribution.

4. The UAV sensing method according to claim 1, characterized in that, The step of generating a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map includes: For any feature point in the first feature map, the contribution of the feature point to the projection of the bird's-eye view space under each parallax assumption is determined based on the depth distribution of the feature point. Based on the geometric confidence level corresponding to the feature point, the contribution of the feature point under each disparity assumption is weighted to obtain the bird's-eye view feature contribution. The bird's-eye view feature map is generated based on the bird's-eye view feature contribution corresponding to each feature point.

5. The UAV sensing method according to claim 4, characterized in that, The method further includes: Based on the image features of a local region in the first image, an observation quality factor is determined, wherein the observation quality factor characterizes the observation quality of the local region; The step of weighting the contribution of the feature point under each disparity assumption based on the geometric confidence level corresponding to the feature point to obtain the bird's-eye view feature contribution includes: Based on the geometric confidence level and the observation quality factor corresponding to the feature point, the contribution of the feature point under each of the disparity assumptions is weighted to obtain the bird's-eye view feature contribution.

6. The UAV sensing method according to any one of claims 1-5, characterized in that, The method further includes: The drone is detected in the bird's-eye view feature map, and the three-dimensional spatial state information and detection confidence of the drone are obtained. The detection confidence represents the reliability of the drone detection results. Based on the sensor status information of the target vehicle, information credibility data is determined, and the information credibility data is used by the receiving end to evaluate the credibility of the three-dimensional spatial status information; Based on the three-dimensional spatial state information, the detection confidence level, and the information credibility data, a collaborative perception message is generated, which is used to assist the receiving end in perceiving the UAV.

7. The UAV sensing method according to claim 6, characterized in that, The method further includes: Receive the cooperative sensing messages from other traffic participants; The information credibility of the other traffic participants is determined based on the information credibility data in the collaborative perception message; Based on the credibility of the information, the three-dimensional spatial state information from multiple other traffic participants is fused to obtain the fused UAV state information.

8. The UAV sensing method according to claim 7, characterized in that, Before fusing the three-dimensional spatial state information from multiple other traffic participants based on the information credibility, the method further includes: The communication delay duration is determined based on the timestamp carried in the collaborative sensing message and the current time of the target vehicle; Based on the communication delay duration, motion compensation is performed on the three-dimensional spatial state information in the collaborative sensing message to obtain the three-dimensional spatial state information at the current moment.

9. The UAV sensing method according to claim 8, characterized in that, Based on the information credibility, the three-dimensional spatial state information from multiple other traffic participants is fused to obtain the fused UAV state information, including: Based on the credibility of the information, the fusion weights corresponding to each of the other traffic participants are determined; Based on the fusion weights, the three-dimensional spatial state information of the other traffic participants at the current moment is weighted and fused to obtain the fused UAV state information.

10. The UAV perception method according to any one of claims 1 to 5, characterized in that, The method further includes: When the target drone disappears from the perception field of view, the motion trajectory sequence of the target drone is acquired; Based on the motion trajectory sequence, trajectory extrapolation and intent classification are performed to determine the predicted trajectory of the target UAV and its probability distribution on at least one behavioral intent; Based on the predicted trajectory and the probability distribution, the probability of the existence of the target drone is determined within the current perception blind spot of the target vehicle. When the probability of existence is higher than a preset threshold, a virtual prediction target is generated within the perception blind zone, and the state of the virtual prediction target is determined according to the prediction trajectory.

11. A drone sensing device, characterized in that, The device includes: The feature extraction module is used to obtain a first feature map and a second feature map based on a stereo image containing a drone. The stereo image includes a first image and a second image. The first feature map is obtained from the first image, and the second feature map is obtained from the second image. A feature map includes multiple channels, and each feature point on the feature map corresponds to a feature vector composed of the channels. The feature grouping module is used to divide the multiple channels contained in the first feature map and the multiple channels contained in the second feature map into multiple channel groups according to the correspondence of the channels; The feature matching module is used to determine, for each channel group, the feature similarity between each feature point on the first feature map and the corresponding feature point on the second feature map within the channel group based on multiple disparity assumptions. The disparity assumptions are used to indicate the pixel offset of the corresponding feature point on the second feature map relative to each feature point on the first feature map. The depth perception module is used to determine the depth distribution and geometric confidence of each feature point in the first feature map based on the feature similarity of the feature point under each channel group and each disparity assumption. The geometric confidence represents the reliability of the depth estimation result of the corresponding feature point. The spatial mapping module is used to generate a bird's-eye view feature map based on the depth distribution and geometric confidence of each feature point in the first feature map. The bird's-eye view feature map is used to determine the three-dimensional spatial position of the UAV.

12. A vehicle, characterized in that, The vehicles include: A binocular camera is used to acquire binocular image data of the drone surrounding the vehicle; A data processing unit is communicatively connected to the binocular camera, and the data processing unit is configured to perform the UAV perception method as described in any one of claims 1-10.

Images