An aerial view angle blind area target identification and positioning method and device

Abstract

The application discloses a kind of identification and positioning method and device of aerial visual angle blind area target, belong to computer vision and remote sensing field.The method includes: obtaining the multisensor data of laser radar, camera, IMU and GNSS;Laser radar point cloud is converted to local horizontal coordinate system, and the region with elevation mutation feature is identified as blind area detection candidate area;The specific type and pixel range of aerial visual angle blind area target are identified;The point cloud data corresponding to target mask is extracted, combined with the pose and coordinate information of unmanned aerial vehicle, the three-dimensional coordinates of target under geocentric coordinate system are solved, and the center position is calculated to realize accurate positioning.The technical route of the application is coarse positioning by laser radar point cloud detection candidate area, accurate identification by image semantic segmentation, coordinate mapping solution, solves the problem that existing technology cannot be efficiently identified and positioned, and the accuracy is low, realizes the type identification and high-precision positioning of aerial visual angle blind area target.

Description

Technical Field

[0001] This invention relates to a method and apparatus for identifying and locating targets in aerial blind spots, belonging to the fields of computer vision and remote sensing. Background Technology

[0002] Unmanned aerial vehicle (UAV) remote sensing technology has been widely applied in civilian fields such as surveying and inspection, as well as military fields such as reconnaissance and positioning. However, when observed from above, targets such as culverts, gas stations, and overpasses cannot be directly observed due to the obstruction of their overhead structures, forming "aerial blind spots." A key characteristic of these blind spots is that they are spaces obscured by structures, meaning they have physical roof structures that prevent direct observation from the air, while they are often partially or fully visible from a ground perspective. In some application scenarios, the internal information of these blind spots can be supplemented through ground-view observation. Therefore, it is necessary to develop corresponding identification and positioning methods specifically for these blind spot targets with overhead obstruction and accessible ground access.

[0003] Currently, with the development of deep learning technology, aerial target recognition methods based on visible light cameras or LiDAR are relatively mature and can effectively identify common targets such as people, vehicles, and buildings. However, when applying these methods to aerial blind zone recognition tasks, if a purely vision-based recognition algorithm is used, it usually relies on a large amount of labeled data for training. That is, by learning the salient features of samples, the model's generalization ability in various scenarios is improved. However, for aerial blind zone targets, the feature differences between samples are usually large, and it is difficult to obtain a large number of labeled samples. Directly applying this method often results in high false positive and false negative rates, leading to reduced recognition accuracy and reliability. In addition, purely vision-based recognition algorithms cannot obtain the accurate location of the target, resulting in insufficient accuracy when applied to target localization. On the other hand, if a LiDAR-based recognition and localization method is used, although it can identify and locate various targets through accurate three-dimensional spatial geometric information, the straight-line distance from the LiDAR-equipped UAV platform to the aerial blind zone target is usually far. As the point cloud density gradually thins, the recognition ability of this type of algorithm decreases significantly, thus affecting the accurate identification and localization of aerial blind zone targets. Therefore, there is an urgent need for a solution that combines the advantages of the two methods mentioned above for aerial blind spot target identification and localization, in order to solve the problems of accuracy and reliability in current aerial blind spot target identification and localization. Summary of the Invention

[0004] To address the problems of low efficiency and accuracy in identifying and locating aerial targets in blind spots using current vision-based or lidar-based identification and positioning algorithms, this invention provides a method and apparatus for identifying and locating aerial targets in blind spots. This method combines the advantages of both visual recognition algorithms and lidar recognition, achieving high-efficiency identification and positioning with higher accuracy and reliability. The technical solution is as follows:

[0005] A method for identifying and locating targets in aerial blind spots, comprising the following steps:

[0006] S1: Acquire multi-sensor data mounted on the UAV platform, including: 3D point cloud data P collected by LiDAR. Lidar RGB two-dimensional image data I acquired by the camera RGB The attitude information of the UAV collected by the inertial measurement unit, and the three-dimensional coordinates g of the UAV collected by the global navigation satellite system. UAV ;

[0007] S2: Transfer the three-dimensional point cloud data P Lidar Perform coordinate system transformation to generate 3D point cloud data P of the scene in the local horizontal coordinate system. Local ;

[0008] S3: Based on the three-dimensional point cloud data P of the scene Local The system identifies regions with abrupt elevation changes and designates them as candidate detection regions P for aerial blind spots. Det ;

[0009] S4: In the RGB two-dimensional image data I RGB In the middle, combined with the detection candidate region P Det A semantic segmentation algorithm is used to identify the specific type and pixel range of targets in the aerial blind spot, and the target mask M of the target in the blind spot is obtained.

[0010] S5: Extract the three-dimensional point cloud data P Lidar In the two-dimensional image data I RGB Projection P on the plane Image-Lidar For all points covered by the target mask M, obtain the position information p of each point in the local horizontal coordinate system. i The relative position information of this point with respect to the UAV platform is Δp. i , Δp i ≈p i ;

[0011] S6: Based on the relative position information Δp i With the three-dimensional coordinates g of the UAV UAVThe three-dimensional coordinates g of the target in the aerial blind zone in the geocentric coordinate system are obtained by using coordinate calculation. i And the three-dimensional coordinates g of targets belonging to the same aerial blind zone. i The set G = {g1, g2, ... g} is composed of {g1, g2, ... g}. n}, where n is a positive integer, g i g1 to g n Calculate the center position of the three-dimensional coordinate set G, taking any one of the coordinates. This completes the localization of targets in the aerial blind spot.

[0012] Furthermore, step S1 also includes performing joint intrinsic and extrinsic parameter calibration on the lidar and the camera, establishing a projection relationship as follows: P Image-Lidar =K[R cam |t cam ]·P Lidar Where K is the camera intrinsic parameter matrix, R cam t cam Let be the extrinsic rotation matrix and translation vector of the laser radar to the camera.

[0013] Furthermore, in S2, the three-dimensional point cloud data P acquired by the lidar... Lidar The transformation from the lidar coordinate system to the local horizontal coordinate system specifically includes: calculating the rotation matrix R based on the UAV attitude information.

[0014]

[0015] Wherein, α, β and γ respectively represent the pitch angle, roll angle and yaw angle of the UAV attitude information;

[0016] The scene 3D point cloud data P in the local horizontal coordinate system Local P Local =R·P Lidar .

[0017] Furthermore, in S3, the identification of the elevation change feature region is as follows: selecting the scene's three-dimensional point cloud data P Local neighboring point p i and p j And convert it to spherical coordinates p i =(r i ,θ i ,φ i ) T and p j =(r j ,θ j ,φ j ) T Where i,j∈[1,m] and i≠j, m is PLocal The total number of points in the array, where i, j, and m are all positive integers. Adjacent points p i and p j The elevation difference Δh is calculated using the formula Δh=|r i cosθ i -r j cosθ j | The elevation difference Δh is greater than the preset threshold h th A continuous set of points is labeled as a detection candidate region P. Det Then the adjacent point p i and p j Marked as detection candidate points, the continuous point set refers to the three-dimensional point cloud data P of the scene. Local A set of continuously distributed candidate detection points.

[0018] Furthermore, in S4, in the RGB two-dimensional image data I RGB In the middle, combined with the detection candidate region P Det The semantic segmentation algorithm is used to identify the specific type and pixel range of targets in the aerial view blind zone, and the target mask M is obtained. The steps include:

[0019] S41: The detection candidate region P Det The data is first transformed back to the lidar coordinate system, and then projected onto the two-dimensional image data. RGB Alignment yields the candidate scatter plot I. Det ;

[0020] S42: The semantic segmentation algorithm adopts an encoder and decoder structure, the encoder and decoder structure includes I RGB Encoder, I Det Encoder and decoder, the two-dimensional image data I RGB After I RGB The encoder forms multi-scale, multi-level intermediate feature maps; the detection candidate scatter plot I Det After the I Det The encoder generates an attention map of the same scale as the intermediate feature layer; the decoder employs multiple upsampling modules to upsample the corresponding layer's attention map generated by the I... RGB The intermediate feature map generated by the encoder and the I Det The attention maps generated by the encoder are concatenated, and finally, after multiple convolutions, they are based on the detection candidate scatter map I. Det Finally, the target mask M is generated.

[0021] Furthermore, in S41, the detection candidate region P DetThe data is first transformed back to the lidar coordinate system, and then projected onto the two-dimensional image data. RGB Alignment yields the candidate scatter plot I. Det Includes: the detection candidate region P Det The inverse matrix R of the rotation matrix R T Transform back to the lidar coordinate system, and then use the projection relationship, P Image-Det =K[R cam |t cam ]·P Det , with two-dimensional image data I RGB Alignment yields the candidate scatter plot I. Det .

[0022] Furthermore, in step S6, the coordinate calculation includes the following steps:

[0023] S61: The three-dimensional coordinates of the UAV g UAV The coordinates in the WGS-84 coordinate system are, g UAV = (B UAV ,L UAV H UAV ) T The coordinates converted to the ECEF coordinate system are G. UAV =(X UAV ,Y UAV Z UAV ) T ;

[0024] S62: Calculate the coordinates of the target in the blind spot of the aerial view in the ECEF coordinate system as G. i =(X i ,Y i Z i ) T The calculation formula is: in It is the rotation matrix for transforming from the ENU coordinate system to the ECEF coordinate system;

[0025] S63: Position the target in the blind spot of the aerial view in the ECEF coordinate system. i Convert to coordinates g in the WGS-84 coordinate system i = (B i ,L i H i ) T .

[0026] An apparatus for identifying and locating targets in the blind spot of an aerial view, comprising:

[0027] The data acquisition module is used to acquire 3D point cloud data from the lidar mounted on the UAV platform, 2D image data from the camera, IMU attitude information, and GNSS 3D coordinates.

[0028] The candidate region detection module is used to identify candidate regions for aerial view blind zone detection with elevation change characteristics based on the three-dimensional point cloud data.

[0029] The semantic segmentation module is used to identify the specific type and pixel range of targets in the aerial view blind spot based on the detection candidate region in the two-dimensional image of the camera;

[0030] The coordinate calculation module is used to associate the point cloud data within the pixel range and, in conjunction with IMU attitude information and GNSS three-dimensional coordinates, calculate the three-dimensional coordinates of the aerial view blind zone in the geocentric coordinate system.

[0031] The beneficial effects of this invention are:

[0032] (1) This invention achieves type identification and high-precision positioning of targets in the aerial view blind zone by using the technical route of coarse localization of candidate areas by lidar point cloud detection, accurate identification by image semantic segmentation, and coordinate mapping calculation.

[0033] (2) It fully utilizes the high-precision three-dimensional geometric perception capability of lidar, the rich texture information of camera, and the high-precision positioning and attitude determination capability of IMU and GNSS, and completes the high-precision positioning of targets in the blind zone of the aerial view through multi-source information fusion.

[0034] (3) In view of the characteristics of the aerial blind spot target of the structure, the detection candidate area in the scene is identified by the three-dimensional point cloud data of the lidar, and an attention mechanism is designed for the candidate area to improve the accuracy of the semantic segmentation algorithm.

[0035] (4) The present invention provides a method and device for identifying and locating targets in the airborne blind spot, which combines the advantages of visual recognition algorithm and lidar recognition to achieve accurate and efficient identification and location of targets in the airborne blind spot, providing important information for ground-based observation missions. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a three-dimensional structural diagram of the aerial blind spot target identification and positioning device carried by the unmanned aerial vehicle platform according to an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating a method for identifying and locating targets in the aerial blind spot according to Embodiment 1 of the present invention.

[0039] Figure 3 This is a network schematic diagram of the encoder-decoder structure provided in Embodiment 1 of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of the aerial blind spot target identification and positioning device provided in Embodiment 2 of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, this invention provides a three-dimensional structural diagram of an aerial target identification and localization device mounted on an unmanned aerial vehicle (UAV) platform. In the diagram, 1 is a lidar, 2 is a visible light camera, 3 is a Global Navigation Satellite System (GNSS) module, and 4 is the UAV fuselage, which houses an inertial measurement unit (IMU). The lidar 1 and camera 2 are rigidly connected via a structural component, which in turn is rigidly connected to the UAV fuselage 4. This structural component can employ a custom-designed bracket structure, which can be made of lightweight, high-strength materials such as aluminum alloy or carbon fiber. Since the relative poses of the lidar, camera, IMU, and GNSS module mounted on the UAV platform are constant, the intrinsic and extrinsic parameters of all sensors will not change after calibration.

[0043] Example 1

[0044] like Figure 2 As shown, this embodiment provides a method for identifying and locating targets in the aerial blind spot, including the following steps:

[0045] S1: Acquire data from multiple sensors mounted on the UAV platform, including: 3D point cloud data P collected by LiDAR. Lidar RGB two-dimensional image data I acquired by the camera RGB The attitude information of the UAV collected by the inertial measurement unit (IMU) and the three-dimensional coordinates of the UAV collected by the Global Navigation Satellite System (GNSS) UAV .

[0046] Specifically, 3D point cloud data P LidarThe coordinate system in question is called the lidar coordinate system. This is a right-handed Cartesian coordinate system with the lidar's optical center as the origin. Its XOY plane is parallel to the lidar base plane, the positive X-axis is aligned with the UAV's nose, and the positive Z-axis is perpendicular to the base and pointing upwards. (Two-dimensional image data I) RGB The dimensions are H×W×3, where H and W are the height and width of the image, and 3 is the number of color channels, including red, green, and blue channels. The UAV attitude information acquired by the Inertial Measurement Unit (IMU) includes pitch, roll, and yaw angles, denoted as α, β, and γ, respectively, and belongs to the Front-Right-Ground (FRD) coordinate system. The three-dimensional coordinates of the UAV acquired by the Global Navigation Satellite System (GNSS) are represented as g. UAV = (B UAV ,L UAV H UAV ) T Its coordinate system is the geocentric geodetic coordinate system. This embodiment uses the WGS-84 geocentric geodetic coordinate system, where B is represented by the coordinate system. UAV L UAV and H UAV These represent latitude, longitude, and elevation, respectively.

[0047] Step S1 also includes joint intrinsic and extrinsic parameter calibration of the lidar and camera to realize the 3D point cloud data P Lidar With two-dimensional image data I RGB Alignment, i.e., establishing the projection relationship as P Image-Lidar =K[R cam |t cam ]·P Lidar Where K is the camera intrinsic parameter matrix, R cam t cam Let be the extrinsic rotation matrix and translation vector of the laser radar to the camera.

[0048] S2: Calculate the rotation matrix R based on the UAV attitude information, and then use the 3D point cloud data P collected by the lidar. Lidar The scene's 3D point cloud P is obtained by transforming the lidar coordinate system to the local horizontal coordinate system. Local =R·P Lidar .

[0049] In this embodiment, the local horizontal coordinate system is the East-North-Sky (ENU) coordinate system. The rotation matrix R is the rotation matrix from the lidar coordinate system to the local horizontal coordinate system. Its value is determined by the sine and cosine of α, β, and γ of the UAV attitude information collected by the inertial measurement unit (IMU) in step S1. The specific calculation formula is as follows:

[0050]

[0051] Specifically, when the top of the UAV is vertically upward and the nose is facing due east, the lidar coordinate system coincides with the local horizontal coordinate system. It is worth noting that the lidar coordinate system is essentially equivalent to the front-left-up (FLU) coordinate system. Therefore, when transforming the attitude angles α, β, and γ acquired by the inertial measurement unit (IMU) in the FRD coordinate system to the lidar coordinate system, the positive directions of the pitch and yaw angles are reversed, while the positive direction of the roll angle remains unchanged. Therefore, the formula for calculating the rotation matrix R actually includes the transformation from the FRD coordinate system to the FLU coordinate system.

[0052] After obtaining the rotation matrix R, the 3D point cloud data P of the scene in the local horizontal coordinate system can be obtained through matrix transformation. Local =R·P Lidar Because the lidar is rigidly connected to the UAV platform, the 3D point cloud data P is affected by the UAV's attitude. Lidar The Z-axis data in the image cannot directly reflect the target elevation information. However, after the transformation in step S2, the scene's 3D point cloud data P... Local The Z-axis data represents the relative elevation of each point.

[0053] S3: Based on the 3D point cloud P of the scene in the local horizontal coordinate system Local The system identifies regions in the scene with abrupt elevation changes and designates these regions as candidate regions P for aerial blind spot detection. Det The identification of the elevation change features is achieved through analysis of the scene's 3D point cloud data P. Local In the process, determine whether the elevation difference Δh between adjacent point pairs exceeds a preset threshold h. th This is achieved by satisfying Δh>h th A continuous set of points is labeled as a detection candidate region P. Det .

[0054] Specifically, in this embodiment, the identification of regions with elevation change characteristics is mainly accomplished through the following method: acquiring scene 3D point cloud data P Local any point p in i And convert it to spherical coordinates p i =(r i ,θ i ,φ i ) T , where θ i φ is the zenith angle. i For an inscribed angle, r i The radial distance is selected. The 3D point cloud data P is used. Local In and p i The adjacent point is p j p j =(r j ,θj ,φ j ) T Where i,j∈[1,m] and i≠j, m is P Local The total number of points in the array, where i, j, and m are all positive integers. Define p. i neighboring point p j P for scene 3D point cloud data Local Chinese Envoy | θ i -θ j |or|φ i -φ j Find several points that represent the minimum or second minimum value. A threshold value h for the elevation difference is manually set. th , for p i Its any adjacent point p j Calculate the elevation difference Δh = |r i cosθ i -r j cosθ j |, if Δh>h th Then mark p i With p j To detect candidate points, a clustering algorithm (such as K-means, DBSCAN, etc.) is applied to divide all candidate points into several clusters. If the number of points in a single cluster exceeds a manually set threshold C, then... th If the points belonging to this cluster are considered to be in the scene's 3D point cloud data P Local The detection candidate region P is defined as the set of detection candidate points that satisfy the continuous distribution condition. Det .

[0055] S4: In two-dimensional image data I RGB In the middle, combined with the detection candidate region P Det A semantic segmentation algorithm is used to identify the specific type and pixel range of targets in the aerial blind spot, and the target mask M is obtained.

[0056] Specifically, in this embodiment, the candidate region P is detected. Det Through the inverse matrix R of the rotation matrix R T Transform back to the lidar coordinate system, and then use the projection relationship P. Image-Det =K[R cam |t cam ]·P Det , with two-dimensional image data I RGB Alignment yields the candidate scatter plot I. Det It was subsequently applied to semantic segmentation algorithms.

[0057] like Figure 3As shown, the semantic segmentation algorithm described in this embodiment can employ an encoder-decoder structure (such as the U-Net network or its variants), comprising three parts: an I... RGB Encoder (RGB image encoder), one I Det An encoder (candidate scatter plot encoder) and a decoder.

[0058] The I RGB The encoder is used to extract input two-dimensional image data I RGB Multi-level visual features are extracted to generate multi-scale intermediate feature maps at various levels, with the following structure:

[0059] Two-dimensional image data I RGB First, the feature map passes through two convolutional layers to obtain the first-level intermediate feature map. Then, this first-level intermediate feature map is fed into a series of downsampling modules, each consisting of a 2x downsampling layer and two convolutional layers. With each downsampling module, the spatial resolution of the feature map decreases while the number of channels increases, thus forming multi-scale intermediate feature maps at various levels for subsequent feature fusion and decoding.

[0060] The I Det The encoder is used to process the input aerial view blind zone detection candidate scatter plot I Det And generate attention maps at various levels across multiple scales, with the following structure:

[0061] Candidate Scatter Plot I Det First, an attention encoder is used to detect the candidate scatter plot I using a 15×15 average pooling layer. Det Transform into a smoothed image; then change the number of channels through a 1×1 convolution, and combine with the two-dimensional image data I. RGB Scale alignment was performed; finally, the Sigmoid activation function was used to compress the value range, resulting in the first-level attention map. Next, it was compared with I... RGB The encoder is similar, detecting candidate scatter plot I Det It also goes through a series of downsampling modules with the same structure, and the output of each module is a value similar to I. RGB The encoder corresponds to an attention map with a consistent scale to the intermediate feature maps of the layer.

[0062] The core of this embodiment lies in utilizing I Det The encoder generates an attention map to modulate I RGB The encoder extracts visual features, enabling the network to focus on target areas in blind spots. Specifically, in I RGB Each layer of the encoder outputs an intermediate feature map that is then compared with I. DetThe encoder performs element-wise multiplication on the attention maps of the corresponding layers (as shown by "□" in the figure). This operation generates modulation features, in which the feature weights of the target region in the blind zone are increased, while the feature weights of the background region are suppressed.

[0063] The decoder progressively upsamples the high-level semantic features extracted by the encoder to restore them to the original image resolution, and finally generates a semantic segmentation target mask M, the structure of which is as follows:

[0064] The decoder consists of a series of upsampling modules, each consisting of two convolutional layers and a 2x upsampling layer, functioning as the opposite of the downsampling modules. To fuse low-level spatial features with high-level semantic features, each layer of the decoder uses skip connections. Specifically, at each upsampling stage, the output feature map of the current layer of the decoder will be compared with I... RGB The encoder's corresponding layer's "raw intermediate features" and the features processed by I Det The "modulation features" of the encoder's attention modulation are concatenated (as shown in the figure). As shown in the diagram, this process simultaneously preserves the spatial details in the original intermediate features and the attention information in the modulated features. After upsampling and skip connections in the final layer, the feature map is restored to its original size through three convolutional layers, generating the target mask M.

[0065] Through the above structure, the semantic segmentation algorithm described in this embodiment can effectively fuse I Det With prior information about blind spots, the pixel range and semantic category of targets in the aerial view blind spot are generated under the guidance of the attention mechanism. These targets include, but are not limited to, culverts, canopies (such as gas station roofs), overpasses, and other targets with structural shading.

[0066] S5: Extract the three-dimensional point cloud data P Lidar In the two-dimensional image data I RGB Projection P on the plane Image-Lidar For all points covered by the target mask M, obtain the position information p of each point in the local horizontal coordinate system. i =(x i ,y i ,z i ) T And calculate the relative position information Δp with the UAV platform. i ≈p i In this embodiment, the relative position information Δp i It is the relative position of 3D point cloud data to the GNSS positioning center carried by the UAV platform, and its calculation formula is: in This refers to the relative position from the lidar optical center to the Global Navigation Satellite System (GNSS) positioning center. Because UAVs typically fly tens to hundreds of meters away from their targets, while the distance between lidar and GNSS is usually only a few centimeters, therefore... Δp i It can be considered to be approximately equal to p i .

[0067] S6: Based on relative position Δp i With the three-dimensional coordinates g of the UAV UAV The three-dimensional coordinates g of the target in the blind spot of the aerial view in the geocentric coordinate system are calculated. i And the three-dimensional coordinates of targets belonging to the same aerial blind spot are grouped into a set G = {g1, g2, ... g...} n}, calculate the center position g of the three-dimensional coordinate set, thereby completing the location of the aerial blind spot.

[0068] In this embodiment, the specific process of coordinate calculation is as follows: First, the coordinates g of the UAV in the WGS-84 coordinate system are calculated. UAV = (B UAV ,L UAV H UAV ) T Convert to coordinates G in the Earth-centered Earth-fixed (ECEF) coordinate system. UAV =(X UAV ,Y UAV Z UAV ) T Next, calculate the position G of the target in the blind spot of the aerial view in the ECEF coordinate system. i =(X i ,Y i Z i ) T The calculation formula is: in It is the rotation matrix for transforming from the ENU coordinate system to the ECEF coordinate system; finally, G... i Convert to coordinates g in the WGS-84 coordinate system i = (B i ,L i H i ) T .

[0069] The embodiment described herein The calculation method can take G = {g1, g2, ... g} n The arithmetic mean of all coordinates.

[0070] Example 2

[0071] like Figure 4As shown, this embodiment provides a device for identifying and locating targets in the aerial blind spot, including the following modules:

[0072] The data acquisition module is used to acquire 3D point cloud data from the lidar mounted on the UAV platform, 2D image data from the camera, IMU attitude information, and GNSS 3D coordinates.

[0073] The candidate region detection module is used to identify candidate regions for aerial view blind zone detection with elevation change characteristics based on the three-dimensional point cloud data.

[0074] The semantic segmentation module is used to identify the specific type and pixel range of targets in the aerial view blind spot based on the detection candidate region in the two-dimensional image of the camera;

[0075] The coordinate calculation module is used to associate the point cloud data within the pixel range and, in conjunction with IMU attitude information and GNSS three-dimensional coordinates, calculate the three-dimensional coordinates of the aerial view blind zone in the geocentric coordinate system.

[0076] In summary, the methods and apparatus provided in Embodiments 1 and 2, through steps such as acquiring sensor data, identifying candidate detection areas for aerial blind zone targets, semantic segmentation of blind zone targets, and calculation of blind zone target coordinates, integrate data from multiple sensors, identify the specific type of aerial blind zone targets, and accurately locate their position in the geocentric coordinate system.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying and locating targets in aerial blind spots, characterized in that, It includes: S1: Acquire multi-sensor data mounted on the UAV platform, including: 3D point cloud data collected by LiDAR. RGB two-dimensional image data captured by the camera The attitude information of the UAV collected by the inertial measurement unit and the three-dimensional coordinates of the UAV collected by the global navigation satellite system. ; S2: Transfer the three-dimensional point cloud data Perform coordinate system transformation to generate 3D point cloud data of the scene in the local horizontal coordinate system. ; S3: Based on the 3D point cloud data of the scene The system identifies regions with abrupt elevation changes and designates them as candidate areas for detection in aerial blind spots. ; S4: In the RGB two-dimensional image data In, combined with the detection candidate region Semantic segmentation algorithms are used to identify the specific type and pixel range of targets in the aerial blind spot, and the target mask of the targets in the blind spot is obtained. ; S5: Extract the three-dimensional point cloud data In the two-dimensional image data Projection on a plane All targets masked in the middle The covered points are used to obtain the position information of each point in the local horizontal coordinate system. The relative position information of this point with respect to the drone platform is as follows: , ; S6: Based on the relative position information With the three-dimensional coordinates of the UAV The three-dimensional coordinates of the target in the aerial blind spot in the geocentric coordinate system were obtained by using coordinate calculation. and the three-dimensional coordinates of targets belonging to the same aerial blind spot. Form a set where n is a positive integer. for to Calculate the center position of the three-dimensional coordinate set G, taking any one of the coordinates. This allows for the localization of targets in the aerial blind spot.

2. The method for identifying and locating targets in aerial blind spots according to claim 1, characterized in that, S1 further includes: performing joint intrinsic and extrinsic parameter calibration on the lidar and the camera, and establishing a projection relationship as follows: ,in For the camera intrinsic parameter matrix, , Let be the extrinsic rotation matrix and translation vector of the laser radar to the camera.

3. The method for identifying and locating targets in aerial blind spots according to claim 1, characterized in that, In step S2, the three-dimensional point cloud data acquired by the lidar The transformation from the lidar coordinate system to the local horizontal coordinate system specifically includes: calculating the rotation matrix based on the UAV attitude information. , ， in, , and The pitch angle, roll angle, and yaw angle respectively represent the attitude information of the UAV; the scene 3D point cloud data in the local horizontal coordinate system , .

4. The method for identifying and locating targets in aerial blind spots according to claim 1, characterized in that, In step S3, the identification of the elevation change feature region is achieved by selecting the scene's three-dimensional point cloud data. adjacent points and and convert it to spherical coordinates. and ,in and , yes The total number of points in the middle. All are positive integers; adjacent points and elevation difference Its calculation formula is Elevation difference Greater than the preset threshold A continuous set of points is marked as a detection candidate region. Then adjacent points and Marked as detection candidate points, the continuous point set refers to the three-dimensional point cloud data of the scene. A set of continuously distributed candidate detection points.

5. The method for identifying and locating targets in aerial blind spots according to claim 1, characterized in that, In step S4, the RGB two-dimensional image data In, combined with the detection candidate region Semantic segmentation algorithms are used to identify the specific type and pixel range of targets in the aerial blind spot, and target masks are obtained. It includes the following steps: S41: The detection candidate region The data is first transformed back to the lidar coordinate system, and then projected onto the two-dimensional image data. Alignment yields a scatter plot of candidate points. ; S42: The semantic segmentation algorithm adopts an encoder and decoder structure, the encoder and decoder structure includes... Encoder Encoder and decoder, the two-dimensional image data go through The encoder generates multi-scale, multi-level intermediate feature maps; the detection candidate scatter plot After the above The encoder generates an attention map of the same scale as the intermediate feature layer; the decoder employs multiple upsampling modules to upsample the corresponding layers of the attention map generated by the encoder. The intermediate feature map generated by the encoder and the ... The attention maps generated by the encoder are concatenated, and finally, after multiple convolutions, the results are based on the detection candidate scatter plot. Finally, the target mask is generated. .

6. The method for identifying and locating targets in aerial blind spots according to claim 5, characterized in that, In S41, the detection candidate region The data is first transformed back to the lidar coordinate system, and then projected onto the two-dimensional image data. Alignment yields a scatter plot of candidate points. Includes: the detection candidate region After rotation matrix inverse matrix Transform back to the lidar coordinate system, and then utilize the projection relationship. , with two-dimensional image data Alignment yields a scatter plot of candidate points. .

7. The method for identifying and locating targets in aerial blind spots according to claim 1, characterized in that, In step S6, the coordinate calculation includes the following steps: S61: The three-dimensional coordinates of the UAV The coordinates in the WGS-84 coordinate system are, Converted to coordinates in the ECEF coordinate system, ; S62: Calculate the coordinates of the target in the blind spot of the aerial view in the ECEF coordinate system. The calculation formula is: ,in It is the rotation matrix for transforming from the ENU coordinate system to the ECEF coordinate system; S63: Restore the coordinates of targets in the blind spot of the aerial view to the ECEF coordinate system. Convert to coordinates in the WGS-84 coordinate system .

8. An apparatus for identifying and locating targets in aerial blind spots, as described in any one of claims 1-7, characterized in that... It includes: The data acquisition module is used to acquire 3D point cloud data from the lidar mounted on the UAV platform, 2D image data from the camera, IMU attitude information, and GNSS 3D coordinates. The candidate region detection module is used to identify candidate regions for aerial view blind zone detection with elevation change characteristics based on the three-dimensional point cloud data. The semantic segmentation module is used to identify the specific type and pixel range of targets in the aerial view blind spot based on the detection candidate region in the two-dimensional image of the camera; The coordinate calculation module is used to associate the point cloud data within the pixel range and, in conjunction with IMU attitude information and GNSS three-dimensional coordinates, calculate the three-dimensional coordinates of the aerial view blind zone in the geocentric coordinate system.

Images