Low-altitude situation three-dimensional reconstruction method and device based on multiple view angles of unmanned aerial vehicle

Abstract

The invention provides a low-altitude situation three-dimensional reconstruction method and device based on multiple view angles of an unmanned aerial vehicle, and the method comprises the steps: carrying out the geographic space classification of the multi-view angle video data of the unmanned aerial vehicle, and obtaining a geographic space importance degree mark; generating a low-altitude situation acquisition scheme for the sub-regions according to the sub-region importance scores to obtain the low-altitude situation acquisition scheme; performing unmanned aerial vehicle low-altitude situation data re-acquisition according to the low-altitude situation acquisition scheme to obtain a multi-view low-altitude situation enhanced video; and sampling the multi-view low-altitude situation enhanced video data and the unmanned aerial vehicle multi-view video data according to different sampling rates according to the sub-region importance score, and performing three-dimensional reconstruction according to the sampled image. And high-precision low-altitude situation reconstruction is performed in a targeted manner.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and 3D reconstruction technology, and specifically relates to a method and device for low-altitude situational 3D reconstruction based on multiple perspectives of UAVs. Background Technology

[0002] With the development of map display technology, the number and complexity of 3D scene construction are increasing. Traditional 3D model construction requires manual creation under expert guidance, which is costly and time-consuming. Using 3D reconstruction technology to build low-altitude situational 3D models based on UAV video can help users reconstruct these models more quickly and accurately quantify the low-altitude environment, such as terrain and buildings. Extracting low-altitude situational point clouds from multi-view UAV video to generate a 3D model of the low-altitude situation is an effective method.

[0003] Current visible light-based 3D reconstruction methods acquire multi-angle images of the target area to complete the 3D reconstruction. However, current 3D reconstruction methods suffer from the following problems: 1) Low-altitude situational awareness distinguishes between key and non-key areas, making it impossible to differentiate the importance of different sub-regions; 2) Current methods use the same acquisition scheme, failing to tailor situational awareness acquisition based on the importance of different sub-regions. Therefore, current 3D reconstruction methods cannot improve the results for key areas. Based on this, there is an urgent need for a 3D reconstruction method capable of enhancing the reconstruction of key areas. Summary of the Invention

[0004] This invention provides a method and apparatus for 3D reconstruction of low-altitude situational awareness based on multiple perspectives from unmanned aerial vehicles (UAVs), addressing the problem of targeted, high-precision low-altitude situational awareness reconstruction for key sub-regions. The generated model is loaded into a map engine, improving the efficiency and accuracy of map scene construction.

[0005] The first aspect of this invention provides a method for three-dimensional reconstruction of low-altitude situation based on multiple views from an unmanned aerial vehicle (UAV), comprising: S1. Perform geospatial classification on the multi-view video data of the UAV to obtain geospatial importance labels; the geospatial importance labels include geographic type, category of key target, name of key target, number of key targets, importance score of sub-region, and latitude and longitude of sub-region; S2. Generate a low-altitude situation acquisition scheme for the sub-region based on the importance score of the sub-region. S3. Based on the aforementioned low-altitude situation acquisition scheme, re-acquire low-altitude situation data from the UAV to obtain multi-view low-altitude situation enhanced video. S4. Based on the importance score of the sub-region, the multi-view low-altitude situational awareness enhancement video and UAV multi-view video data are sampled at different sampling rates, and three-dimensional reconstruction is performed based on the sampled images.

[0006] Optionally, geospatial classification can be performed on the multi-view video data from drones, including: Geospatial classification is performed on multi-view video data from drones using a pre-trained geospatial large model.

[0007] Optionally, geospatial classification can be performed on the multi-view video data from drones, including: By using a pre-trained geospatial large model, the number of movable targets in the video is identified, and after normalization, the business traffic of the sub-region is obtained. The importance score of each sub-region is obtained based on the business traffic of each sub-region.

[0008] Optionally, a low-altitude situational awareness acquisition scheme can be generated for a sub-region based on its importance score, including: Using a task knowledge graph, a low-altitude situational awareness acquisition scheme is generated for each sub-region based on its importance score.

[0009] Optionally, a task knowledge graph is used to generate a low-altitude situational awareness acquisition scheme for a sub-region based on its importance score, including: For sub-regions with high business traffic, spatial expansion is performed on the identified building type targets within them using a task knowledge graph to obtain the expanded spatial boundaries of the targets. By calculating waypoints using a task knowledge graph, recommended routes that avoid high-traffic sub-regions are obtained.

[0010] Optional low-altitude situational awareness acquisition schemes include: waypoints, flight numbers, and recommended routes.

[0011] The second aspect of the present invention provides a three-dimensional reconstruction device for low-altitude situation based on multiple perspectives of an unmanned aerial vehicle (UAV), for performing the three-dimensional reconstruction method for low-altitude situation based on multiple perspectives of an UAV as described in any one of the first aspects.

[0012] A third aspect of the present invention provides a computer-readable storage medium, comprising: a memory and a processor; The memory is configured to store executable instructions; The processor is configured to implement, when executing the executable instructions stored in the memory, the method for low-altitude situational awareness reconstruction based on multiple UAV perspectives as described in any one of the first aspects.

[0013] The fourth aspect of the present invention provides a computer program product, the computer program product including instructions, which, when executed by a computer, implement the method for low-altitude situational awareness reconstruction based on multiple perspectives of an unmanned aerial vehicle as described in any one of the first aspects.

[0014] This invention provides a method and apparatus for 3D reconstruction of low-altitude situational awareness based on multi-view UAVs. It can identify important areas in the low-altitude situational awareness, generate a UAV low-altitude situational awareness re-acquisition scheme, and perform 3D reconstruction, improving the accuracy of the reconstructed area. The model generated by multi-angle visible light is loaded into the map engine via file read / write, reducing the cost of 3D model design for the map engine. A targeted situational awareness acquisition scheme is generated, thereby completing the 3D reconstruction of the low-altitude situational awareness. It can identify important sub-regions in the low-altitude situational awareness data, improving the semantic understanding of the low-altitude situational awareness and marking important information. It can generate a UAV low-altitude situational awareness re-acquisition scheme based on the task knowledge graph, enabling focused detection of key areas and improving the quality of detection data for key areas at the source data level, thus improving the reconstruction accuracy of key areas. Using this method, key low-altitude areas can be identified, increasing the detection power of key areas and enabling rapid and low-cost 3D reconstruction of low-altitude areas, especially improving the reconstruction quality of important areas. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the computing environment; Figure 2 This is a schematic diagram of a low-altitude situation 3D reconstruction method based on multiple perspectives of UAVs; Figure 3 This is a schematic diagram of a stereo matching algorithm; Figure 4 As shown, this embodiment provides a three-dimensional reconstruction device for low-altitude situation based on multiple perspectives of UAVs; Figure 5 This is a schematic diagram of the reconstruction effect. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The features and illustrative embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific setups and methods set forth below, but covers any improvements, substitutions, and modifications to structures, methods, and devices without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description to avoid unnecessarily obscuring the invention.

[0019] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing and simplifying the invention, and should not be construed as limiting the invention. Furthermore, the use of ordinal numbers (e.g., "first and second," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly, encompassing both direct connection and indirect connection via an intermediate medium. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0021] It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other, and the various embodiments can be referenced and cited in each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0023] The low-altitude situation 3D reconstruction method based on UAV multi-view perspectives provided in this application can be applied to, for example... Figure 1The application environment is shown. Terminal display devices include laptops and desktops, which can communicate with the server via a network. The database stores UAV videos, images, point clouds, and 3D models. The database can be set up independently or integrated into the server. The terminal can send low-altitude situational awareness data to the server. After receiving the data, the server performs 3D reconstruction and returns the reconstruction results to the terminal for user viewing.

[0024] The server generates multi-view situational images based on multi-view video data collected by UAVs, generates multi-view feature maps from the multi-view situational images, performs camera calibration and point cloud feature extraction on the multi-view feature maps to obtain camera motion trajectory, camera intrinsic parameters, and camera extrinsic parameters; performs point cloud feature extraction on the multi-view feature maps to obtain a 3D point cloud of the low-altitude situation, uses the camera calibration and point cloud features as input, and generates a dense point cloud of the low-altitude situation through a stereo matching algorithm; and performs meshing processing on the dense point cloud of the low-altitude situation to obtain a 3D model of the low-altitude situation.

[0025] In addition, in some embodiments, the low-altitude situation 3D reconstruction method based on UAV multi-view can also be implemented by the terminal alone. For example, the UAV terminal can collect low-altitude situation video while performing 3D reconstruction, or the server can obtain the multi-view low-altitude video to be processed from the data storage system and process it.

[0026] The terminal can be any type of drone, and the server can be a standalone computer, a computer cluster, or a cloud platform.

[0027] This application provides a method for three-dimensional reconstruction of low-altitude situation based on multiple views from an unmanned aerial vehicle (UAV), the method being based on: Based on UAV multi-view video data, the low-altitude situation and category to be reconstructed are obtained; the UAV multi-view video data includes the low-altitude situation stationary on the ground, and multi-angle video data with the low-altitude situation as the target; the video data category is the storage category of the UAV video data.

[0028] The multi-view low-altitude situational video is subjected to intelligent geospatial classification to obtain geospatial importance labels. The intelligent geospatial classification uses a pre-trained geospatial large model to evaluate sub-regions in the video. The geospatial importance labels include geographic type, key target category, key target name, and key target quantity, sub-region importance score, and sub-region latitude and longitude. Higher importance scores indicate high-importance regions, and lower importance scores indicate low-importance regions.

[0029] A low-altitude situational awareness acquisition plan is generated for the aforementioned high-importance areas. This acquisition plan generation uses geographical region stratification results as input and leverages task logic mining through a task knowledge graph to rapidly generate an enhanced acquisition plan.

[0030] The aforementioned low-altitude situation acquisition scheme is used to reacquire low-altitude situation data using UAVs, resulting in multi-view enhanced low-altitude situation video.

[0031] The multi-view low-altitude situation enhancement video is processed frame by frame to obtain multi-view low-altitude situation images. For video segments containing high-importance regions, at least 10 frames are extracted; for video segments not containing high-importance regions, at least 6 frames are extracted.

[0032] The multi-view image data is subjected to feature processing to obtain a multi-view feature map; the feature processing includes motion blur removal and high-frequency enhancement filtering methods; the motion blur removal is used to reduce image quality degradation caused by drone probe shaking; the high-frequency enhancement filtering method is used to amplify the edges and texture details of the image and suppress smooth areas, thereby improving the local contrast and clarity of the multi-view image. Camera calibration is performed on the multi-view feature maps to obtain camera intrinsic parameters, camera extrinsic parameters, and camera 3D motion trajectory; the spatial starting point of the 3D motion trajectory coordinates is the coordinate origin, and the 3D coordinates are within a preset range; the 3D motion trajectory includes the coordinates and angle information of each feature map in 3D space when it is captured. Point cloud extraction is performed on the multi-view feature map to obtain a sparse point cloud of the low-altitude situation to be reconstructed; the sparse point cloud includes all three-dimensional points and attributes of the point cloud; the three-dimensional points are the spatial location points of the three-dimensional structure of the low-altitude situation to be reconstructed, and the attributes are the spatial coordinates, category, color and transparency of each three-dimensional point. Using the multi-view feature images, 3D point clouds, and camera motion trajectories as input, a stereo matching algorithm is used to obtain a dense point cloud of the low-altitude situation to be reconstructed; the dense point cloud includes point cloud and spatial structure attributes; the spatial structure attributes include the internal structure of the low-altitude situation and the external environment structure. The dense point cloud of the low-altitude situation is meshed to obtain a three-dimensional model of the low-altitude situation. Optionally, generating low-altitude situational dense point clouds based on stereo matching algorithms specifically includes: The low-altitude situation map, sparse point cloud, and camera parameters are stereo-corrected to obtain a corrected feature map, a projection matrix, and a remapping lookup table. The corrected feature map pairs are used to map identical detection points in the image to the same y-coordinate, resulting in horizontally aligned corrected feature maps. The projection matrix projects the 3D coordinates of the sparse point cloud onto the corrected image coordinates. The remapping lookup table stores the coordinate mapping relationship between the low-altitude situation map and the corrected map. Using the corrected feature map and camera parameters as input, stereo matching calculation is performed. Through feature point detection, feature description calculation, and feature point post-processing, a feature description map is obtained. The feature description map includes feature point coordinates and matching relationships. Specifically, feature point detection locates key points in the feature image and removes abnormal candidate points, resulting in a set of detection points for the feature map. Feature description calculation calculates the gradient values ​​within the neighborhood of each feature point, obtaining a gradient direction histogram. Feature point post-processing performs L2 normalization on the gradient direction values ​​and removes points with strong edge responses, resulting in the feature description map. Depth features are extracted from the feature description map, camera parameters, and depth range. A depth map and a confidence map are obtained through depth hypothesis generation, matching cost calculation, and depth map post-processing. The multi-view image includes the feature description map and the image to be reconstructed. The camera parameters include camera intrinsic and extrinsic parameters. The initial depth map includes the depth value of each pixel, and the confidence map represents the reliability of the labeled depth estimation. The depth hypothesis generation maps each pixel of the feature map to a preset depth range. The matching cost calculation projects each pixel of the feature map into 3D space and then maps it back to the original image, calculating the similarity of corresponding pixels. The depth map post-processing removes outliers and fills in missing regions to obtain a processed initial depth map. The optimized depth map and camera intrinsic parameters are subjected to coordinate mapping, coordinate system transformation, and 3D model generation to obtain a dense point cloud of the low-altitude situation. The camera intrinsic parameter matrix includes focal length and principal point coordinate matrix. The 3D model of the low-altitude situation includes the internal 3D structure of the low-altitude situation, the external 3D structure of the low-altitude situation, model density, and model color. Among them, coordinate mapping is performed by converting pixel coordinates into 3D coordinates to obtain the model voxel coordinates in 3D space; coordinate system transformation is performed by rearranging coordinate axes and transforming coordinate units to obtain the point cloud coordinate system; dense point cloud generation is performed by converting the point cloud coordinates in the coordinate system into a coordinate matrix to obtain the dense point cloud of the low-altitude situation.

[0033] like Figure 2 As shown, this embodiment provides a method for three-dimensional reconstruction of low-altitude situation based on multiple views of UAVs. In this embodiment, the method is applied to... Figure 1 Taking the computing environment in the example, the method for low-altitude situational awareness 3D reconstruction based on UAV multi-view includes the following steps: S1: Input the low-altitude video data of the UAV into a pre-trained geospatial large model to obtain geospatial importance labels; the geospatial importance labels include geographic type, key target category, key target name and key target quantity, sub-region importance score, and sub-region latitude and longitude, wherein a high importance score indicates a high importance region, and a low importance score indicates a low importance region.

[0034] S2: Input the aforementioned geospatial importance markers into the task knowledge graph to obtain a low-altitude situational awareness acquisition scheme. The low-altitude situational awareness acquisition scheme includes: waypoints, flight numbers, and recommended routes; S3: The aforementioned low-altitude situation acquisition scheme is used to re-acquire low-altitude situation data by UAV, resulting in multi-view low-altitude situation enhanced video.

[0035] S4: Extract images frame by frame from the multi-view low-altitude situation enhancement video to obtain multi-view low-altitude situation images. For video segments containing high-importance regions, the number of extracted image frames shall be no less than 10; for video segments not containing high-importance regions, the number of extracted image frames shall be no more than 6.

[0036] S5: Based on the low-altitude situation image, perform feature processing to obtain a multi-view feature map of the low-altitude situation; the feature processing includes image de-motion blurring and high-frequency enhancement filtering methods; S6: Based on the multi-view low-altitude situational feature map, perform camera calibration to obtain the camera intrinsic parameters, camera extrinsic parameters, and three-dimensional spatial motion trajectory of the UAV camera; the spatial starting point of the three-dimensional motion trajectory coordinates is the coordinate origin, and the three-dimensional coordinates are within a preset range; S7: Extract point cloud data from the multi-view feature map to obtain a sparse point cloud of the low-altitude situation to be reconstructed; the sparse point cloud includes three-dimensional points and attributes of the point cloud; the three-dimensional points are the spatial location points of the three-dimensional structure of the low-altitude situation to be reconstructed, and the attributes are the spatial coordinates, category, color and transparency of each three-dimensional point. S8: The sparse point cloud of the low-altitude situation is reconstructed in three dimensions using a stereo matching algorithm to obtain a dense point cloud of the low-altitude situation. S9: The dense point cloud of the low-altitude situation is meshed to obtain a three-dimensional model of the low-altitude situation. By implementing steps S1-S9 above, this embodiment utilizes a large geospatial model and a task knowledge graph to improve the 3D reconstruction effect of key areas in low-altitude situational awareness. Specifically, a pre-trained large geospatial model is used to identify the importance of sub-regions, and regional importance is marked according to importance scores. Based on the importance marking results, a drone re-collection flight route recommendation is generated using the task knowledge graph to complete low-altitude airspace re-collection, resulting in multi-view low-altitude situational awareness enhanced video. Feature processing is performed on the multi-view low-altitude situational awareness enhanced video to obtain multi-view feature maps, camera calibration data, and sparse point clouds of the low-altitude situation. Using camera calibration data, sparse point clouds, and multi-view feature maps as input, stereo matching calculations are used to calculate the depth map of the low-altitude situation to be reconstructed. By finding the corresponding point in 3D space for each pixel in the image, a dense point cloud of the low-altitude situation is obtained, and 3D reconstruction of the low-altitude situation is achieved through meshing processing. The reconstruction process does not require professional experience or manual annotation. It proposes a UAV-based multi-view 3D reconstruction method for low-altitude situational awareness using low-altitude situational video, obtaining a 3D model of the low-altitude situation. This allows users to further analyze the low-altitude situation through the 3D model, helping them to easily and efficiently understand low-altitude situational information. The results are as follows: Figure 5 As shown.

[0037] In step S1, low-altitude video data from the UAV is input into a pre-trained geospatial model to obtain geospatial importance labels. These geospatial importance labels include geographic type, key target category, key target name, number of key targets, sub-region importance score, and sub-region latitude and longitude. ,in This refers to video captured by a drone at low altitude. This indicates the results of the geospatial importance ranking. Indicates geographical type, Indicates key objective categories, Indicates the name of the key objective. Indicates the number of key objectives. Indicates the importance score of the sub-region. This represents the latitude and longitude of a sub-region.

[0038] In S1, the pre-trained geospatial large model identifies the number N of movable targets in video D, and after normalization, obtains the service traffic of sub-regions. Service traffic greater than 0.8m is marked as important regions; service traffic greater than 0.4 and less than 0.8m is marked as medium-importance regions; and service traffic less than 0.4m is marked as low-importance regions.

[0039] In S1, the pre-trained geospatial large model identifies high-value targets in video D and obtains the importance weights of sub-regions after normalization. The high-value targets include: residential buildings and protected historical buildings; regions with an importance weight greater than 0.8 are marked as important; regions with an importance weight greater than 0.4 and less than 0.8 are marked as medium-importance; and regions with an importance weight less than 0.4 are marked as low-importance.

[0040] In S2, the geospatial importance markers are input into the task knowledge graph to obtain a low-altitude situational awareness acquisition scheme. This scheme includes: waypoints, flight numbers, and recommended routes. ,in This indicates the low-altitude situational awareness acquisition scheme. Indicates the recommended route. Indicates waypoints, Indicates the flight number; In S2, the low-altitude situational awareness acquisition scheme, for building-type targets, performs spatial dilation processing using a task knowledge graph to obtain the expanded spatial boundary of high-value targets. This spatial dilation process involves obtaining the spatial boundary of high-value targets from the task knowledge graph, dilating the boundary by S=A⊕B, and obtaining the expanded boundary coordinates. Here, S represents the expanded boundary coordinates, A represents the original boundary coordinates, and B represents the dilation coefficient obtained from the task knowledge graph.

[0041] In S2, the low-altitude situational awareness acquisition scheme calculates waypoints using a task knowledge graph to obtain recommended routes that avoid high-traffic sub-regions. The waypoint calculation is performed by setting a low weight for high-traffic areas, thus ensuring that the recommended routes avoid these sub-regions.

[0042] In S3, the UAV conducts low-altitude situational awareness data collection in the target area along a recommended flight path, resulting in multi-view enhanced low-altitude situational awareness video. During this low-altitude situational awareness collection, the UAV flies along the recommended flight path, passing all marked waypoints. S4: Extract images frame by frame from the multi-view low-altitude situation enhancement video to obtain multi-view low-altitude situation images. For the low-altitude situation images, 15 frames are extracted from the video segments in high-importance areas; 7 frames are extracted from the video segments in medium-importance areas; and 2 frames are extracted from the video segments in low-importance areas.

[0043] In S5, feature extraction is performed based on multi-view low-altitude situation maps to obtain multi-view low-altitude situation feature maps; each low-altitude situation map is processed by motion-de-blurring to improve the blurry afterimages caused by motion in the low-altitude situation image; each low-altitude situation map is enhanced by high-frequency filtering to improve the detail information of the low-altitude situation map. In S6, camera calibration data is generated based on the low-altitude situation map to be reconstructed. Specifically, this includes: generating the three-dimensional pose information of the camera when the low-altitude situation map was acquired for each situation structure location point, and representing the camera's motion trajectory in three-dimensional space using coordinates; generating camera intrinsic parameters for each location point, including focal length, principal point, and tilt coefficient; and generating camera extrinsic parameters for each location point in the situation image, generating a rotation matrix and translation vector to characterize the camera's position and attitude information in the world coordinate system. In S7, a sparse point cloud of the situation is generated based on the low-altitude situation feature map to be reconstructed. Specifically, this includes: performing coordinate transformation on the two-dimensional coordinates of the situation structure location points to obtain the three-dimensional coordinates of the situation structure location points. The situation structure location points are recorded as three-dimensional points, the three-dimensional coordinates of the situation structure location points are recorded as the three-dimensional coordinates of the three-dimensional points, and the structure category of the situation structure location points is recorded as the category of the three-dimensional points. All three-dimensional points and the coordinates and categories of each three-dimensional point are combined to form three-dimensional sparse point cloud data.

[0044] This embodiment first utilizes a low-altitude situational awareness map acquired by an unmanned aerial vehicle (UAV) to obtain the three-dimensional pose information of the optical probe based on the UAV's movement. Then, key structural location points are marked on the low-altitude situational awareness map, and the two-dimensional coordinates of each location point are obtained. Finally, the two-dimensional coordinates of each situational structural location point are transformed to a three-dimensional coordinate system using the recorded three-dimensional pose information to obtain the three-dimensional coordinates and category information of each location point.

[0045] In this embodiment, when collecting low-altitude situational images, a surround acquisition method can be used to obtain multiple views from different angles, which facilitates the determination of the two-dimensional coordinates of all situational structure location points.

[0046] The location points of the situation structure can be determined according to user needs. As an example, the structure location points include two types: target edge and target support point.

[0047] In S7, sparse point cloud features are extracted from multi-view low-altitude situation maps. Specifically, this includes: labeling the coordinates of feature points in three-dimensional space, recording the category of each three-dimensional point, and recording the attributes of each three-dimensional point, including: x-axis, y-axis, z-axis coordinates, transparency, and color parameters. The transparency range is [0, 1], and the color is represented by (r, g, b).

[0048] In S7, the preset region of the sparse point cloud can be a region of (64, 64, 64). Other sizes of regions can be adopted as needed. The generated three-dimensional point cloud can also be uniformly sampled to obtain the processed point cloud coordinates. This example does not impose any limitations.

[0049] In S8, for the aforementioned stereo matching algorithm, a low-altitude situational depth map is extracted to obtain a low-altitude situational dense point cloud; the preset depth value range is [0, 255]. If the depth value is close to 0, it indicates that the pixel is closer to the UAV probe; if the depth value is closer to 255, it indicates that the position of the pixel is farther away from the UAV probe. Figure 3 As shown, the stereo matching algorithm includes stereo correction, stereo matching calculation, depth map generation, and dense point cloud generation. Stereo correction is used to resolve the error relationship between pixels in different images and improve the accuracy of 3D reconstruction; stereo matching calculation is used to find corresponding points in multi-view images taken from different angles and establish the association between the same feature points in different images; depth map generation is used to calculate parallax to obtain scene depth information; dense point cloud generation is used to transform the information in the depth map into a dense 3D point cloud in space to obtain a low-altitude situational dense point cloud.

[0050] In this process, stereo correction takes feature maps and camera parameters as input. The feature maps are multi-view low-altitude feature maps with a viewing angle range of [-180, 180]. The camera parameters include extrinsic parameters, intrinsic parameters, and 3D spatial motion trajectory. Each feature map corresponds to a set of camera parameter values. The stereo correction calculation takes camera parameters, multi-view images, and sparse point clouds as inputs to calculate the k-th pixel of the reference image. Under different depth assumptions Matching cost , and obtain the cost volume tensor .in, This represents the reference image at the k-th pixel, where d represents the assumed depth value. Indicates the minimum depth value. Indicates the maximum depth value. Let C represent the matching cost of the reference image at the assumed depth value, where C represents the cost volume tensor, D is the number of depth samples, and F is the number of feature channels.

[0051] Among them, the depth map is used to extract the cost volume tensor. Inferring the optimal depth value of the k-th pixel A multi-view depth map is obtained from the optimal depth value of each pixel. The inference method adopted is the optimization inference method. .in, This represents the inferred optimal depth value for the k-th pixel. This represents the depth value of the k-th pixel. This represents the depth value of a multi-view image.

[0052] Dense point cloud generation is used to generate the multi-view depth map. Generate dense point clouds Outliers are filtered out through post-processing of dense point clouds. Where p is the pixel coordinate, K is the camera intrinsic parameter, T is the camera translation parameter, and R represents the camera rotation parameter. This represents the coordinates of the dense point cloud after filtering outliers, and r represents the threshold for filtering outliers.

[0053] In S9, the dense point cloud of the low-altitude situation is processed into a grid. Specifically, the Poisson grid algorithm is used to extract the surface grid of the dense point cloud to obtain a triangular grid of the dense point cloud of the low-altitude situation, thus obtaining a three-dimensional model of the low-altitude situation.

[0054] Example 2 Based on the same idea, this application also provides a device for implementing the above-mentioned method for three-dimensional reconstruction of low-altitude situation based on multiple UAV perspectives. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in the embodiments of the device for three-dimensional reconstruction of low-altitude situation based on multiple UAV perspectives provided below can be found in the limitations of the method for three-dimensional reconstruction of low-altitude situation based on multiple UAV perspectives described above, and will not be repeated here.

[0055] like Figure 4 As shown, this embodiment provides a device for three-dimensional reconstruction of low-altitude situation based on multiple views from an unmanned aerial vehicle (UAV). The device for three-dimensional reconstruction of low-altitude situation based on multiple views from an UAV includes: The data processing module M1 is used to generate low-altitude situation feature maps from low-altitude situation video data collected by the UAV; the video data is video data collected by the UAV during its circling flight, with the circling angle in the range of [-180, 180]; the low-altitude situation feature map includes multiple feature maps and the time and category of each feature map in the video data; the feature map is a low-altitude situation map after image enhancement and sharpening; the category is the storage format of the feature image.

[0056] The point cloud generation module M2 is used to generate sparse point clouds for camera calibration and low-altitude situational awareness. The camera calibration includes camera intrinsic parameters, camera extrinsic parameters, and camera motion trajectory. The sparse point cloud includes multiple three-dimensional points and the three-dimensional coordinates and category of each three-dimensional point. The three-dimensional points are the structural location points of the low-altitude situational awareness to be reconstructed, and their categories are the structural categories of the low-altitude situational awareness structural location points. The three-dimensional coordinates of the three-dimensional points are the origin of the coordinate system, and the three-dimensional coordinates of all the three-dimensional points are within a preset range.

[0057] The 3D reconstruction module M3 is used to perform 3D reconstruction of the low-altitude situation to be reconstructed based on camera calibration, multi-angle feature maps, and sparse point clouds, obtaining a 3D model of the low-altitude situation. The 3D reconstruction includes dense point cloud generation and 3D model generation. The dense point cloud generation includes multiple 3D points and the 3D coordinates and category of each 3D point; the 3D points are the structural location points of the low-altitude situation to be reconstructed, and the category is the structural category of the low-altitude situation structural location point; the 3D coordinates of the 3D points are at the origin, and the 3D coordinates of all 3D points are within a preset range. The 3D model generation uses the dense point cloud as input and extracts the 3D model of the low-altitude situation using a curved mesh. The visualization module M4 is used to visualize and interact with the three-dimensional model, allowing users to view the low-altitude situation three-dimensional model in various ways.

[0058] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of low-altitude situation based on multiple perspectives from unmanned aerial vehicles (UAVs), characterized in that, include: S1. Perform geospatial classification on the multi-view video data of the UAV to obtain geospatial importance labels; the geospatial importance labels include geographic type, category of key target, name of key target, number of key targets, importance score of sub-region, and latitude and longitude of sub-region; S2. Generate a low-altitude situation acquisition scheme for the sub-region based on the importance score of the sub-region. S3. Based on the aforementioned low-altitude situation acquisition scheme, re-acquire low-altitude situation data from the UAV to obtain multi-view low-altitude situation enhanced video. S4. Based on the importance score of the sub-region, the multi-view low-altitude situation enhancement video and UAV multi-view video data are sampled at different sampling rates, and three-dimensional reconstruction is performed based on the sampled images.

2. The method for low-altitude situational awareness 3D reconstruction based on multiple UAV perspectives according to claim 1, characterized in that, Geospatial classification of multi-view video data from drones, including: Geospatial classification is performed on multi-view video data from drones using a pre-trained geospatial large model.

3. The method for low-altitude situational awareness reconstruction based on multiple UAV perspectives according to claim 2, characterized in that, Geospatial classification of multi-view video data from drones, including: By using a pre-trained geospatial large model, the number of movable targets in the video is identified, and after normalization, the business traffic of the sub-region is obtained. The importance score of each sub-region is obtained based on the business traffic of each sub-region.

4. The method for low-altitude situational awareness reconstruction based on multiple UAV perspectives according to claim 3, characterized in that, Based on the importance score of the sub-region, a low-altitude situational awareness acquisition scheme is generated for each sub-region, including: Using a task knowledge graph, a low-altitude situational awareness acquisition scheme is generated for each sub-region based on its importance score.

5. The method for low-altitude situational awareness reconstruction based on multiple UAV perspectives according to claim 4, characterized in that, Using a task knowledge graph, a low-altitude situational awareness acquisition scheme is generated for each sub-region based on its importance score, including: For sub-regions with high business traffic, spatial expansion is performed on the identified building type targets within them using a task knowledge graph to obtain the expanded spatial boundaries of the targets. By calculating waypoints using the task knowledge graph, recommended routes that avoid high-traffic sub-regions are obtained.

6. The method for low-altitude situational awareness 3D reconstruction based on multiple UAV perspectives according to claim 1, characterized in that, The low-altitude situational awareness acquisition scheme includes: waypoints, flight numbers, and recommended routes.

7. A low-altitude situational awareness 3D reconstruction device based on multiple perspectives from unmanned aerial vehicles (UAVs), characterized in that, Used to perform the low-altitude situational awareness 3D reconstruction method based on multiple UAV perspectives as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, include: Memory and processor; The memory is configured to store executable instructions; The processor is configured to implement the low-altitude situational awareness 3D reconstruction method based on UAV multi-view as described in any one of claims 1-6 when executing the executable instructions stored in the memory.

9. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computer, implement the low-altitude situational awareness 3D reconstruction method based on multiple UAV perspectives as described in any one of claims 1-6.

Images