Unmanned aerial vehicle live-action three-dimensional modeling method and device, computer equipment and medium

Abstract

The invention relates to an unmanned aerial vehicle live-action three-dimensional modeling method and device, computer equipment and a medium, and the method comprises the steps: obtaining original image data and corresponding original positioning data collected in the flying process of an unmanned aerial vehicle; acquiring synchronous differential data fed back by the base station; performing PPK joint calculation on the original positioning data and the synchronous differential data to obtain POS data; and performing live-action three-dimensional reconstruction according to the original image data and the POS data. In the whole process, accurate POS data are obtained by acquiring synchronous differential data fed back by the base station and performing PPK joint calculation on the synchronous differential data and original positioning data acquired by the unmanned aerial vehicle, so that the positioning accuracy is greatly improved; and the original image data and the precise POS data are combined to carry out live-action three-dimensional reconstruction, so that the problem of geometric distortion of the model caused by low precision of the positioning data is avoided, and finally high-precision unmanned aerial vehicle live-action three-dimensional modeling is realized.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for UAV real-scene 3D modeling. Background Technology

[0002] Oblique photogrammetry, as a mainstream method for rapidly acquiring 3D models of urban landscapes, plays a crucial role in numerous fields such as urban planning, surveying and mapping, and disaster monitoring due to its ability to capture geographic information from multiple angles and in all directions. Small unmanned aerial vehicles (UAVs), with their significant advantages of low cost and high portability, have been widely used in this field, becoming an important carrier for implementing oblique photogrammetry and facilitating the large-scale acquisition of geospatial data.

[0003] However, the Global Navigation Satellite System (GNSS) receiver modules built into small (ordinary) drones have significant limitations, typically supporting only single-point positioning. This positioning method provides low accuracy, particularly in elevation, where errors are extremely pronounced, often only reaching the meter level. This low-precision positioning data makes it difficult for realistic 3D models built from such data to meet practical requirements in terms of geometric accuracy. They fail to accurately reflect the details and features of real-world geographical scenes, severely impacting the quality and usability of the 3D models.

[0004] Therefore, there is an urgent need for a high-precision 3D modeling solution for drone real-world scenes. Summary of the Invention

[0005] Therefore, it is necessary to provide a high-precision method, device, computer equipment, computer-readable storage medium, and computer program product for UAV real-scene 3D modeling to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for real-world 3D modeling of unmanned aerial vehicles (UAVs). The method includes:

[0007] Acquire raw image data and corresponding raw positioning data collected during the drone's flight;

[0008] Obtain the synchronous differential data fed back from the base station;

[0009] The raw positioning data and the synchronous differential data are jointly calculated using PPK (Post-Processed Kinematic Positioning with Fusion) to obtain POS (Position and Orientation System) data.

[0010] Real-world 3D reconstruction is performed based on the original image data and POS data.

[0011] In one embodiment, the raw positioning data and the synchronization differential data are jointly calculated using PPK to obtain POS data, including:

[0012] The initial positioning result is obtained by performing PPK joint calculation on the original positioning data and the synchronous differential data.

[0013] The initial positioning results are filtered, smoothed, and optimized to obtain POS data.

[0014] In one embodiment, the initial positioning results are filtered, smoothed, and optimized to obtain POS data including:

[0015] Extract the elevation and latitude / longitude components from the initial positioning results;

[0016] Kalman filtering is applied to smooth and optimize the elevation component;

[0017] Once the Kalman filtering smoothing and optimization of the elevation component is completed, the latitude and longitude components are then smoothed and optimized using Kalman filtering to obtain POS data. The POS data includes the processed elevation component and the processed latitude and longitude components.

[0018] In one embodiment, real-scene 3D reconstruction based on raw image data and POS data includes:

[0019] Based on the original image data and POS data, calculate the actual image overlap, spatial distribution of POS precision, and texture richness based on image features of the original image data;

[0020] Based on a pre-trained machine learning model, actual image overlap, POS accuracy spatial distribution, and texture richness based on image features, the success probability of aerial triangulation is predicted.

[0021] If the predicted success probability is greater than the preset success probability threshold, then real-scene 3D reconstruction is performed based on the original image data and POS data.

[0022] In one embodiment, the above-mentioned UAV real-scene 3D modeling method further includes:

[0023] If the predicted success probability is not greater than the preset success probability threshold, then the weak areas corresponding to the aerial triangulation solution are analyzed based on the actual image overlap, POS accuracy spatial distribution, and texture richness based on image features.

[0024] Generate supplementary flight routes based on the weak areas and control the drones to fly the supplementary flight routes;

[0025] Return to the steps for obtaining the raw image data and corresponding raw positioning data collected during the drone's flight.

[0026] In one embodiment, real-scene 3D reconstruction based on raw image data and POS data includes:

[0027] Import the raw image data and POS data into the preset 3D modeling software;

[0028] In the preset 3D modeling software, the POS data is used as a strong constraint to perform aerial triangulation and obtain the aerial triangulation results.

[0029] Based on the aerial triangulation results, density matching, mesh construction, and texture mapping are performed sequentially to generate a realistic 3D model.

[0030] In one embodiment, acquiring the raw image data and corresponding raw positioning data collected during the drone's flight includes:

[0031] Obtain the data boundary file of the target area;

[0032] Generate photographic flight paths based on the data boundary file;

[0033] Acquire raw image data and corresponding GNSS observation data collected by drones during their flight photography routes.

[0034] Secondly, this application also provides a drone-based 3D modeling device. The device includes:

[0035] The first parameter acquisition module is used to acquire the raw image data and corresponding raw positioning data collected during the flight of the UAV.

[0036] The second parameter acquisition module is used to acquire the synchronous differential data fed back by the base station;

[0037] The joint calculation module is used to perform PPK joint calculation on the raw positioning data and the synchronous differential data to obtain POS data;

[0038] The 3D reconstruction module is used to perform real-scene 3D reconstruction based on the original image data and POS data.

[0039] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0040] Acquire raw image data and corresponding raw positioning data collected during the drone's flight;

[0041] Obtain the synchronous differential data fed back from the base station;

[0042] The original positioning data and the synchronous differential data are combined to perform PPK calculation to obtain POS data;

[0043] Real-world 3D reconstruction is performed based on the original image data and POS data.

[0044] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] Acquire raw image data and corresponding raw positioning data collected during the drone's flight;

[0046] Obtain the synchronous differential data fed back from the base station;

[0047] The original positioning data and the synchronous differential data are combined to perform PPK calculation to obtain POS data;

[0048] Real-world 3D reconstruction is performed based on the original image data and POS data.

[0049] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0050] Acquire raw image data and corresponding raw positioning data collected during the drone's flight;

[0051] Obtain the synchronous differential data fed back from the base station;

[0052] The original positioning data and the synchronous differential data are combined to perform PPK calculation to obtain POS data;

[0053] Real-world 3D reconstruction is performed based on the original image data and POS data.

[0054] The aforementioned UAV real-scene 3D modeling method, apparatus, computer equipment, storage medium, and computer program products acquire raw image data and corresponding raw positioning data collected during UAV flight; acquire synchronization differential data fed back from the base station; perform PPK joint calculation on the raw positioning data and synchronization differential data to obtain POS data; and perform real-scene 3D reconstruction based on the raw image data and POS data. Throughout the process, by acquiring the synchronization differential data fed back from the base station and performing PPK joint calculation with the raw positioning data collected by the UAV to obtain accurate POS data, the positioning accuracy is greatly improved. Furthermore, by combining the raw image data and accurate POS data for real-scene 3D reconstruction, the geometric distortion problem caused by low positioning data accuracy is avoided, ultimately achieving high-precision UAV real-scene 3D modeling. Attached Figure Description

[0055] Figure 1 This is an application environment diagram of the UAV real-scene 3D modeling method in one embodiment;

[0056] Figure 2 This is a flowchart illustrating a method for UAV real-scene 3D modeling in one embodiment;

[0057] Figure 3 This is a flowchart illustrating the UAV real-scene 3D modeling method in another embodiment;

[0058] Figure 4 This is a flowchart illustrating the UAV real-scene 3D modeling method in yet another embodiment;

[0059] Figure 5 This is a structural block diagram of a device that illustrates the process of a method in one embodiment;

[0060] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The UAV real-scene 3D modeling method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with drone 104 via a network. The user operates terminal 102, sending a real-scene 3D modeling command to drone 104. Drone 104 responds to the command and begins flying according to the user-selected flight area, collecting raw image data and corresponding raw positioning data during the flight, and transmitting this collected data back to terminal 102. Terminal 102 receives the data transmitted back from drone 104, acquiring the raw image data and corresponding raw positioning data collected during the drone's flight; acquiring the synchronization differential data fed back by the base station; performing PPK joint calculation on the raw positioning data and synchronization differential data to obtain POS data; and performing real-scene 3D reconstruction based on the raw image data and POS data. Furthermore, terminal 102 can display the final generated real-scene 3D model to the user.

[0063] In one embodiment, such as Figure 2 As shown, a method for real-world 3D modeling of UAVs is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0064] S200: Acquires raw image data and corresponding raw positioning data collected during the flight of the UAV.

[0065] During flight, an unmanned aerial vehicle (UAV) equipped with image acquisition devices (such as cameras) captures a series of images of the ground or target area according to preset shooting parameters (such as shooting intervals and exposure times). This image data contains information such as the appearance and texture of features in the target area, serving as the foundational visual material for subsequent 3D reconstruction. Additionally, the positioning equipment on the UAV (such as a GNSS receiver) simultaneously records the UAV's own position information (longitude, latitude, altitude) and attitude information (heading angle, pitch angle, roll angle) while capturing each raw image. This positioning data provides a spatial reference for the image, helping to determine its accurate location in 3D space. Here, the raw image data provides visual information for subsequent 3D reconstruction, while the raw positioning data endows the image with spatial attributes; together, they provide the necessary foundational data for subsequent accurate modeling.

[0066] S400: Acquires synchronous differential data fed back from the base station.

[0067] The base stations here can be local base stations or network base stations, specifically local or network CORS (Continuously Operating Reference Stations). A local CORS base station is a fixed reference station established within a specific area, equipped with high-precision GNSS receiving equipment. It continuously receives satellite signals and compares the received satellite observation data with known reference station coordinates to calculate differential correction information. Local CORS base stations typically serve users within a certain range, providing relatively timely and stable differential data. A network CORS base station is a system composed of multiple CORS base stations connected by a network, offering wider coverage. It utilizes wide-area differential technology to integrate and process reference station data from different regions, providing users with cross-regional differential services.

[0068] Local or network CORS reference stations calculate differential correction information based on real-time received satellite signals and known coordinates. This information is transmitted in real-time to the GNSS receiver on the UAV via wireless communication links (such as radios, networks, etc.). Synchronous differential data can eliminate or reduce errors generated during satellite signal propagation, such as satellite clock bias, ionospheric delay, and tropospheric delay, thereby improving positioning accuracy. The raw positioning data collected by the UAV is fused with the synchronous differential data obtained from the local or network CORS reference station. Through specific algorithms and models, errors in the raw positioning data are eliminated, resulting in more accurate positioning information. This joint calculation method fully utilizes the high-precision data from the CORS reference station, improving the accuracy of UAV positioning.

[0069] Here, the synchronous differential data acquired from local or network CORS base stations is crucial for correcting errors in the UAV's original positioning data. Through joint calculations, more accurate correction information can be provided for subsequent precise position calculations, thereby improving the spatial positioning accuracy of the entire modeling process.

[0070] S600: Performs PPK joint calculation on the raw positioning data and the synchronous differential data to obtain POS data.

[0071] PPK is a high-precision GNSS positioning technology that utilizes satellite data observed by a base station (local or network CORS base station) and a rover (GNSS receiver on the UAV) within the same time period, performing joint calculations through post-processing. During this joint calculation, the raw positioning data collected by the UAV is matched and calculated with synchronous differential data obtained from the base station, eliminating errors such as satellite clock bias, ionospheric delay, and tropospheric delay, thus obtaining more accurate UAV position and attitude information. Compared to Real-Time Dynamic Differential (RTK), PPK is unaffected by communication link stability and achieves higher positioning accuracy. POS data, or Positioning and Orientation System data, contains the precise position (longitude, latitude, altitude) and attitude (heading angle, pitch angle, roll angle) information of the UAV at the time of each image capture. POS data is the key bridge connecting raw image data with three-dimensional spatial coordinates, accurately mapping two-dimensional image data into three-dimensional space. For example, when photographing a bridge, POS data can accurately record the UAV's position and attitude relative to the bridge at the time of each image capture, ensuring the accuracy of subsequent 3D reconstruction.

[0072] Here, by using PPK joint calculation, the original positioning data and synchronous differential data are fused together, which greatly improves the accuracy of the UAV's position and attitude information, providing an accurate spatial reference for subsequent real-scene 3D reconstruction and ensuring that the reconstructed 3D model has high geometric accuracy.

[0073] S800: Real-world 3D reconstruction based on original image data and POS data.

[0074] Using computer vision, photogrammetry, and other technologies and algorithms, raw image data and POS data are processed and analyzed. Through operations such as feature extraction, matching, and spatial triangulation, multiple 2D image data with overlapping areas are converted into 3D models with realistic geometry and texture. In this process, POS data provides precise spatial location and pose information of the images, enabling the reconstructed 3D model to accurately reflect the spatial structure and appearance of the actual scene. For example, in cultural heritage protection, high-precision 3D models of ancient buildings can be generated through real-scene 3D reconstruction, providing detailed data support for protection and restoration.

[0075] The aforementioned UAV real-scene 3D modeling method acquires raw image data and corresponding raw positioning data collected during UAV flight; acquires synchronization differential data fed back from the base station; performs PPK joint calculation on the raw positioning data and synchronization differential data to obtain POS data; and performs real-scene 3D reconstruction based on the raw image data and POS data. Throughout the process, by acquiring the synchronization differential data fed back from the base station and performing PPK joint calculation with the raw positioning data collected by the UAV to obtain accurate POS data, the positioning accuracy is greatly improved. Furthermore, by combining the raw image data and accurate POS data for real-scene 3D reconstruction, the geometric distortion problem caused by low positioning data accuracy is avoided, ultimately achieving high-precision UAV real-scene 3D modeling.

[0076] In one embodiment, such as Figure 3 As shown, the original positioning data and the synchronization differential data are jointly calculated using PPK to obtain the POS data, which includes:

[0077] S620: Perform PPK joint calculation on the original positioning data and the synchronous differential data to obtain the initial positioning result.

[0078] Specifically, the PPK joint calculation process mainly includes two stages: differential data generation and joint calculation. I. Differential data generation includes: 1) Calculating the distance between the base station and the satellites; based on the satellite ephemeris data recorded by the base station, the satellite position at each observation time is calculated. Combining the known coordinates of the base station, the distance between the base station and each satellite is calculated using the distance formula. 2) Generating differential corrections; comparing the calculated distance between the base station and the satellites with the pseudorange measured by the GNSS receiver at the base station to obtain the pseudorange differential correction. Similarly, carrier phase differential corrections can also be generated using carrier phase observation data. II. Joint calculation includes: 1) Data matching; precisely matching the UAV's observation data with the differential corrections generated by the base station according to time. This ensures that the UAV's data corresponds to the correct differential correction at each observation time for subsequent joint calculations. 2) Establishing observation equations; establishing the UAV's observation equations based on the GNSS positioning principle. The observation equations include unknowns such as the distance between the UAV and the satellites, the satellite's position, the UAV's coordinates, and various error terms. The differential correction from the base station is incorporated into the UAV's observation equations to eliminate or reduce the influence of common errors, resulting in corrected observation equations. 3) Parameter estimation: Parameter estimation methods, such as the least squares method, are used to solve the corrected observation equations, estimating unknown parameters such as the UAV's coordinates and clock bias. The least squares method obtains the optimal parameter estimation result by minimizing the sum of squared residuals between the observed and calculated values. 4) Output results: The high-precision positioning results obtained through joint calculation (including the UAV's coordinates, clock bias, etc.) are output in a specified format.

[0079] Here, by performing PPK joint calculations on the original positioning data containing errors and the synchronous differential data that can correct the errors, the errors introduced during satellite signal propagation are initially eliminated, and relatively more accurate initial information on the UAV's position and attitude is obtained, providing a data foundation for further optimization processing.

[0080] S640: Filter, smooth, and optimize the initial positioning results to obtain POS data.

[0081] Specific filtering algorithms (such as Kalman filtering) are used to process the initial positioning results. Kalman filtering is a highly efficient recursive filter that can estimate and predict the system's state based on the system's state equation and observation equation. When processing the initial positioning results, it can filter out noise and outliers, making the positioning data smoother and more stable. For example, during UAV flight, external interference may cause temporary fluctuations in positioning data; filtering and smoothing can eliminate these fluctuations, resulting in more continuous positioning information. The data obtained after filtering, smoothing, and optimization includes the UAV's precise position (longitude, latitude, altitude) and attitude (heading angle, pitch angle, roll angle). POS data is a crucial bridge connecting the original image data and three-dimensional spatial coordinates; it can accurately map two-dimensional image data into three-dimensional space, providing accurate spatial references for subsequent real-world 3D reconstruction.

[0082] In one embodiment, the initial positioning results are filtered, smoothed, and optimized to obtain POS data including:

[0083] Step 1: Extract the elevation and latitude / longitude components from the initial positioning results.

[0084] The elevation component refers to the vertical altitude information of the UAV in the initial positioning result, usually expressed as altitude. Elevation data is crucial for applications such as 3D modeling and terrain analysis; however, during GNSS positioning, elevation measurements are easily affected by factors such as atmospheric refraction and multipath effects, resulting in relatively low accuracy. The latitude and longitude component represents the horizontal position information of the UAV in the initial positioning result. Longitude determines the east-west position, and latitude determines the north-south position. Latitude and longitude data are key parameters for determining the UAV's horizontal position on the Earth's surface, and are also affected by factors such as satellite signal propagation errors and receiver performance.

[0085] Step 2: Perform Kalman filtering smoothing and optimization on the elevation component.

[0086] In practical applications, the main factors leading to elevation errors include: 1. Inherent limitations of navigation and positioning systems (the core reason). Consumer drones primarily rely on a combination of GNSS and IMU (Inertial Measurement Unit) to determine their position, but their accuracy is worst in the vertical direction. The influence of satellite geometry: GNSS systems are inherently more accurate in the horizontal (X, Y) direction than in the vertical (Z) direction. Satellites are mainly distributed in the zenith hemisphere; this geometric structure, known in imaging as the "elevation angle," results in significantly greater uncertainty in vertical calculations than in the horizontal direction. Therefore, latitude and longitude accuracy is high and optimization is unnecessary. Single-point positioning mode: Most consumer drones use standard single-point positioning, relying solely on satellite signals received by the drone itself to calculate its position. This method is affected by various factors such as atmospheric delay (ionosphere, troposphere), satellite clock errors, and orbital errors. These effects are particularly pronounced in the vertical component, leading to elevation data drifts of several meters or even tens of meters. Cumulative error of the IMU: The IMU calculates displacement by measuring acceleration and angular velocity, but this process introduces drift. While highly accurate in the short term, small errors accumulate after a few minutes of flight, especially in the vertical direction, leading to increasingly inaccurate altitude data. 2. Barometer Dependence and Susceptibility to Interference: The primary sensor for measuring relative altitude in consumer drones is the barometer. Its principle is that air pressure decreases with increasing altitude. Environmental Sensitivity: Barometers are highly susceptible to environmental interference. Temperature changes, wind speed (wind pressure), and weather system changes (such as cloud cover causing a drop in air pressure) can all cause drastic fluctuations in air pressure readings. The drone's own rotor downwash can also interfere with the barometer, resulting in unstable altitude readings during hovering or ascent / descendance. Relative Reference: Barometers measure relative altitude, i.e., altitude relative to the takeoff point. They cannot directly provide absolute altitude. To obtain the true altitude, the precise altitude of the takeoff point is needed, along with the geodetic height provided by GNSS. The two must be combined for conversion, introducing new sources of error. Therefore, prioritizing smoothing and optimization of the elevation component reduces the possibility of overall model tilt or translation, and yields more accurate altitude readings.

[0087] Specifically, Kalman filtering can not only estimate elevation data in real time, but also smooth the current estimate using historical data, eliminating noise and outliers. Furthermore, based on the principles of Kalman filtering, the filter parameters are continuously adjusted to gradually bring the elevation estimate closer to the true value, thereby optimizing the elevation component.

[0088] Step 3: After the Kalman filtering smoothing and optimization of the elevation component is completed, the latitude and longitude components are then smoothed and optimized using Kalman filtering to obtain POS data. The POS data includes the processed elevation component and the processed latitude and longitude components.

[0089] After the elevation component undergoes Kalman filtering smoothing and optimization, the latitude and longitude components are processed in the same way. This order is based on the characteristics of elevation and latitude / longitude in positioning and the consideration of processing effects. Processing the elevation component first ensures that the vertical positioning accuracy meets the requirements, while processing the latitude and longitude components ensures that the horizontal positioning accuracy matches the vertical accuracy. Similar to the elevation component processing, when performing Kalman filtering on the latitude and longitude components, the optimal latitude and longitude estimates for the current moment are calculated based on the previous moment's estimated latitude and longitude values ​​and the current moment's observed latitude and longitude values. Smoothing eliminates noise and outliers in the data, and adjusting the filter parameters optimizes the latitude and longitude components.

[0090] In one embodiment, such as Figure 4 As shown, the real-scene 3D reconstruction based on the original image data and POS data includes:

[0091] S820: Based on the original image data and POS data, calculate the actual image overlap, spatial distribution of POS precision, and texture richness based on image features of the original image data.

[0092] Actual image overlap reflects the spatial overlap between adjacent original images. By analyzing the shooting position and angle information of adjacent images in the original image data (which can be obtained from POS data), and combining this with the coverage area of ​​the images, the overlap ratio between adjacent images is calculated. Actual image overlap is crucial for aerial triangulation; appropriate overlap ensures sufficient corresponding points for matching and calculation during aerial triangulation, thereby improving the accuracy of the solution. POS data contains position and attitude information, and its accuracy may vary at different spatial locations. By analyzing the POS data and considering the impact of various error factors (such as GNSS signal obstruction, multipath effects, etc.) on the POS accuracy at different locations, a spatial distribution map of POS accuracy within the survey area is drawn. Understanding the spatial distribution of POS accuracy helps assess the reliability of aerial triangulation in different regions. Image features, such as edges, corners, and textures, are extracted from the original image data. Based on the distribution and complexity of these image features in the images, the texture richness of the images is quantified. Images with rich textures provide more information for feature matching in aerial triangulation, helping to improve the accuracy and stability of the solution.

[0093] Specifically, when calculating the actual image overlap, the shooting position and pose information of each original image are first determined based on the POS data. The coverage area of ​​the image is then calculated by combining the image's shooting parameters (such as focal length and sensor size). For adjacent images, the actual image overlap is obtained by calculating the ratio of their overlapping area to the total coverage area. Regarding the spatial distribution of POS accuracy, factors such as GNSS signal errors are considered, and the accuracy of POS data is evaluated at different spatial locations. For example, in mountainous areas, POS accuracy may be lower due to signal obstruction; in open areas, accuracy is higher. By analyzing the error magnitude of POS data at each location, a spatial distribution map of accuracy is drawn, using different colors or values ​​to represent the accuracy level of different areas. When calculating the texture richness based on image features, image processing algorithms (such as edge detection algorithms) are used to extract features such as edges and corners from the original image. The number, distribution density, and complexity of these features are statistically analyzed to quantify the texture richness of the image.

[0094] Here, these three metrics are calculated to comprehensively evaluate the quality of the raw image data and POS data, and their impact on aerial triangulation. Actual image overlap, spatial distribution of POS accuracy, and texture richness reflect the applicability of the data in aerial triangulation from different perspectives, providing basic data for subsequent prediction of the success probability of aerial triangulation.

[0095] S840: Based on a pre-trained machine learning model, actual image overlap, POS accuracy spatial distribution, and texture richness based on image features, predict the success probability of aerial triangulation.

[0096] A pre-trained machine learning model is a machine learning model that has been pre-trained. Specifically, it is trained on a large amount of historical data (including combinations of different actual image overlap, POS accuracy spatial distribution, and texture richness, along with the corresponding success or failure results of aerial triangulation). It learns the complex relationship between these metrics and the probability of successful aerial triangulation. During prediction, the calculated actual image overlap, POS accuracy spatial distribution, and texture richness are used as input features and fed into the pre-trained machine learning model. Based on the input features, the machine learning model uses its learned patterns and rules to output a value between 0 and 1, representing the probability of successful aerial triangulation. This probability value reflects the likelihood of successful aerial triangulation under the current data conditions.

[0097] S862: If the predicted success probability is greater than the preset success probability threshold, then perform real-scene 3D reconstruction based on the original image data and POS data.

[0098] The preset success probability threshold is a pre-defined standard value used to determine whether aerial triangulation has a high probability of success. For example, it can be set to 95%. When the predicted success probability is greater than this threshold, the data quality is considered to meet the requirements, and real-scene 3D reconstruction can be performed. When the predicted success probability is greater than the preset threshold, it indicates that the current data conditions are favorable for aerial triangulation. At this point, the original image data and POS data are imported into the 3D modeling software. Through a series of steps, including aerial triangulation (calculating the exterior orientation elements and coordinates of corresponding points in the image), dense matching (generating dense point clouds), and surface reconstruction (constructing the surface of the 3D model), a real-scene 3D model is finally generated.

[0099] In one embodiment, such as Figure 4 As shown, the above-mentioned UAV real-scene 3D modeling method also includes:

[0100] S863: If the predicted success probability is not greater than the preset success probability threshold, then the weak areas corresponding to the aerial triangulation solution are analyzed based on the actual image overlap, POS accuracy spatial distribution, and texture richness based on image features.

[0101] If the predicted success probability is not greater than the preset success probability threshold, it indicates that the data conditions are unfavorable for aerial triangulation. Further analysis of the weak areas in aerial triangulation is needed, based on the actual image overlap, spatial distribution of POS accuracy, and the richness of texture based on image features. Specifically, weak areas include POS transition points, low overlap, or areas with weak texture. Low overlap leads to insufficient numbers of corresponding points, affecting the accuracy and stability of the solution. Areas with low POS accuracy, such as those with GNSS signal obstruction or multipath effects causing accuracy transitions, will result in inaccurate determination of image exterior orientation elements, thus affecting the calculation of relative positions between images in aerial triangulation, becoming weak areas. Areas with poor texture, lacking obvious feature points, make it difficult to find enough corresponding points during matching, easily leading to aerial triangulation failure, and are potential weak areas.

[0102] In practical applications, corresponding thresholds can be set for three types of indicators: actual image overlap, spatial distribution of POS accuracy, and texture richness based on image features. The actual values ​​of different regions are compared with these three thresholds to identify areas where the indicator values ​​are below the corresponding thresholds; these areas are then identified as weak regions. By comprehensively analyzing these three indicators, we can accurately locate weak regions that may lead to failure in aerial triangulation calculations, providing clear targets for subsequent re-flight strategies and improving the targeting and effectiveness of re-flight efforts.

[0103] S864: Generates a replacement flight path based on the weak area and controls the UAV to fly the replacement flight path.

[0104] Based on the weak areas identified through S863 analysis, intelligent flight path planning technology is used to generate supplementary flight paths. This technology considers the specific location and extent of the weak areas, as well as factors such as the UAV's flight performance (e.g., maximum speed, endurance) and shooting parameters (e.g., focal length, shooting interval). The generated supplementary flight path must ensure that the UAV can cover all weak areas during flight, and that the acquired image data meets the requirements for aerial triangulation. The generated supplementary flight path file is then injected into the UAV's flight control system using a dedicated data conversion and transmission tool. The flight control system then controls the consumer-grade multi-lens UAV to perform automatic flight according to the instructions in the flight path file.

[0105] S865: Returns to the steps for acquiring raw image data and corresponding raw positioning data collected during the drone's flight.

[0106] During flight, the UAV follows a predetermined flight path to capture raw image data (oblique image sequences) and their corresponding raw GNSS observation data. Returning to the data acquisition step completes the process, creating a closed loop. Based on the supplemented, complete dataset, subsequent PPK joint calculations and aerial triangulation are performed again, improving the accuracy of the POS data and the success rate of aerial triangulation, ultimately generating a high-quality realistic 3D model. This continuous data optimization and reprocessing ensures the accuracy and reliability of the 3D modeling.

[0107] In one embodiment, real-scene 3D reconstruction based on raw image data and POS data includes:

[0108] Step 1: Import the original image data and POS data into the preset 3D modeling software.

[0109] Preset 3D modeling software is a software tool specifically designed for processing image and spatial data to construct 3D models. It possesses powerful data processing and modeling algorithms, capable of integrating raw image data and POS data to complete the conversion from 2D images to 3D models.

[0110] Step 2: Perform aerial triangulation using POS data as a strong constraint in the preset 3D modeling software to obtain the aerial triangulation results.

[0111] POS data provides precise position and orientation information at the time of image capture, which is used as a strong constraint during aerial triangulation. This means that when calculating the relative positions and orientations between images, the initial values ​​provided by the POS data are preferentially referenced, thereby improving the accuracy and stability of the solution. Aerial triangulation is a core step in photogrammetry. Its purpose is to determine the precise spatial position (X, Y, Z) and orientation (rotation angle) of each image by analyzing corresponding points in multiple images, and to generate the coordinates of a massive 3D point cloud. This data is a key element in constructing the 3D model, directly reflecting the spatial distribution and morphology of ground features.

[0112] Here, using POS data as a strong constraint for aerial triangulation can fully utilize the high-precision information of POS data and improve the accuracy of aerial triangulation. The core data obtained from aerial triangulation, such as the precise position and pose of the photographs and the 3D point cloud coordinates, provides an accurate spatial reference for subsequent steps such as dense matching, mesh construction, and texture mapping, directly affecting the quality of the final 3D model.

[0113] Step 3: Based on the aerial triangulation results, perform density matching, mesh construction and texture mapping in sequence to generate a realistic 3D model.

[0114] Dense matching, based on the 3D point cloud obtained from aerial triangulation, further performs dense point correspondence matching between images. By analyzing the texture and features in the images, more corresponding points are found, thus generating a denser 3D point cloud. Dense matching can improve the detail and accuracy of the model, making the model more realistically reflect the actual shape of the ground features.

[0115] Mesh construction involves connecting and topologically processing densely matched 3D point clouds to create a mesh model composed of triangles or polygons. The mesh model forms the basic skeleton of a 3D model, defining its geometry and surface structure. Proper mesh construction ensures the smoothness and continuity of the model, improving its visual appeal.

[0116] Texture mapping maps texture information from the original image onto a mesh model, giving the model a realistic appearance and texture. By selecting appropriate images and mapping algorithms, details such as color and texture in the image can be accurately assigned to the mesh model, generating a highly realistic 3D model of the scene.

[0117] Here, density matching, mesh construction, and texture mapping are performed sequentially, representing a process of gradually refining the 3D model based on the aerial triangulation results. Density matching increases the model's detail and accuracy, mesh construction establishes the model's basic structure, and texture mapping gives the model a realistic appearance. These three steps work together to ultimately generate a high-quality realistic 3D model, realizing the transformation from raw data to a visualized 3D model.

[0118] In one embodiment, acquiring the raw image data and corresponding raw positioning data collected during the drone's flight includes:

[0119] Step 1: Obtain the data boundary file of the target area.

[0120] Obtaining the data boundary file for the target area is the starting point of the entire data acquisition process. By determining the boundary of the target area, a foundation is provided for generating a compliant photographic flight path, ensuring that the data collected by the drone can completely cover the target area and avoiding data omissions or the collection of data from irrelevant areas.

[0121] Step 2: Generate the photography flight path based on the data boundary file.

[0122] A photographic flight path is the route a drone takes during flight, following a predetermined path to capture images. For multi-lens oblique photography, the flight path needs to comprehensively consider the shooting angles and coverage areas of multiple lenses to ensure that image data of the target area can be acquired from different angles. Here, generating a photographic flight path based on the data boundary file transforms the spatial information of the target area into an executable flight path for the drone. A reasonable photographic flight path ensures that the drone efficiently collects image data during flight, improving the efficiency and quality of data acquisition, while avoiding problems such as repeated flights or incomplete data coverage caused by an unreasonable flight path.

[0123] Step 3: Obtain the raw image data and corresponding GNSS observation data collected by the UAV during its flight photography route.

[0124] GNSS observation data, obtained through the Global Navigation Satellite System, records the UAV's position (longitude, latitude, and altitude) at various moments during its flight. This data provides precise spatial positioning for imagery data, serving as a crucial basis for subsequent aerial triangulation and 3D reconstruction. Acquiring the raw imagery data and corresponding GNSS observation data collected during the UAV's flight path is a critical step in completing the data acquisition task. The raw imagery data provides visual information about the target area, while the GNSS observation data provides spatial location information. The combination of these two provides a complete and accurate data foundation for subsequent 3D reconstruction, ensuring the generation of a high-quality realistic 3D model.

[0125] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0126] Based on the same inventive concept, this application also provides a UAV real-scene 3D modeling device for implementing the aforementioned UAV real-scene 3D modeling method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more UAV real-scene 3D modeling device embodiments provided below can be found in the limitations of the UAV real-scene 3D modeling method described above, and will not be repeated here.

[0127] In one embodiment, such as Figure 5 As shown, a drone-based 3D modeling device is provided. The device includes:

[0128] The first parameter acquisition module 200 is used to acquire the raw image data and corresponding raw positioning data collected during the flight of the UAV.

[0129] The second parameter acquisition module 400 is used to acquire the synchronous differential data fed back by the base station;

[0130] The joint calculation module 600 is used to perform PPK joint calculation on the raw positioning data and the synchronous differential data to obtain POS data;

[0131] The 3D reconstruction module 800 is used to perform real-scene 3D reconstruction based on the original image data and POS data.

[0132] In one embodiment, the joint calculation module 600 is further configured to perform PPK joint calculation on the original positioning data and the synchronous differential data to obtain the initial positioning result; and to filter, smooth and optimize the initial positioning result to obtain POS data.

[0133] In one embodiment, the joint calculation module 600 is further used to extract the elevation component and latitude and longitude components from the initial positioning result; perform Kalman filtering smoothing and optimization on the elevation component; and after the Kalman filtering smoothing and optimization of the elevation component is completed, perform Kalman filtering smoothing and optimization on the latitude and longitude components to obtain POS data, which includes the processed elevation component and the processed latitude and longitude components.

[0134] In one embodiment, the 3D reconstruction module 800 is further configured to calculate the actual image overlap, POS precision spatial distribution, and texture richness based on image features of the original image data and POS data; predict the success probability of aerial triangulation based on the pre-trained machine learning model, the actual image overlap, the POS precision spatial distribution, and the texture richness based on image features; and perform real-scene 3D reconstruction based on the original image data and POS data if the predicted success probability is greater than a preset success probability threshold.

[0135] In one embodiment, the 3D reconstruction module 800 is further configured to, if the predicted success probability is not greater than a preset success probability threshold, analyze the weak areas corresponding to the aerial triangulation solution based on the actual image overlap, POS accuracy spatial distribution, and texture richness based on image features; generate a supplementary flight path based on the weak areas, and control the UAV to fly the supplementary flight path; and control the first parameter acquisition module 200 to re-execute the operation of acquiring the original image data and corresponding original positioning data collected during the UAV flight.

[0136] In one embodiment, the 3D reconstruction module 800 is further used to import the original image data and POS data into a preset 3D modeling software; to perform aerial triangulation in the preset 3D modeling software with POS data as a strong constraint to obtain the aerial triangulation result; and to perform density matching, mesh construction and texture mapping processing in sequence based on the aerial triangulation result to generate a real-scene 3D model.

[0137] In one embodiment, the first parameter acquisition module 200 is further configured to acquire a data boundary file of the target area; generate a photography flight path based on the data boundary file; and acquire the original image data and corresponding GNSS observation data collected by the UAV during the photography flight path.

[0138] The modules in the aforementioned UAV-based 3D modeling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0139] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for real-world 3D modeling of unmanned aerial vehicles (UAVs). The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0140] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described UAV real-scene 3D modeling method.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described UAV real-scene 3D modeling method.

[0143] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described UAV real-scene 3D modeling method.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for real-world 3D modeling of unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire raw image data and corresponding raw positioning data collected during the drone's flight; Obtain the synchronous differential data fed back from the base station; The original positioning data and the synchronous differential data are combined using PPK calculation to obtain POS data; Real-world 3D reconstruction is performed based on the original image data and the POS data.

2. The method according to claim 1, characterized in that, The step of performing PPK joint calculation on the original positioning data and the synchronization differential data to obtain POS data includes: The original positioning data and the synchronous differential data are combined using PPK calculation to obtain the initial positioning result; The initial positioning results are filtered, smoothed, and optimized to obtain the POS data.

3. The method according to claim 2, characterized in that, The process of filtering, smoothing, and optimizing the initial positioning result to obtain the POS data includes: Extract the elevation and latitude / longitude components from the initial positioning results; The elevation components are smoothed and optimized using Kalman filtering; Once the Kalman filtering smoothing and optimization of the elevation component is completed, the Kalman filtering smoothing and optimization of the latitude and longitude components are then performed to obtain the POS data. The POS data includes the processed elevation component and the processed latitude and longitude components.

4. The method according to claim 1, characterized in that, The real-scene 3D reconstruction based on the original image data and the POS data includes: Based on the original image data and the POS data, calculate the actual image overlap, POS precision spatial distribution, and texture richness based on image features of the original image data; Based on the pre-trained machine learning model, the actual image overlap, the spatial distribution of POS precision, and the richness of texture based on image features, the success probability of aerial triangulation is predicted. If the predicted success probability is greater than the preset success probability threshold, then real-scene 3D reconstruction is performed based on the original image data and the POS data.

5. The method according to claim 4, characterized in that, The method further includes: If the predicted success probability is not greater than the preset success probability threshold, then the weak areas corresponding to the aerial triangulation solution are analyzed based on the actual image overlap, the spatial distribution of POS accuracy, and the texture richness based on image features. Generate a replacement flight path based on the weak area, and control the UAV to fly the replacement flight path; Return to the steps of obtaining the raw image data and corresponding raw positioning data collected during the drone's flight.

6. The method according to claim 1, characterized in that, The real-scene 3D reconstruction based on the original image data and the POS data includes: Import the original image data and the POS data into the preset 3D modeling software; In the preset 3D modeling software, the POS data is used as a strong constraint to perform aerial triangulation, and the aerial triangulation result is obtained. Based on the aerial triangulation results, density matching, mesh construction, and texture mapping are performed sequentially to generate the real-world 3D model.

7. The method according to claim 1, characterized in that, The acquisition of raw image data and corresponding raw positioning data collected during the drone's flight includes: Obtain the data boundary file of the target area; Generate a photography flight path based on the data boundary file; Acquire the raw image data and corresponding GNSS observation data collected by the UAV flying the aforementioned photographic route.

8. A drone-based real-scene 3D modeling device, characterized in that, The device includes: The first parameter acquisition module is used to acquire the raw image data and corresponding raw positioning data collected during the flight of the UAV. The second parameter acquisition module is used to acquire the synchronous differential data fed back by the base station; The joint calculation module is used to perform PPK joint calculation on the original positioning data and the synchronization differential data to obtain POS data; The 3D reconstruction module is used to perform real-scene 3D reconstruction based on the original image data and the POS data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images