Full-automatic three-dimensional real scene modeling method based on unmanned aerial vehicle oblique photography technology

Abstract

The invention discloses a full-automatic three-dimensional live-action modeling method based on an unmanned aerial vehicle oblique photography technology, and particularly relates to the technical field of unmanned aerial vehicle photography, and the method comprises the following steps: firstly, dividing a modeling region into a plurality of unmanned aerial vehicle oblique photography regions, and presetting an initial flight route and a parameter combination; an initial image is collected according to a preset route, terrain and shelter analysis is carried out, and a region complexity level is output through fuzzy reasoning; determining whether the initial image is adopted or not according to the classification, and if not, re-collecting the image based on an optimization strategy; and finally, gathering and summarizing the adopted images, and inputting the collected images into three-dimensional modeling software to generate an accurate three-dimensional model. According to the method, the flight strategy of the unmanned aerial vehicle is dynamically optimized, and especially in complex terrains and shelter dense areas, the coverage degree and quality of image acquisition are ensured, so that the detail retention degree of a three-dimensional model is remarkably improved, resources of the unmanned aerial vehicle are more effectively utilized, unnecessary flight and data redundancy are reduced, and the modeling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle photography, and more specifically, to a fully automatic three-dimensional real scene modeling method based on unmanned aerial vehicle tilt photography technology. Background Art

[0002] In the field of drone technology and 3D real-scene modeling, with the widespread application of drone oblique photography technology, more and more fields have begun to rely on high-precision 3D models, such as terrain mapping, building modeling, urban planning, disaster assessment, etc. However, the existing 3D modeling technology still has some challenges and shortcomings when facing complex terrain and densely obstructed environments.

[0003] In areas with complex terrain or densely built-up areas, traditional drone photography methods cannot guarantee sufficient image overlap and coverage, resulting in reduced accuracy of 3D models in these areas. In areas with dense obstructions such as trees and buildings, image quality and coverage are severely affected, resulting in holes, distortion or discontinuities in the model.

[0004] In the existing 3D modeling process, there is a lack of dynamic optimization mechanism in the modeling data acquisition stage, which may result in the modeling data quality being insufficient to complete the expected 3D modeling task. Therefore, a fully automatic 3D real-scene modeling method based on UAV oblique photography technology is proposed here to solve the above problem. Summary of the invention

[0005] To achieve the above object, the present invention provides the following technical solutions: The fully automatic 3D real scene modeling method based on UAV tilt photography technology includes the following steps: Step 1: Divide the modeling area into a plurality of drone oblique photography areas, each drone oblique photography area being preset with an initial flight photography route and an initial flight photography parameter combination on the initial flight photography route; Step 2: performing image acquisition on the oblique photography area of ​​the target UAV according to a preset initial flight photography route and an initial flight photography parameter combination on the initial flight photography route to obtain an initial image acquisition set of the oblique photography area of ​​the target UAV; Step 3: Based on the initial image collection set of the target UAV oblique photography area, terrain analysis and occlusion analysis are performed, and then fuzzy reasoning is performed based on the results of the terrain analysis and occlusion analysis to output the classification category of the target UAV oblique photography area; Step 4: determine whether the initial image acquisition set is adopted and used according to the classification category of the target drone oblique photography area. If not, re-acquire images of the target drone oblique photography area according to a preset optimization strategy to obtain a re-acquired optimized image acquisition set for adoption and use; Step 5: All the initial image acquisition sets and optimized image acquisition sets adopted and used are summarized and input into the 3D modeling software to build the 3D model.

[0006] In a preferred embodiment, the initial flight photography parameter combination consists of the drone tilt angle, the drone flight speed, the drone shooting frequency, the drone shooting image resolution, and the drone route spacing.

[0007] In a preferred embodiment, dividing the modeling area into a plurality of drone oblique photography areas means: according to the endurance of the drone, dividing the modeling area into a plurality of sub-areas that can be covered in a single flight, namely, drone oblique photography areas.

[0008] In a preferred embodiment, performing terrain analysis refers to using a convolutional neural network model to output a terrain complexity index based on an initial image acquisition set.

[0009] In a preferred embodiment, the convolutional neural network model design and training logic are as follows: The input layer receives the preprocessed terrain image, and then uses multiple convolutional layers to extract the preset local feature indicators in the image. After the convolutional layer, a pooling layer is added. After the convolutional layer and the pooling layer, a fully connected layer is added to map the data corresponding to the local feature indicators into a feature vector of fixed length. The output layer generates the value of the terrain complexity index. The loss function is selected as the mean square error to measure the difference between the model prediction results and the actual annotations. The goal is to minimize the loss function. The training is completed when the model performance reaches the preset standard.

[0010] In a preferred embodiment, performing obstruction analysis refers to: Use Canny edge detection to identify the edges of occluders in the image. The formula is: ; is a Gaussian filter, is the input image, * indicates the convolution operation; Based on the results of edge detection, the region growing algorithm is used to identify the complete occlusion area. The region growing formula is: ; R is the growth area, p is the pixel point in the image, p0 is the initial seed point, D(p,p0) represents the distance metric between pixels, T is the preset threshold, and finally a marked image is obtained, the occluded area is marked as 1, and the non-occluded area is marked as 0; Then calculate the number of pixels in each occluded area and convert it into the actual area using the formula: ; Indicates the actual area size of each pixel. represents the total number of pixels in the occluder area number i, represents the actual area of ​​the obstruction with the obstruction area number i; Finally, the proportion of the area occupied by all occluders is calculated to obtain the occluder density index, and the formula is: ; represents the total area of ​​the target drone's oblique photography area, n is the total number of obstruction areas, Indicates the density index of occluders.

[0011] The distance metric between pixels is obtained by any of the following distance calculation methods: Euclidean distance used to measure the geometric distance between pixel values, absolute value of grayscale difference used to measure the brightness difference between pixels, Manhattan distance used to measure the axis-aligned distance between pixel values, and Mahalanobis distance used to measure the similarity between multidimensional data.

[0012] In a preferred embodiment, fuzzy reasoning is performed based on the results of terrain analysis and obstruction analysis, and the classification category of the target drone oblique photography area is outputted as follows: The terrain complexity index and obstruction density index of the target UAV oblique photography area are taken as input variables, and the classification category is taken as the output variable. The input variables are fuzzy processed and the values ​​of the input variables are converted into fuzzy sets. The output variables are fuzzy processed and the output variables are converted into fuzzy sets. Fuzzy rules are formulated to describe the category requirements under different data type combinations. Finally, the classification category of the target UAV oblique photography area is obtained as low-level, medium-level or high-level complex areas.

[0013] In a preferred embodiment, when the classification category of the target UAV oblique photography area is a low-level complex area, the initial image acquisition set is adopted and used; when the classification category of the target UAV oblique photography area is a medium-level complex area, the particle swarm optimization algorithm is used to optimize the UAV's flight strategy; when the classification category of the target UAV oblique photography area is a high-level complex area, the genetic algorithm is used to optimize the UAV's flight strategy.

[0014] Technical effects and advantages of the present invention: The present invention ensures the coverage and quality of image acquisition by dynamically optimizing the flight strategy and shooting parameters of the drone, especially in complex terrain and densely obstructed areas, thereby significantly improving the detail retention of the three-dimensional model. The present invention integrates the analysis of the terrain complexity index and the dense obstruction index, and combines fuzzy logic to intelligently divide the modeling area. It can automatically determine the complexity level of different areas and adjust the flight strategy accordingly, without manual intervention, to achieve a fully automated three-dimensional modeling process.

[0015] The present invention divides the modeling area according to the endurance of the UAV, and optimizes the flight strategy by using intelligent algorithms (such as particle swarm optimization algorithm and genetic algorithm), so as to maximize the use of the resources of the UAV, reduce unnecessary flights and data redundancy, and improve the modeling efficiency. The present invention can effectively cope with highly complex and highly obstructed environments, and ensure that high-quality image data can still be obtained in difficult modeling areas by performing detailed terrain and obstruction analysis before modeling, and generate accurate three-dimensional models. Since the present invention adopts fuzzy logic and intelligent algorithms, the modeling strategy can be flexibly adjusted according to different modeling requirements and environmental conditions. The method is not only suitable for general terrain modeling, but also can be extended to various scenarios such as buildings, ancient building protection, and urban planning. Through the automated and intelligent modeling method, the dependence on manual operation and experience judgment is reduced, thereby reducing labor costs and improving the repeatability and reliability of the overall modeling process. By performing real-time optimization and adjustment in the modeling data acquisition stage, it is ensured that the required image data can be efficiently obtained in each flight, the number and time of repeated flights are reduced, and the work efficiency is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1 This is a schematic diagram of the fully automatic three-dimensional real scene modeling method based on drone tilt photography technology in the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Reference Figure 1 The following embodiments are obtained: Embodiment 1: A fully automatic 3D real scene modeling method based on drone oblique photography technology comprises the following steps: Step 1. Divide the modeling area into multiple UAV oblique photography areas. Each UAV oblique photography area is preset with an initial flight photography route and an initial flight photography parameter combination on the initial flight photography route; Division based on UAV flight characteristics: Considering the UAV's endurance, the UAV's battery life limits the time of a single flight mission. Therefore, according to the UAV's endurance, the modeling area is divided into multiple sub-areas that can be covered in a single flight.

[0019] The initial flight photography parameter combination consists of the drone tilt angle, drone flight speed, drone shooting frequency, drone image resolution, and drone route spacing.

[0020] The tilt angle of a drone is the angle between the drone camera and the ground. Usually, the tilt angle is between 10° and 45°. Oblique photography can capture more side details, which is essential for generating high-precision 3D models. A smaller tilt angle helps capture more surface information, while a larger tilt angle helps capture the sides of buildings or other vertical structures.

[0021] Drone flight speed is the speed at which the drone moves in the air, usually measured in meters per second (m / s). Faster flight speeds may result in more blurry images; slower flight speeds can capture clearer images. Speed ​​also interacts with shooting frequency to determine the coverage density of the image.

[0022] The drone shooting frequency is the frequency at which the drone takes images during flight, usually measured in frames per second (frames / second) or per meter of flight distance. A higher shooting frequency ensures that continuous and overlapping images can be obtained even at high speeds, thereby improving the accuracy of the 3D model. A low shooting frequency may result in large gaps between images, affecting the integrity of the model.

[0023] The resolution of drone images is the resolution of the images taken by the drone camera, usually expressed in pixels (such as 4000x3000 pixels). Higher image resolution can capture more details and generate more sophisticated 3D models. However, higher resolution also means a larger amount of data, which may increase the complexity of data processing.

[0024] The UAV flight spacing is the horizontal spacing between UAVs on different routes, usually in meters (m). Smaller flight spacing can increase the overlap of images, ensuring that no area is missed, thereby improving the coverage and accuracy of the 3D model. Larger flight spacing may result in some areas not being fully photographed, affecting the integrity of the model. These parameters need to be optimized and combined according to specific regional modeling requirements when necessary.

[0025] Step 2: According to the preset initial flight photography route and the initial flight photography parameter combination on the initial flight photography route, image acquisition is performed on the oblique photography area of ​​the target UAV to obtain an initial image acquisition set of the oblique photography area of ​​the target UAV.

[0026] Determine the initial flight photography route: Plan the initial flight route of the drone, which should cover the entire target area and ensure that all important terrain features or buildings can be fully photographed along the way. It can be preset by experts. Set the route spacing: According to the preset route spacing, determine the specific position and altitude of the drone on different flight paths to ensure sufficient image overlap.

[0027] Set the initial flight photography parameter combination: Tilt angle: Set the tilt angle of the camera (such as 30°) according to the mission requirements to capture the side and bottom details of the target area. Flight speed: Select an appropriate flight speed (such as 5 meters / second) to ensure that the image is not blurred due to flying too fast during the acquisition process. Shooting frequency: Set the shooting frequency (such as 2 seconds / frame) so that the drone can continuously capture high-resolution images at a predetermined time interval during flight. Image resolution: Determine the camera resolution setting (such as 4000x3000 pixels) to ensure the clarity and details of the image. Route spacing: Ensure the spacing between routes (such as 20 meters) to meet the requirements of area coverage and image overlap.

[0028] Execute flight mission: According to the preset flight route and parameter combination, start the drone and start flying in the target area. During the flight, the drone automatically collects images according to the set parameter combination. The camera continuously captures images at the set tilt angle and shooting frequency, and records the location information of each image (such as GPS coordinates).

[0029] Step 3: Based on the initial image collection set of the target UAV oblique photography area, terrain analysis and occlusion analysis are performed, and then fuzzy reasoning is performed based on the results of the terrain analysis and occlusion analysis to output the classification category of the target UAV oblique photography area; Terrain analysis refers to using a convolutional neural network model to output a terrain complexity index based on the initial image collection set. The design and training logic of the convolutional neural network model are as follows: Preprocess the collected images, including denoising, white balance adjustment, color correction, etc., to ensure the quality and consistency of the images. Model design: Input layer: The input layer receives the preprocessed terrain image, usually an RGB three-channel image. Convolution layer: Use multiple convolution layers to extract local features in the image. The convolution layer can capture low-level and high-level features such as edges and textures in the image. Pooling layer: After the convolution layer, add a pooling layer (usually maximum pooling) to reduce the size of the feature map, reduce computational complexity, and retain key features. Fully connected layer: After the convolution layer and the pooling layer, add a fully connected layer to map the features into a fixed-length feature vector. Output layer: The output layer usually uses an activation function such as Softmax to generate a classification result of terrain complexity, and then converts the classification result of terrain complexity into a numerical value of the complexity index.

[0030] Softmax output: Assume that the output layer of the neural network has K neurons, each neuron corresponds to a category of terrain complexity (for example: low complexity, medium complexity, high complexity). The Softmax function converts the output of the network into a probability distribution, which represents the probability that the input image belongs to each complexity category. Usually, the classification result depends on the category with the highest probability in the Softmax output. For example, using Represents the probability output of the i-th category, if The highest, then the input image is classified as the second category (such as medium complexity), category correspondence table: Assume the category is: Category 1: low complexity Category 2: medium complexity; Category 3: high complexity; Assign a numerical index to each complexity category. For example: low complexity: index value is 1; medium complexity: index value is 2; high complexity: index value is 3; If the input image is classified into a certain complexity category, directly use the numerical index of the category as the final complexity index. For example, if an image is classified as "medium complexity", its complexity index is 2.

[0031] If the user wants to reflect the complexity more accurately, he can consider using the probability of Softmax output for weighted averaging. Even if the probability of a certain category is not the highest, it can still have an impact on the final index. ; is the preset weight coefficient of the ith category and is a positive value, m represents the total number of complexity categories, Represents the complexity index. For example, if the output of Softmax is: 0.2, 0.7, 0.1, FZ=(0.1×1)+(0.7×2)+(0.2×3)=0.1+1.4+0.6=2.1.

[0032] Depending on the method chosen, the complexity index of the final output may be an integer (if the classification result is used directly) or a decimal (if a weighted average is used).

[0033] Model training: Select a suitable loss function such as mean square error to measure the difference between the model prediction results and the actual annotations, train the model, adjust the network weights to minimize the loss function, use the validation set to monitor the performance of the model to prevent overfitting, and complete the training when the model performance reaches the preset standard.

[0034] Performing occlusion analysis means: Use Canny edge detection to identify the edges of occluders in the image. The formula is: ; is a Gaussian filter, is the input image, * indicates the convolution operation; the result of Canny edge detection is a binary image showing the edge of the object. The edge of the object is clearly marked as white (pixel value is 255), while the non-edge area is marked as black (pixel value is 0). This is because the purpose of Canny edge detection is to highlight the significant edges in the image for subsequent image analysis or processing.

[0035] Specific steps and results: Original image: Suppose we have a grayscale image or color image containing multiple objects, usually drone images of buildings, terrain or other structures. Grayscale processing: If the input is a color image, Canny edge detection usually converts the image to a grayscale image first, because edge detection mainly relies on brightness changes rather than color changes. Gaussian filtering: Before applying Canny edge detection, the image is Gaussian filtered to remove noise and make edge detection more stable. The filtered image becomes slightly blurry and less noisy. Gradient calculation: The Canny algorithm determines the areas in the image with the largest pixel brightness changes by calculating the gradient of the image. These areas are potential edge locations. Non-maximum suppression: By suppressing the pixel values ​​in non-edge areas, the edges are made thinner and more accurate. Dual threshold detection: Canny edge detection uses two thresholds, a high threshold and a low threshold. Pixels above the high threshold are considered strong edges, pixels below the low threshold are considered non-edges, and pixels in between are judged as edges based on their connectivity. Edge tracking: By connecting connected weak edge pixels to strong edges, a complete edge map is finally generated. In the output image of Canny edge detection: White pixels (255): represent the detected edge parts in the image. For example, the outline of a building, the edge of a tree, the boundary of a road, etc. Black pixels (0): represent the non-edge parts of the image, that is, the background area or the internal area of ​​an object. For example, if the input image is a photo of a building, the binary image after Canny edge detection will show the outline of the building and important structural lines (such as window frames, roof edges, etc.) as white, while the sky, the inside of the wall or the shadow part will be displayed as black.

[0036] Based on the results of edge detection, the region growing algorithm is used to identify the complete occlusion area. The region growing formula is: ; R is the growth area, p is the pixel point in the image, p0 is the initial seed point, D(p,p0) represents the distance metric between pixels, T is the preset threshold, and finally a marked image is obtained, the occluded area is marked as 1, and the non-occluded area is marked as 0; Then calculate the number of pixels in each occluded area and convert it into the actual area (square meters or square centimeters) using the formula: ; Indicates the actual area size of each pixel. represents the total number of pixels in the occluder area number i, represents the actual area of ​​the occluder with occluder area number i; finally, the proportion of all occluder areas is calculated to obtain the occluder density index, and the formula is: ; represents the total area of ​​the target drone's oblique photography area, n is the total number of obstruction areas, Represents the occlusion density index. The larger the occlusion density index, the more occlusions there are in the target area or the denser the distribution of occlusions in the area. This means that the line of sight in this area is highly obstructed, which may affect the image quality captured by the drone, resulting in more obstructed areas in the image, thus affecting the accuracy and integrity of the 3D model. Specifically, when the occlusion density index is large: the line of sight is more severely obstructed, and the flight altitude, angle or shooting frequency of the drone may need to be adjusted to minimize the impact of occlusion. More intensive shooting may be required, or multiple flights may be required to collect images from different angles to ensure that the obstructed parts are covered.

[0037] The distance metric between pixels is obtained by any of the following distance calculation methods: Euclidean distance for measuring the geometric distance between pixel values, absolute value of grayscale difference for measuring the brightness difference between pixels, Manhattan distance for measuring the axis-aligned distance between pixel values, and Mahalanobis distance for measuring the similarity between multidimensional data; Spatial structure: If the task involves the spatial relationship or geometric features of pixels, such as shape analysis or edge detection, Euclidean distance or Manhattan distance is preferred. Brightness or color: If the task focuses on brightness differences or color changes, such as thresholding or region growing, absolute value of grayscale difference is a suitable choice. Multidimensional features: If the task involves comparison or clustering analysis between multidimensional features, such as color space processing or multidimensional feature classification, choose Mahalanobis distance. In specific applications, choosing an appropriate distance calculation method can effectively improve the accuracy and efficiency of image processing tasks.

[0038] Step 4: Determine whether the initial image acquisition set is adopted and used according to the classification category of the target drone's oblique photography area. If not, re-acquire images of the target drone's oblique photography area according to a preset optimization strategy to obtain the re-acquired optimized image acquisition set for adoption and use.

[0039] The terrain complexity index and obstruction density index of the target UAV oblique photography area are taken as input variables, and the classification category is taken as the output variable. The input variables are fuzzy processed and the values ​​of the input variables are converted into fuzzy sets. The output variables are fuzzy processed and the output variables are converted into fuzzy sets. Fuzzy rules are formulated to describe the category requirements under different data type combinations. Finally, the classification category of the target UAV oblique photography area is obtained as low-level, medium-level or high-level complex areas.

[0040] Determine fuzzy sets: Define the terrain complexity index and the occluder density index as fuzzy sets respectively. The terrain complexity index can be divided into three fuzzy sets of low, medium and high, and the occluder density index can also be divided into three fuzzy sets of low, medium and high.

[0041] Membership function definition: A membership function is defined for each fuzzy set. The membership function is used to indicate the degree to which an input variable belongs to a fuzzy set. The membership function can take common forms such as triangular function, trapezoidal function or Gaussian function. The specific shape of the membership function depends on the actual data distribution and requirements.

[0042] Fuzzy processing: Input the actual terrain complexity index and obstruction density index into the membership function and calculate its membership, that is, the membership of the input variable in the three fuzzy sets of low, medium and high. For example, a terrain complexity index may have a membership of 0.2 for low complexity, 0.5 for medium complexity and 0.3 for high complexity.

[0043] Determine the output fuzzy set: divide the output variables into three fuzzy sets of low, medium and high levels, corresponding to different levels of regional complexity.

[0044] Membership function definition: A membership function is defined for each fuzzy set of the output variable to represent the regional divisions of different complexity levels.

[0045] Output fuzzification processing: Based on the results of fuzzy rule reasoning, calculate the membership degree of the output variable to low, medium or high level.

[0046] Establish a fuzzy rule base: According to the combination of terrain complexity index and occluder density index, formulate several fuzzy rules. Each rule describes the corresponding output complexity level under different input variable combinations. For example: if the terrain complexity index is low and the occluder density index is low, the output is a low-level complex area. If the terrain complexity index is medium and the occluder density index is high, the output is a medium-level complex area. If the terrain complexity index is high and the occluder density index is high, the output is a high-level complex area.

[0047] Fuzzy reasoning: Based on the membership of the input variables and the fuzzy rules, the output membership of each rule is calculated through fuzzy reasoning mechanisms such as the minimum-maximum method or the center method. Multiple rules may have an impact on the same output fuzzy set, and these impacts need to be integrated through fuzzy reasoning methods.

[0048] The output variable membership obtained by fuzzy reasoning is defuzzified and converted into a specific output value. The defuzzification method can use the centroid method or the maximum membership method. According to the defuzzified output value, the complexity level of the target UAV oblique photography area is determined as a low, medium or high complexity area.

[0049] Step 5: All the initial image acquisition sets and optimized image acquisition sets adopted and used are summarized and input into the 3D modeling software to build the 3D model.

[0050] When the classification category of the target UAV oblique photography area is a low-level complex area, the initial image acquisition set is adopted and used. When the classification category of the target UAV oblique photography area is a medium-level complex area, the particle swarm optimization algorithm is used to optimize the UAV's flight strategy. When the classification category of the target UAV oblique photography area is a high-level complex area, the genetic algorithm is used to optimize the UAV's flight strategy.

[0051] Particle swarm optimization (PSO) is suitable for the area of ​​intermediate complexity because of its small computational complexity and simple implementation.

[0052] Particle swarm optimization algorithm usage steps: Initialize particles: Use the flight strategy parameters of the drone as the "position" of the particles and initialize multiple particles (each particle represents a flight strategy).

[0053] Fitness evaluation: Calculate the fitness of each particle. The fitness function can be indicators such as model quality and coverage.

[0054] Update speed and position: Update the speed and position of each particle based on the particle's own experience (individual optimal position) and the group's experience (global optimal position). The update formula is as follows: ; represents the velocity of particle i at time t, represents the position of particle i at time t, w is the inertia weight, , is the acceleration constant, , is a random number, represents the historical optimal position of the particle, represents the global optimal position.

[0055] Iterative optimization: Repeat the fitness evaluation and update steps until the number of iterations or optimization conditions are met.

[0056] Output the optimal strategy: Finally, the particle with the highest fitness is selected as the flight strategy of the drone.

[0057] Genetic algorithms are particularly suitable for complex, multi-dimensional optimization problems. In the optimization of UAV flight strategy, genetic algorithms can help determine the best flight path, shooting angle and frequency, thereby improving the modeling accuracy of high-complexity areas. The following are the steps for genetic algorithms to adjust flight strategies: Initialize the population: Define individuals and populations: First, encode the UAV's flight strategy (including the flight route and the combination of flight photography parameters along the flight route) as an "individual" in the genetic algorithm, and multiple individuals form a "population". During initialization, randomly generate multiple different flight strategies as the initial population. Set the objective function: The objective function can be set as an indicator of modeling quality, such as modeling detail retention. The detail retention measures the degree to which the details of the original terrain or objects in the three-dimensional model are retained. This is usually related to image resolution, shooting angle, and shooting frequency. The objective function will be used to evaluate the pros and cons of each individual. The objective function formula is: ; D is the modeling detail retention, M is the number of detail areas, is the weight of the jth detail region (based on importance), is the detail retention score of the jth detail area. Before modeling, the UAV performs preliminary data collection to evaluate the detail areas to estimate the modeling requirements of these areas. Experts can evaluate the preliminary images or data of these detail areas according to actual needs and give detail retention scores . In 3D modeling, detail areas refer to areas that require focused attention and high-precision modeling. These areas usually include parts with complex structures, important features, or subtle changes. Building modeling: Detail areas may include fine structures such as building edges, windows, decorative elements, door frames, etc. Terrain modeling: Detail areas may include areas of sharp changes in terrain, such as ridges, river valleys, cliffs, or areas with dense vegetation. Drone oblique photography area: In drone oblique photography, detail areas are usually those parts that require high-resolution shooting and more perspective coverage to ensure that the 3D model can accurately reflect the actual situation. Therefore, the multiple drone oblique photography areas obtained by dividing the detail area and the modeling area may be different. Considering the drone's endurance, the drone's battery life limits the time of a single flight mission. Therefore, according to the drone's endurance, the modeling area is divided into multiple sub-areas that can be covered in a single flight, namely the drone oblique photography area.

[0058] Fitness evaluation: Calculate fitness: For each individual in the population (i.e. each flight strategy), calculate its fitness according to the objective function. Individuals with high fitness indicate that the flight strategy is more likely to generate high-quality 3D models. Select excellent individuals: According to the fitness value, select the best performing individuals from the population as the "parents" of the next generation. Common selection methods include roulette selection, tournament selection, etc.

[0059] Genetic operations: Crossover: Perform a crossover operation on the selected parent individuals to generate the next generation of individuals. The crossover operation generates new individuals by mixing the genes (parameters of the flight strategy) of the two parent individuals. For example, some genes of the flight path and shooting angle in the parent individuals can be exchanged. Mutation: Perform random mutation on the newly generated individuals, randomly changing certain gene values ​​(such as changing the flight angle) with a certain probability to increase the diversity of the population and avoid falling into the local optimal solution.

[0060] Generate a new generation of population: Update the population: Replace part or all of the old population with newly generated individuals to form a new generation of population. This process includes retaining individuals with high fitness directly to the next generation (elite retention strategy) to ensure the continuity of the optimization process.

[0061] Repeated iteration: Iterative evolution: Repeat the fitness evaluation, selection, crossover, mutation and other steps. After multiple generations of iterations, the individuals in the population gradually tend to the optimal flight strategy. Each iteration will produce a better solution until the set number of iterations is reached or the termination condition is met (such as fitness no longer significantly improved).

[0062] Output optimization results: Determine the optimal flight strategy: When the algorithm converges, select the individual with the highest fitness as the final optimized UAV flight strategy. Apply this strategy to actual flights in highly complex modeling areas to obtain high-quality image data.

[0063] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0064] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0065] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0067] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fully automatic three-dimensional real scene modeling method based on drone oblique photography technology, characterized in that: The following steps are involved: Step 1: Divide the modeling area into a plurality of drone oblique photography areas, each drone oblique photography area being preset with an initial flight photography route and an initial flight photography parameter combination on the initial flight photography route; Step 2: performing image acquisition on the oblique photography area of ​​the target UAV according to a preset initial flight photography route and an initial flight photography parameter combination on the initial flight photography route to obtain an initial image acquisition set of the oblique photography area of ​​the target UAV; Step 3: Based on the initial image collection set of the target UAV oblique photography area, terrain analysis and occlusion analysis are performed, and then fuzzy reasoning is performed based on the results of the terrain analysis and occlusion analysis to output the classification category of the target UAV oblique photography area; Step 4: determine whether the initial image acquisition set is adopted and used according to the classification category of the target drone oblique photography area. If not, re-acquire images of the target drone oblique photography area according to a preset optimization strategy to obtain a re-acquired optimized image acquisition set for adoption and use; Step 5: All the initial image acquisition sets and optimized image acquisition sets adopted and used are summarized and input into the 3D modeling software to build the 3D model.

2. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 1 is characterized in that: The initial flight photography parameter combination consists of the drone tilt angle, drone flight speed, drone shooting frequency, drone image resolution, and drone route spacing.

3. The fully automatic 3D real scene modeling method based on drone oblique photography technology according to claim 2 is characterized in that: Dividing the modeling area into a plurality of drone oblique photography areas means: dividing the modeling area into a plurality of sub-areas, namely, drone oblique photography areas, which can be covered in a single flight according to the endurance of the drone.

4. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 3 is characterized in that: Performing terrain analysis refers to using a convolutional neural network model to output a terrain complexity index based on an initial image collection set.

5. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 4 is characterized in that: The convolutional neural network model design and training logic are as follows: The input layer receives the preprocessed terrain image, and then uses multiple convolutional layers to extract the preset local feature indicators in the image. After the convolutional layer, a pooling layer is added. After the convolutional layer and the pooling layer, a fully connected layer is added to map the data corresponding to the local feature indicators into a feature vector of fixed length. The output layer generates the value of the terrain complexity index. The loss function is selected as the mean square error to measure the difference between the model prediction results and the actual annotations. The goal is to minimize the loss function. The training is completed when the model performance reaches the preset standard.

6. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 5 is characterized in that: Performing occlusion analysis means: Use Canny edge detection to identify the edges of occluders in the image. The formula is: ; is a Gaussian filter, is the input image, * indicates the convolution operation; Based on the results of edge detection, the region growing algorithm is used to identify the complete occlusion area. The region growing formula is: ; R is the growth area, p is the pixel point in the image, p0 is the initial seed point, D(p,p0) represents the distance metric between pixels, T is the preset threshold, and finally a marked image is obtained, the occluded area is marked as 1, and the non-occluded area is marked as 0; Then calculate the number of pixels in each occluded area and convert it into the actual area using the formula: ; Indicates the actual area size of each pixel. represents the total number of pixels in the occluder area number i, represents the actual area of ​​the obstruction with the obstruction area number i; Finally, the proportion of the area occupied by all obstructions is calculated to obtain the obstruction density index, the formula is: ; represents the total area of ​​the target drone's oblique photography area, n is the total number of obstruction areas, Indicates the density index of occluders.

7. The fully automatic 3D real scene modeling method based on drone oblique photography technology according to claim 6 is characterized in that: The distance metric between pixels is obtained by any of the following distance calculation methods: Euclidean distance used to measure the geometric distance between pixel values, absolute value of grayscale difference used to measure the brightness difference between pixels, Manhattan distance used to measure the axis-aligned distance between pixel values, and Mahalanobis distance used to measure the similarity between multidimensional data.

8. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 7 is characterized in that: Based on the results of terrain analysis and occlusion analysis, fuzzy reasoning is performed to output the classification categories of the target drone oblique photography area: The terrain complexity index and obstruction density index of the target UAV oblique photography area are taken as input variables, and the classification category is taken as the output variable. The input variables are fuzzy processed and the values ​​of the input variables are converted into fuzzy sets. The output variables are fuzzy processed and the output variables are converted into fuzzy sets. Fuzzy rules are formulated to describe the category requirements under different data type combinations. Finally, the classification category of the target UAV oblique photography area is obtained as low-level, medium-level or high-level complex areas.

9. The fully automatic 3D real scene modeling method based on drone tilt photography technology according to claim 8 is characterized in that: When the classification category of the target UAV oblique photography area is a low-level complex area, the initial image acquisition set is adopted and used. When the classification category of the target UAV oblique photography area is a medium-level complex area, the particle swarm optimization algorithm is used to optimize the UAV's flight strategy. When the classification category of the target UAV oblique photography area is a high-level complex area, the genetic algorithm is used to optimize the UAV's flight strategy.

Images