Unmanned aerial vehicle based scene simulation method, system and computer program product

Abstract

The present disclosure provides a kind of scene simulation method, system and computer program product based on unmanned aerial vehicle, it is related to unmanned aerial vehicle scene simulation technical field, wherein scene simulation method includes: based on unmanned aerial vehicle, the shooting of the scene to be simulated is carried out, and the image set of the scene to be simulated is obtained;Based on the image set, the sparse point cloud set of the scene to be simulated is determined;Based on the image set, the identification label set of at least part of the shooting object in the image set is determined;Based on the image set, the sparse point cloud set and the identification label set, 3DGS algorithm is used to carry out three-dimensional reconstruction to the scene to be simulated, to generate the three-dimensional simulation scene corresponding to the scene to be simulated.

Description

Technical Field

[0001] This specification relates to the field of unmanned aerial vehicle (UAV) scene simulation technology, and in particular to a UAV-based scene simulation method, system, and computer program product. Background Technology

[0002] In drone testing scenarios, before conducting real-world flight tests, the drone's flight path planning, obstacle avoidance, and object tracking and recognition capabilities can be tested in a virtual environment. To further improve the reliability of virtual environment testing, a scenario simulation system can be used to reconstruct a 3D simulation of a real-world scenario, allowing the drone to be tested in a virtual 3D simulation environment. This effectively reflects the problems that the drone may encounter in a real-world scenario and allows for timely improvements, resulting in good testing outcomes.

[0003] However, traditional scene simulation systems often require the creation of a large number of digital assets, such as using polygonal meshes for 3D scene representation, and require a lot of manual post-editing work, making the 3D simulation reconstruction of real-world scenes costly in terms of manpower, resources, and time.

[0004] In view of this, some embodiments of this specification provide a scene simulation method, system, and computer program product based on unmanned aerial vehicles (UAVs), which can improve the production efficiency of scene 3D reconstruction. Summary of the Invention

[0005] This specification provides one or more embodiments of a scene simulation based on a drone. The method includes: acquiring a set of captured images of the scene to be simulated based on drone photography of the scene; determining a sparse point cloud set of the scene to be simulated based on the captured image set; determining a set of identification tags for at least some of the captured objects in the captured image set based on the captured image set; and using a 3DGS algorithm to perform three-dimensional reconstruction of the scene to be simulated based on the captured image set, the sparse point cloud set, and the identification tag set, so as to generate a three-dimensional simulation scene corresponding to the scene to be simulated.

[0006] In some embodiments, the drone's flight path for photographing the simulated scene includes at least two drone movement paths at different flight altitudes that circle and photograph the simulated scene.

[0007] In some embodiments, acquiring a set of images of the scene to be simulated based on the drone's shooting of the scene to be simulated includes: planning the drone's shooting flight trajectory according to the scene to be simulated; controlling the drone to fly according to the shooting flight trajectory and acquiring images during the flight, wherein there is at least a partial overlapping area between adjacent images.

[0008] In some embodiments, based on the drone's capture of the scene to be simulated, a set of captured images of the scene to be simulated is obtained, including: acquiring a first captured image captured by the drone; determining the blur level of the first captured image; when the blur level of the first captured image deviates from a preset range, acquiring the shooting position information corresponding to the first captured image; and controlling the drone to recapture the first captured image according to the shooting position information.

[0009] In some embodiments, determining the blur level of a first captured image includes: acquiring sampled image features of the first captured image; and determining the blur level of the first captured image based on the sampled image features and a pre-trained shallow convolutional neural network.

[0010] In some embodiments, extracting sampled image features from the first captured image includes: constructing a Laplacian pyramid corresponding to the first captured image; sampling image blocks of a preset size at intervals in each layer of the Laplacian pyramid as sampled blocks corresponding to the first captured image; stitching all sampled blocks together to obtain a feature map corresponding to the first captured image, and using the Laplacian pyramid features corresponding to the feature map as sampled image features.

[0011] In some embodiments, based on the drone's capture of the scene to be simulated, a set of captured images of the scene to be simulated is obtained, including: acquiring a second captured image taken by the drone; identifying moving objects contained in the second captured image; performing masking processing on the portion of the second captured image containing moving objects to obtain an optimized captured image that ignores the moving objects; the set of captured images includes the optimized captured image.

[0012] In some embodiments, identifying moving objects contained in the second captured image includes: obtaining depth information corresponding to multiple objects based on the depth map corresponding to the second captured image; determining motion vectors corresponding to multiple objects based on the second captured image and captured images at adjacent time points, and determining the average value of motion vectors of objects with the same depth information; and identifying objects whose motion vectors deviate from the average value of motion vectors by a preset amount as moving objects.

[0013] In some embodiments, determining a sparse point cloud set of the scene to be simulated based on a set of captured images includes: obtaining feature point information of captured images in the set of captured images; obtaining shooting location information corresponding to the captured images based on the feature point information; and constructing a sparse point cloud set based on the shooting location information and the feature point information.

[0014] In some embodiments, determining a set of identification tags for at least a portion of the captured objects in the captured image set based on the captured image set includes: identifying at least a portion of the captured objects contained in the captured images and labeling the captured objects with identification tags; performing a standardization process on the identification tags corresponding to the captured objects in the captured image set to unify the identification tags corresponding to the same captured objects contained in different captured images; and using the captured image portion corresponding to the captured object and the identification tags as a set of identification tags.

[0015] This specification also provides one or more embodiments of a scene simulation system based on a drone. The system includes: a scene acquisition module for acquiring a set of captured images of the scene to be simulated based on drone photography of the scene; a sparse reconstruction module for determining a sparse point cloud set of the scene to be simulated based on the captured image set; an object recognition module for determining a set of recognition tags for at least some of the captured objects in the captured image set; and a reconstruction simulation module for performing three-dimensional reconstruction of the scene to be simulated using a 3DGS algorithm based on the captured image set, the sparse point cloud set, and the recognition tag set, to generate a three-dimensional simulation scene corresponding to the scene to be simulated.

[0016] Some embodiments of this specification also provide a computer program product, including computer instructions or a computer program, which, when at least a portion of the computer instructions or computer program is executed by a processor, can implement the UAV-based scene simulation method provided in any embodiment of this specification. Attached Figure Description

[0017] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0018] Figure 1 This is a schematic diagram of a scene simulation system according to some embodiments of this specification.

[0019] Figure 2 This is an exemplary flowchart illustrating a scene simulation method based on a drone, according to some embodiments of this specification.

[0020] Figure 3 This is an exemplary flowchart illustrating a method for determining a sparse point cloud set according to some embodiments of this specification.

[0021] Figure 4 This is an exemplary flowchart illustrating a set of identification tags according to some embodiments of this specification.

[0022] Figure 5 This is an exemplary rendering of a three-dimensional simulation scene according to some embodiments of this specification.

[0023] Figure 6 This is an exemplary mesh occupancy map of a three-dimensional simulation scene according to some embodiments of this specification.

[0024] Figure 7 This is an exemplary depth map of a three-dimensional simulation scene according to some embodiments of this specification.

[0025] Figure 8 This is a scene diagram illustrating a drone taking pictures of a simulated scene according to some embodiments of this specification.

[0026] Figure 9 This is an exemplary flowchart illustrating how a drone can film a simulated scene according to some embodiments of this specification.

[0027] Figure 10 This is an exemplary flowchart illustrating an acquisition of a set of captured images according to some embodiments of this specification.

[0028] Figure 11 This is an exemplary flowchart illustrating another method for acquiring a set of captured images, according to some embodiments of this specification.

[0029] Figure 12 This is an exemplary flowchart illustrating the identification of a moving object according to some embodiments of this specification.

[0030] Figure 13 This is a schematic diagram of the functional modules of a drone-based scene simulation system according to some embodiments of this specification. Detailed Implementation

[0031] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0032] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0033] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0034] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Such adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0035] Figure 1 This is a schematic diagram of a scene simulation system according to some embodiments of this specification. For example... Figure 1 As shown, the scene simulation system 100 may include a data acquisition device 110, a processing device 120, a network 130, a storage medium 140, and a terminal device 150, wherein the data acquisition device 110, the processing device 120, and the terminal device 150 can transmit data through the network 130.

[0036] In some embodiments, the acquisition device 110 can be a device capable of acquiring information from a simulated scene to obtain the necessary data for the 3D simulation reconstruction of the simulated scene. In some embodiments, the acquisition device 110 can be a scanning device, such as a LiDAR scanner, which obtains the 3D coordinates of each point in the simulated scene by measuring the reflection time after a laser beam is irradiated onto the surface of an object in the simulated scene. In some embodiments, the acquisition device 110 can be an image capturing device, such as a drone equipped with a capturing device, which captures a set of omnidirectional images of the simulated scene by flying around it. In some embodiments, the acquisition device 110 can be a drone capable of both image capturing and laser scanning.

[0037] In some embodiments, the processing device 120 may be a computer device with high computing performance, used to perform 3D reconstruction of the scene to be simulated based on the acquired images of the scene to be simulated provided by the acquisition device 110 and related user instructions. In some embodiments, the processing device 120 may be a single computer device or a computing cluster composed of multiple computer devices, thereby providing powerful computing power and efficient response for 3D scene simulation reconstruction. In some embodiments, the processing device 120 may be a server, which may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Distribution Network), and big data and artificial intelligence platforms.

[0038] In some embodiments, network 130 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 130 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, etc. Network 130 can have multiple access points, through which acquisition device 110, processing device 120, and terminal device 150 can access network 130.

[0039] In some embodiments, storage medium 140 can be used to store data and / or instructions related to the scene simulation system 100. In some embodiments, storage medium 140 can store raw captured images obtained from acquisition device 110, scene 3D reconstruction models obtained from processing device 120, 3D simulation reconstruction instructions obtained from terminal device 150, etc. In some embodiments, storage medium 140 may include one or more storage components, each of which may be an independent device or part of other devices. In some embodiments, storage medium 140 may be disposed in processing device 120. In some embodiments, storage medium 140 may include random access memory (RAM), read-only memory (ROM), mass storage, removable memory, volatile read-write memory, etc., or any combination thereof. Exemplarily, mass storage may include disks, optical disks, solid-state drives, etc. In some embodiments, storage medium 140 may be implemented on a cloud platform.

[0040] In some embodiments, the terminal device 150 may include, but is not limited to, desktop computers, smartphones, laptops, VR (Virtual Reality) devices, and tablets. The terminal device 150 may be a scene simulation application client, a browser client, or an instant messaging client. Users can install and run the application corresponding to the scene simulation system on the terminal device 150, display scene simulation results through the terminal device 150, and input drone virtual scene flight test commands through a human-machine interface device, etc. The human-machine interface device may be part of the terminal device 150, or it may be separate from the terminal device 150 but connected to it by a signal. As an example, the terminal device 150 may be connected to the processing device 120 via a network 130, allowing users to send 3D reconstruction commands of the scene to be simulated to the processing device 120 and obtain the corresponding 3D reconstruction results from the processing device 120.

[0041] It should be noted that, Figure 1 The scene simulation system 100 shown is merely an example. The scene simulation system and scenarios described in the embodiments of this specification are for the purpose of more clearly illustrating the technical solutions of the embodiments of this specification, and do not constitute a limitation on the technical solutions provided in the embodiments of this specification. Those skilled in the art will recognize that with the continuous development and evolution of various device forms in the field of reconstruction simulation technology, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems. For example, in some application scenarios, the processing device 120, storage medium 140, and terminal device 150 can be integrated into one unit; similarly, in other application scenarios, the processing device 120 and terminal device 150 can be integrated into one unit.

[0042] In UAV testing scenarios, before conducting real-world flight tests, the UAV's flight path planning and obstacle avoidance capabilities can be tested in a virtual environment. To further improve the reliability of virtual environment testing, a scene simulation system can be used to reconstruct a 3D simulation of the real-world scene, allowing the UAV to be tested in a virtual 3D simulation environment. This effectively reflects potential problems the UAV might encounter in the real-world scenario and allows for timely improvements, resulting in good testing outcomes. However, traditional scene simulation systems often require the creation of large amounts of digital assets, such as using polygonal meshes for 3D scene representation, and necessitate extensive manual post-editing work, making the 3D reconstruction of real-world scenes costly in terms of manpower, resources, and time. To overcome these problems, this specification provides a UAV-based scene simulation method, system, and computer program product. Specifically, Figure 2 This is an exemplary flowchart illustrating a drone-based scene simulation method according to some embodiments of this specification. In some embodiments, Figure 2The process 200 shown can be executed by the processing device 120. In some embodiments, the process 200 can be implemented by a drone-based scene simulation system 1300 deployed on the processing device 120.

[0043] In some embodiments, such as Figure 2 As shown, process 200 may include the following steps.

[0044] Step 210: Based on the drone's capture of the scene to be simulated, acquire a set of captured images of the scene to be simulated. In some embodiments, step 210 can be implemented by the scene acquisition module 1310.

[0045] In some embodiments, by using a drone to capture images of the simulated scene from all angles, various parameter information of the simulated scene can be obtained from the captured images, such as the appearance structure, size data, and spatial orientation of the scene components. This parameter information can then be used to achieve 3D reconstruction of the simulated scene. Compared to traditional scene simulation methods that rely on manual reconstruction based on mesh rendering, using drones to capture images of the simulated scene allows for more efficient and convenient data acquisition and reconstruction. The specific methods for obtaining the captured image set will be explained in detail later and will not be elaborated upon here.

[0046] Step 220: Based on the set of captured images, determine the sparse point cloud set of the scene to be simulated. In some embodiments, step 220 can be implemented by the sparse reconstruction module 1320.

[0047] Figure 3 This is an exemplary flowchart illustrating how to determine a sparse point cloud set according to some embodiments of this specification. In some embodiments, Figure 3 The illustrated process 300 can be executed by the processing device 120. In some embodiments, process 300 can be implemented by a scene simulation system 1300 (such as a sparse reconstruction module 1320) deployed on the processing device 120. In some embodiments, such as Figure 3 As shown, process 300 may include the following steps.

[0048] Step 310: Obtain feature point information of the captured images in the captured image set.

[0049] In some embodiments, feature point information refers to salient and stable feature points identified and extracted from captured images. These feature points are typically regions with high contrast or unique texture in the captured image, maintaining consistency under different viewing angles and lighting conditions, thus providing a foundation for subsequent feature matching and 3D reconstruction. In some embodiments, the SURF (Speeded Up Robust Features) algorithm can be used to acquire feature point information. By utilizing the Hessian matrix, feature point detection and recognition can be achieved quickly, making it suitable for real-time feature point recognition applications. In some embodiments, the SIFT (Scale-Invariant Feature Transform) algorithm can also be used to acquire feature point information. This algorithm constructs a scale space by performing multiple blurring and scaling operations on the image, and achieves feature point detection and recognition by differencing images at different scales in the scale space. Those skilled in the art can choose appropriate feature point information acquisition methods according to actual needs, and no limitations are imposed here.

[0050] Step 320: Based on feature point information, obtain the shooting location information corresponding to the captured image.

[0051] In some embodiments, feature points in different captured images can be compared to find matching feature point pairs, and the matching feature point pairs can be used to obtain the shooting location information of the captured images. The shooting location information may include the location of the drone when the image was captured, and the shooting angle of the shooting device carried by the drone. Those skilled in the art can select appropriate algorithms to obtain the shooting location information according to actual needs, and no limitation is made here.

[0052] Step 330: Construct a sparse point cloud set based on the shooting location information and feature point information.

[0053] In some embodiments, a sparse point cloud can be generated through triangulation based on the shooting location information and feature point information to ensure that each 3D point in the sparse point cloud set corresponds to a pair of matching feature points, thereby effectively representing the structure of the scene to be simulated in 3D space. In some embodiments, the above process 300 can be implemented using the COLMAP (Construction of Large Map from Photographs) tool. COLMAP is a tool that integrates Structure-from-Motion (SfM) and Multi-View Stereo (MVS) technologies, capable of calculating camera pose and constructing a high-precision 3D point cloud from a set of multi-view images. Those skilled in the art can also choose other suitable algorithms or tools to obtain the sparse point cloud set, which is not limited here.

[0054] In some embodiments, considering that the scene to be simulated can be an open outdoor scene, such as an outdoor plaza scene or a park scene, in order to avoid the difficulty in obtaining sparse point clouds of open areas such as the sky in the scene to be simulated, the method of initializing sky point clouds can be added during the construction of sparse point cloud set, which can more accurately represent the spatial relationship between open areas such as the sky in the 3D reconstructed scene and the scene to be simulated.

[0055] Step 230: Based on the set of captured images, determine a set of identification tags for at least a portion of the captured objects in the set of captured images. In some embodiments, step 230 can be implemented by the object recognition module 1330. In some embodiments, step 230 and step 220 can be executed synchronously or asynchronously, and the order of execution of steps 220 and 230 is not limited when executed asynchronously.

[0056] In some embodiments, the subject of the photograph can be the main content presented in the captured image, which may include constituent objects of the scene to be simulated, such as landscape pillars, pedestrian walkways, and planted trees in the outdoor plaza scene to be simulated, as well as other unrelated objects appearing in the scene to be simulated, such as pedestrians walking in the outdoor plaza scene to be simulated and vehicles driving around, etc., without limitation. In some embodiments, semantic segmentation algorithms can be used to perform semantic recognition on at least some of the subjects in the captured image, and the subjects can be labeled in the form of semantic tags, so as to assist the algorithm model in understanding the object information represented by each subject in the subsequent 3D reconstruction process, and to assign the corresponding object information to the corresponding reconstructed components in the reconstructed 3D simulation scene.

[0057] Figure 4This is an exemplary flowchart illustrating a method for determining a set of identification tags according to some embodiments of this specification. In some embodiments, Figure 4 The illustrated process 400 can be executed by the processing device 120. In some embodiments, process 400 can be implemented by a scene simulation system 1300 (such as an object recognition module 1330) deployed on the processing device 120. In some embodiments, such as Figure 4 As shown, process 400 may include the following steps.

[0058] Step 410: Identify at least some of the subjects in the captured image and label the subjects with identification tags.

[0059] In some embodiments, a pre-trained object prediction network can be used to acquire objects in a captured image. Specifically, this may include acquiring the location and pixel range of at least some objects in the captured image. In some embodiments, during the process of identifying and labeling the captured objects, the SAM2 (SegmentAnything Model 2) model can be used for semantic segmentation and semantic recognition. This model can utilize a self-attention mechanism to capture the relationships between different regions in the image. Those skilled in the art can also choose appropriate semantic segmentation and recognition labeling models or algorithms according to actual needs, which are not limited here.

[0060] Step 420: Standardize the identification labels corresponding to the captured objects in the captured image set to unify the identification labels corresponding to the same captured objects contained in different captured images.

[0061] In some embodiments, considering that during the process of labeling different images in a set of captured images, the same subject may be assigned different identification labels, in order to overcome this problem, the identification labels corresponding to the subject can be standardized based on a pre-trained deep neural network model, so that the same subject in each image in the set of captured images corresponds to the same identification label, maintaining the consistency of classification for subjects appearing in a sequence of moving images.

[0062] Step 430: Collect the captured image portion corresponding to the captured object and the recognition label as a recognition label set. Based on the above process 400, semantic recognition and labeling of at least some of the captured objects in the captured image can be achieved, thereby assisting the algorithm model in understanding the object information represented by each captured object in the subsequent 3D reconstruction process.

[0063] Step 240: Based on the set of captured images, the set of sparse point clouds, and the set of recognized labels, the 3DGS algorithm is used to reconstruct the three-dimensional scene to be simulated, so as to generate a three-dimensional simulation scene corresponding to the scene to be simulated. In some embodiments, step 240 can be implemented by the reconstruction simulation module 1340.

[0064] In some embodiments, the 3DGS (3D Gaussian Splatting) algorithm is a 3D scene representation and rendering method based on Gaussian distribution. It can be used for efficient modeling and rendering of complex 3D scenes, and is particularly suitable for applications requiring real-time rendering. In some embodiments, the primitives constituting the 3D scene in the 3DGS algorithm are Gaussian ellipsoids. Each Gaussian ellipsoid contains information such as color, its position in 3D space, normals, and semantic recognition. This enables the replication of the scene to be simulated in a virtual 3D space, allowing the obtained 3D simulation scene to be used for simulated flight testing of UAVs in virtual space.

[0065] In some embodiments, the 3D simulation scene reconstructed based on the 3DGS algorithm has the ability to render in real time, enabling the simulation visualization of the UAV when it is being tested in the 3D simulation scene. For example, it can provide corresponding rendering images, depth maps, etc. for a given UAV perspective. The depth map is an image that contains distance information from each pixel in the captured image to the camera. The above simulation visualization can provide the necessary test data for the UAV to simulate obstacle avoidance in the 3D simulation scene, enabling the UAV to plan its flight obstacle avoidance trajectory and perform autonomous flight tests based on the collected image data in the 3D simulation scene.

[0066] In some embodiments, the 3D simulation scene reconstructed based on the 3DGS algorithm stores occupancy grid information corresponding to the scene to be simulated. This occupancy information can be represented by a 3D grid (such as a raster map), where each grid cell has an identifier indicating the occupancy status of the area. For example, an occupancy identifier indicates that the grid cell is occupied by an obstacle, making passage impossible or resulting in a collision; a free identifier indicates that the grid cell is free of obstacles and allows free passage. By using this grid occupancy information, along with the UAV's position and collision volume information, it is possible to determine whether a collision has occurred in the 3D simulation scene, providing effective test reference information for UAV navigation, obstacle avoidance, and other simulation training processes.

[0067] For example, Figure 5 This is an exemplary rendering of a three-dimensional simulation scene according to some embodiments of this specification. Figure 6This is an exemplary mesh occupancy map of a three-dimensional simulation scene according to some embodiments of this specification. Figure 7 This is an exemplary depth map of a three-dimensional simulation scene according to some embodiments of this specification. For example... Figure 5 and / or Figure 7 As shown, rendering the reconstructed 3D simulation scene can generate camera footage and / or depth maps from a first-person perspective of the drone during drone flight simulation testing, or track and present the drone's trajectory from a third-person perspective. In some embodiments, such as Figure 6 and / or Figure 7 As shown, by analyzing the mesh occupancy and / or depth information of the 3D simulation scene, it is possible to determine whether a UAV collides with the 3D simulation scene during simulated flight, given the UAV's simulated flight position and collision volume. Simultaneously, the UAV's ability to recognize and track objects from its own perspective can be trained during simulated flight. The 3D simulation scene provided in the embodiments of this specification can provide valuable environmental information during simulated training of UAV navigation, obstacle avoidance, object tracking, and object recognition.

[0068] The UAV-based scene simulation method provided in the above embodiments can achieve 3D reconstruction of the scene to be simulated based on images captured by the UAV. During the 3D reconstruction process, there is no need to create a large number of digital assets or perform extensive post-editing work manually, thus improving the efficiency of 3D scene reconstruction. Furthermore, the obtained 3D simulation scene is applicable to UAV visualization simulation, obstacle avoidance training, and other application scenarios. The following will further explain and illustrate the specific methods for obtaining the captured image set in conjunction with specific embodiments.

[0069] In some embodiments, the drone's flight path for capturing images of the simulated scene may include at least two drone movement paths at different flight altitudes, circling and capturing images of the simulated scene. It is understood that, to achieve omnidirectional image acquisition of the simulated scene, the drone may be required to circle and capture images of the simulated scene during its flight. Specifically, circling the simulated scene can be done from the outside of the simulated scene, capturing images from an outside-in angle; or it can be done from the inside of the simulated scene, capturing images from an inside-out angle. In some embodiments, taking an open outdoor plaza as an example, the drone may be configured to circle and capture images over the outer perimeter of the open outdoor plaza, and / or within the plaza itself, to obtain the scene images required for subsequent 3D simulation reconstruction.

[0070] In some embodiments, during the process of drones circling and capturing images of the simulated scene, to avoid missing some details in the simulated scene, the drone can be set to capture images at the same flight altitude during a single circling flight, thereby obtaining continuous images of the simulated scene at the same altitude. Meanwhile, considering that the shooting range of a single image is limited by the drone's onboard imaging equipment, it may not be possible to obtain all the information about the simulated scene in the vertical direction through a single circling flight, the drone's flight trajectory for capturing images of the simulated scene can be set to include at least two drone movement trajectories circling and capturing images of the simulated scene at different flight altitudes. This ensures that the acquired images can cover the simulated scene at different altitudes, and different flight altitudes can also provide different shooting perspectives for the drone to capture the simulated scene, further improving the simulation effect of subsequent 3D scene reconstruction. Figure 8 This is a scene illustration of a drone capturing images of a simulated scene according to some embodiments of this specification. In some embodiments, such as Figure 8 As shown, the drone's movement trajectory includes a first movement trajectory 810, a second movement trajectory 820, and a third movement trajectory 830. These trajectories are located at different altitudes, enabling the drone to capture panoramic images of the simulated scene from various heights. In a specific example, the flight altitude of the first movement trajectory 810 can be set to 0.5 meters, the second movement trajectory 820 to 1.5 meters, and the third movement trajectory 830 to 2.5 meters, thus covering omnidirectional image capture of the simulated scene within a 3-meter height range.

[0071] In some embodiments, during the process of taking panoramic photos of the simulated scene, the drone may also choose a spiral trajectory that gradually increases or decreases its flight altitude as its movement trajectory. Those skilled in the art can choose a suitable drone movement trajectory according to actual needs to ensure all-round shooting of the simulated scene, without any limitation.

[0072] Figure 9 This is an exemplary flowchart illustrating how a drone can capture images of a simulated scene according to some embodiments of this specification. In some embodiments, such as Figure 9 As shown, process 900 may include the following steps.

[0073] Step 910: Plan the drone's shooting flight trajectory based on the scenario to be simulated.

[0074] In some embodiments, the flight trajectory can be pre-set by the user according to the simulation scenario. For example, the processing device 120 can set the flight trajectory of the drone and transmit it to the drone equipped with the data acquisition device 110 or the drone acting as the data acquisition device 110 via the network 130. In some embodiments, the flight trajectory can also be planned by the drone itself according to the simulation scenario. For example, the drone can use its onboard sensors to perceive the environment of the simulation scenario and autonomously plan the flight trajectory based on the results of the environmental perception. This is not limited here. In some embodiments, the specific settings of the flight trajectory can refer to the foregoing embodiments. Figure 8 The details and related explanations will not be elaborated here.

[0075] Step 920: Control the drone to fly according to the shooting flight trajectory and acquire images during the flight, wherein there is at least a partial overlap between adjacent images.

[0076] In some embodiments, during the flight of the UAV according to the shooting flight trajectory, images can be acquired based on a preset shooting interval. During the setting of the shooting interval, it can be controlled to ensure that there is at least partial overlap between adjacent images in the direction of flight along the shooting flight trajectory. This prevents excessively long shooting intervals from resulting in some areas of the simulated scene not being captured, thus affecting the quality of subsequent 3D scene reconstruction. In some embodiments, considering that the UAV's flight speed may be affected by external environmental factors such as wind, the shooting interval can be adaptively adjusted when the UAV's flight speed changes to ensure that there is at least partial overlap between adjacent images.

[0077] In some embodiments, when the drone's flight path includes at least two drone movement trajectories that circle and capture the simulated scene at different flight altitudes, in addition to the partial overlap between adjacent captured images, there can also be at least partial overlap between adjacent captured images in the flight altitude direction, which can ensure that the captured images obtained at different flight altitudes also have overlapping areas in the flight altitude direction.

[0078] Based on the foregoing embodiments, it is possible to acquire omnidirectional images of the simulated scene using drone photography. In some embodiments, further considering that during the omnidirectional image capture of the simulated scene by the drone, the image quality may be poor due to external environmental interference or malfunction of the drone's own imaging equipment, for example, the drone may be affected by wind, causing uncontrolled abnormal shaking, which in turn affects the shooting stability of the drone's imaging equipment, resulting in blurry, distorted, or other poor image quality. Similarly, the drone's imaging equipment may malfunction during the shooting process, also leading to blurry, distorted, or other poor image quality. It is understood that when at least some of the captured images in the image set exhibit poor image quality, it may affect the accuracy of scene reconstruction during subsequent 3D reconstruction using the image set, leading to deviations or other anomalies in the 3D reconstructed scene. To overcome the aforementioned problems, in some embodiments of this specification, during the acquisition of the captured image set, the image quality of the captured images can be further analyzed, and images with poor image quality can be re-captured in a timely manner. This effectively avoids the impact of poor-quality captured images on the subsequent 3D reconstruction process of the simulated scene. The image quality analysis process of the captured images will be specifically described and explained below with reference to embodiments.

[0079] Figure 10 This is an exemplary flowchart illustrating an acquisition of a set of captured images according to some embodiments of this specification. In some embodiments, such as Figure 10 As shown, process 1000 may include the following steps.

[0080] Step 1010: Acquire the first image captured by the drone. In some embodiments, reference may be made to, for example... Figures 5 to 6 The relevant instructions for acquiring the first captured image are not detailed here. In some embodiments, the first captured image can be any image from the set of captured images.

[0081] Step 1020: Determine the degree of blur in the first captured image.

[0082] In some embodiments, the degree of blur is a quantitative description of the clarity and sharpness of image details in the first captured image, reflecting the clarity of edges, textures, and other visual features of various objects in the first captured image; the higher the degree of blur in the first captured image, the more severe the loss of image details. In some embodiments, the degree of blur can be used as an image quality analysis index for the first captured image. The determination of the degree of blur in the first captured image will be explained below with reference to specific embodiments.

[0083] In some embodiments, the degree of blur can be evaluated by calculating the Laplace transform of the image; in other embodiments, considering that blurred images are generally dominated by low-frequency components in the frequency domain, the spectrum corresponding to the first captured image can be obtained by Fourier transform, and the high-frequency components in the spectrum can be analyzed to determine the degree of blur of the first captured image.

[0084] In some embodiments, a trained machine learning model can be used to identify and determine the blur level of the first captured image. Specifically, sampled image features of the first captured image can be obtained first, and then, based on these features, the blur level of the first captured image can be determined using a pre-trained shallow convolutional neural network. In a specific example, the shallow convolutional neural network can adopt a fully convolutional structure with 3*3 convolutional kernels (convolutional filters) and residual blocks with short residual connections. The activation function can be ReLU (Rectified Linear Unit) to enable the shallow convolutional neural network to learn complex nonlinear functional relationships. The last activation layer can use the Sigmoid function to directly output the blur level of the first captured image. In some embodiments, the shallow convolutional neural network can be trained in a supervised manner using images with blur level classification labels to improve the accuracy of the shallow convolutional neural network in determining the blur level of the first captured image. In some embodiments, during the extraction of sampled image features from the first captured image, a Laplacian pyramid corresponding to the first captured image can be constructed first. The Laplacian pyramid, a multi-resolution representation method for image processing, is built based on the concepts of Laplacian transform and Gaussian pyramid, allowing image information to be processed at different scales. In a specific instance, Gaussian filtering layers can be used to progressively blur and downsample the first captured image, such as halving the image size, generating a series of gradually blurred images to serve as a Gaussian pyramid. Then, detailed information is extracted from adjacent Gaussian pyramid layers to construct the Laplacian pyramid. Specifically, this can involve upsampling each layer of the Gaussian pyramid, such as through bilinear interpolation, and then subtracting the upsampled image from the Gaussian pyramid image of the layer above it to obtain the Laplacian image of the current layer and construct the Laplacian pyramid. In some embodiments, to achieve rapid computation and reduce the amount of input feature data, for each layer of the Laplacian pyramid, image blocks can be sampled at intervals. The overall blur level of the first captured image is determined by local regions in the first captured image. Specifically, image blocks of a preset size can be sampled at intervals as sampling blocks corresponding to the first captured image, and all sampling blocks are stitched together to obtain the feature map corresponding to the first captured image. In some embodiments, the Laplacian pyramid features corresponding to the above feature map can be used as sampling image features and input into a shallow convolutional neural network to confirm the blur level. Those skilled in the art can also choose other methods to confirm the blur level of the first captured image according to actual needs, which are not limited here.

[0085] Step 1030: When the blur level of the first captured image deviates from the preset range, obtain the shooting position information corresponding to the first captured image.

[0086] In some embodiments, the preset range can be a range where the blur level characterization value is higher than a preset threshold, which is manually set beforehand. When the blur level of the first captured image deviates from the preset range, it indicates that the current first captured image has a high degree of blur and significant loss of image detail, which may affect subsequent 3D scene reconstruction. This may be due to the influence of external environmental factors such as wind and lighting during the drone's flight and shooting, or it may be due to a malfunction of the shooting equipment carried by the drone during the shooting process, which is not limited here. In some embodiments, when the blur level of the first captured image deviates from the preset range, it is possible to re-capture the image at the same location to obtain a clearer captured image to replace the current first captured image. At this time, it is necessary to obtain the shooting position information corresponding to the first captured image whose blur level deviates from the preset range. In some embodiments, the shooting position information may include the location of the drone when the first captured image was captured, and the shooting angle of the shooting equipment carried by the drone.

[0087] In some embodiments, the acquisition of shooting location information can refer to the relevant descriptions of steps 310 to 320 in the flowchart 300 of the aforementioned embodiment, which will not be repeated here. In some embodiments, considering that it may be difficult to extract feature point information and directly determine the corresponding shooting location information from the first captured image whose blur level deviates from the preset range, the adjacent shooting location information corresponding to other captured images adjacent to the first captured image and whose blur level does not deviate from the preset range can be obtained first. Then, based on the adjacent shooting location information, combined with the image shooting interval of the UAV and the shooting flight trajectory of the UAV, the shooting location information corresponding to the first captured image can be confirmed. This is not limited here.

[0088] Step 1040: Based on the shooting location information, control the drone to re-capture the first captured image. In some embodiments, the set of captured images of the simulated scene obtained based on the above process 1000 does not include the first captured image whose blur level deviates from a preset range, and may include the first captured image whose blur level does not deviate from the preset range after re-capture.

[0089] In some embodiments, the user can plan a re-capture flight trajectory for the drone based on its current location and the shooting position information corresponding to one or more first captured images that need to be re-captured. The drone can be controlled to fly along the re-capture flight trajectory and re-capture the first captured images during flight according to the corresponding shooting position information. In some embodiments, the drone can also autonomously plan a re-capture flight trajectory based on the shooting position information corresponding to one or more first captured images that need to be re-captured, and automatically re-capture the first captured images according to the re-capture flight trajectory; this is not limited here. In some embodiments, to avoid the drone frequently re-capturing the first captured images and affecting the overall shooting efficiency of the drone in the simulated scene, the first captured images whose blur level deviates from the preset range can be uniformly re-captured after the drone has completed a single capture of the simulated scene. In some embodiments, when the drone's shooting flight trajectory for the simulated scene includes at least two drone movement trajectories that circle and capture the simulated scene at different flight altitudes, the first captured images whose blur level deviates from the preset range in the drone movement trajectory can be uniformly re-captured after the drone has completed the flight shooting process of a single drone movement trajectory; this is not limited here.

[0090] In some embodiments, the above process 1000 can be implemented by the acquisition device 110, such as a camera mounted on a drone's gimbal. During the acquisition of the first captured image, the blur level of the first captured image is simultaneously confirmed based on a preset image blur recognition algorithm or the shallow convolutional neural network model provided in the aforementioned embodiments. It is understood that by employing methods such as interval sampling of the first captured image and using a shallow convolutional neural network model for blur level confirmation, the above process 1000 can achieve millisecond-level image quality analysis on the mobile SOC (System on Chip) chip of the acquisition device 110. This allows for timely filtering of the first captured image whose blur level deviates from the preset range during the drone's capture of the simulated scene, enabling the drone to perform timely reconstruction after a single capture of the simulated scene. This ensures the accuracy of the 3D scene simulation while improving the overall reconstruction efficiency of the 3D scene simulation.

[0091] In some embodiments, the above process 1000 can be implemented in collaboration between the acquisition device 110 and the processing device 120. For example, the acquisition device 110 carried by the drone can acquire the first captured image and transmit the captured image to the processing device 120 via the network 130 for image quality analysis and other operations. Specifically, it can be implemented by the scene simulation system 1300 (such as the scene acquisition module 1310) deployed on the processing device 120. It is understandable that, in order to ensure the integrity of the information collected by the drone during the shooting process of the simulated scene, for any part of the scene to be simulated, there can be more than one captured image to be captured from multiple angles to achieve redundant shooting of the simulated scene; if one of the captured images has an abnormal situation where the blur level deviates from the preset range, the processing device 120 can first determine whether the captured image from other angles can achieve three-dimensional reconstruction of the scene part: if three-dimensional reconstruction can be achieved, then it is only necessary to remove the captured image with the blur level deviating from the preset range from the captured image set, without having to re-capture the captured image with the blur level deviating from the preset range; if three-dimensional reconstruction is difficult to achieve, it means that the captured image with the blur level deviating from the preset range provides key scene information in the three-dimensional reconstruction process of the scene part, and it is necessary to re-capture the captured image according to the process 1000 provided in the aforementioned embodiment, and include the re-captured image in the captured image set.

[0092] Based on the description of the foregoing embodiments, it is possible to remove the first captured image whose blur level deviates from the preset range from the captured image set, and obtain an optimized captured image set by means of re-capturing, so that the subsequent three-dimensional reconstruction process of the simulated scene is not affected by image blur.

[0093] In some embodiments, other image evaluation criteria can also be used as image quality analysis indicators for drone-captured images. For example, a brightness histogram can be used as an overexposure analysis indicator for drone-captured images. A brightness histogram is a graphical tool used in image processing to represent the brightness distribution of an image, showing the frequency of each brightness value (or gray level) in the drone-captured image. When the brightness histogram shows that the proportion of pixels concentrated in high-brightness areas in the drone-captured image exceeds a preset ratio, it can be determined that the drone-captured image has an overexposure anomaly. The image can then be reshot, or the simulated scene contained in the captured image can be reconstructed based on other images taken from different shooting angles in the image set. Those skilled in the art can choose appropriate image evaluation criteria according to the actual needs of the drone-captured scene and apply the image optimization processing provided in the foregoing embodiments to images that do not meet the image evaluation criteria; no limitations are imposed here.

[0094] In some embodiments, further considering that during the process of taking omnidirectional images of some open-ended simulated scenes, drones may easily capture moving objects in the simulated scenes, such as pedestrians walking in an outdoor plaza or vehicles moving at the edge of the plaza, these moving objects can cause inconsistencies in the scene between consecutive frames in the captured image set, thus affecting the quality of the generated 3D simulation scene. To overcome the above problems, in some embodiments of the present invention, moving objects appearing in the captured images can be identified and processed simultaneously with the acquisition of the captured image set, so as to eliminate the impact of moving objects on the reconstruction of the 3D simulation scene. The following will specifically describe and explain the identification and processing of moving objects in the captured images in conjunction with embodiments.

[0095] Figure 11 This is an exemplary flowchart illustrating another method for acquiring a set of captured images, according to some embodiments of this specification. In some embodiments, such as Figure 11 As shown, process 1100 may include the following steps.

[0096] Step 1110: Acquire a second image captured by the drone. In some embodiments, reference may be made to, for example... Figures 5 to 6 The relevant instructions for acquiring the second captured image are not detailed here. In some embodiments, the second captured image can be any image from the set of captured images.

[0097] Step 1120: Identify moving objects contained in the second captured image.

[0098] In some embodiments, a moving object refers to a moving object appearing in the simulated scene, such as a pedestrian, animal, or vehicle moving in the simulated scene. The specific process of identifying moving objects will be further explained below.

[0099] Figure 12 This is an exemplary flowchart illustrating the identification of a moving object according to some embodiments of this specification. In some embodiments, such as Figure 12 As shown, process 1200 may include the following steps.

[0100] Step 1210: Based on the depth map corresponding to the second captured image, obtain the depth information corresponding to multiple captured objects.

[0101] In some embodiments, a depth map is an image that contains distance information from each pixel in the captured image to the camera. It is usually represented in the form of grayscale values ​​or color values. Pixels with higher brightness are closer to the camera, and pixels with lower brightness are farther away from the camera. The depth map corresponding to the second captured image can be obtained through a pre-trained depth prediction network.

[0102] In some embodiments, the subject being photographed is the main content presented in the second captured image, including both constituent objects of the scene to be simulated, such as landscape pillars, pedestrian walkways, and planted trees in an outdoor plaza scene, and moving objects moving within the scene to be simulated, without limitation. In some embodiments, the subject being photographed in the second captured image can be obtained through a pre-trained object prediction network, which specifically includes obtaining the position and pixel range occupied by each subject in the second captured image. In some embodiments, based on the position and pixel range occupied by the subject in the second captured image, combined with the depth map corresponding to the second captured image, the depth information corresponding to each subject can be obtained, which represents the distance between the subject and the drone.

[0103] Step 1220: Based on the second captured image and the captured images at adjacent time points, determine the motion vectors corresponding to multiple captured objects, and determine the average motion vector of captured objects with the same depth information.

[0104] In some embodiments, the motion vector corresponding to each identified object in the second captured image can be obtained by optical flow calculation based on the second captured image and images captured at adjacent time points. Optical flow can reflect the motion information of objects or scenes in multiple consecutive captured images and is typically used to describe the motion vector of each pixel in multiple consecutive frames. In some embodiments, based on the depth information corresponding to the captured object obtained in step 1210, the average motion vector of captured objects with the same depth information can be calculated according to the motion vectors corresponding to captured objects with the same depth information.

[0105] Step 1230: Select the subject whose motion vector deviates from the preset mean of the motion vector as the moving subject.

[0106] In some embodiments, it is understood that in a relatively static simulated scene, the motion vectors corresponding to objects with the same depth information are often consistent or similar. When the motion vector of a certain object deviates significantly from the motion vectors of other objects with the same depth information, the object can be identified as a moving object. In some embodiments, a unified motion object identification and determination can be performed on objects with the same depth information. If there are objects whose motion vectors deviate from the preset average value of the motion vectors, it indicates that the motion vector of that object deviates significantly from the motion vectors of other objects with the same depth information, and it can be identified as a moving object.

[0107] Step 1130: Perform masking processing on the portion of the second captured image containing moving objects to obtain an optimized captured image that ignores moving objects. In some embodiments, the set of captured images of the scene to be simulated obtained based on the above process 1100 includes the optimized captured image after the above masking processing.

[0108] In some embodiments, to avoid the impact of moving objects appearing in the captured image on the quality of subsequent 3D simulation scene generation, for moving objects identified in the second captured image, a masking process can be used to occupy the pixel range occupied by the moving object, so that the pixel region corresponding to the moving object in the second captured image is not used in the subsequent 3D reconstruction process. In some embodiments, masking, as a widely used technique in image processing, can be used to selectively process certain parts of an image without affecting other areas of the image. The mask (or masking plate) can be a binary image of the same size as the second captured image, where each pixel value determines whether a certain pixel in the corresponding second captured image is processed. Taking a black and white binary mask as an example, the black area can represent the part of the second captured image that does not need to participate in the subsequent 3D reconstruction processing, such as the pixel region corresponding to the moving object; the white area can represent the part of the second captured image that needs to participate in the subsequent 3D reconstruction processing, such as other areas besides the pixel region corresponding to the moving object. Understandably, although the moving object may occlude part of the simulated scene that needs to be reconstructed in the second captured image, the moving object will not continuously occlude a certain part of the simulated scene during its movement. Furthermore, the drone will take circling shots of the simulated scene at different flight altitudes when shooting. Therefore, sufficient features can be extracted from other second captured images in the image set to reconstruct the part of the simulated scene occluded by the moving object in one or more second captured images.

[0109] In some embodiments, considering that drones often need to include as much and as clear as possible scene information in a single captured image when shooting simulated scenes, the resolution of the second captured image is usually set to be relatively high. This makes directly performing motion recognition processing in step 1200 on the second captured image consume a lot of computing resources and take a long time. In some embodiments, before performing object recognition and depth information acquisition in step 1210, the second captured image can be downsampled to reduce its resolution, thereby reducing the overall computational load in the process of recognizing moving objects and enabling faster and more efficient recognition of moving objects in the second captured image.

[0110] Based on the descriptions of the foregoing embodiments, it is possible to obtain a set of captured images corresponding to the scene to be simulated, and the obtained set of captured images includes optimized captured images that can be adapted to the subsequent 3D scene reconstruction process.

[0111] Some embodiments of this specification also provide a scene simulation system based on unmanned aerial vehicles (UAVs). Figure 13 This is a functional block diagram of a drone-based scene simulation system according to some embodiments of this specification. In some embodiments, such as Figure 13 As shown, the scene simulation system 1300 may include: a scene acquisition module 1310, a sparse reconstruction module 1320, an object recognition module 1330, and a reconstruction simulation module 1340. In some embodiments, each module of the scene simulation system 1300 may be configured in the processing device 120.

[0112] In some embodiments, the scene acquisition module 1310 can be used to acquire a set of images captured during the execution of the scene simulation method of the embodiments of this specification. In some embodiments, the scene acquisition module 1310 is used to acquire a set of images of the scene to be simulated based on the images captured by the UAV. In some embodiments, the scene acquisition module 1310 can be used to optimize the images captured by the UAV to adapt them to the subsequent three-dimensional scene reconstruction process.

[0113] In some embodiments, the sparse reconstruction module 1320 can be used to perform sparse reconstruction on a set of captured images to obtain a sparse point cloud set during the execution of the scene simulation method of the embodiments of this specification. In some embodiments, the sparse reconstruction module 1320 can be used to determine a sparse point cloud set of the scene to be simulated based on the set of captured images.

[0114] In some embodiments, the object recognition module 1330 can be used to perform object recognition on the captured images in the captured image set and assign semantic recognition tags corresponding to the captured objects, so as to obtain the recognition tag set during the execution of the scene simulation method of this specification embodiment. In some embodiments, the object recognition module 1330 can be used to determine the recognition tag set of at least some captured objects in the captured image set based on the captured image set.

[0115] In some embodiments, the reconstruction simulation module 1340 can be used to perform three-dimensional reconstruction of the scene to be simulated. In some embodiments, the reconstruction simulation module 1340 can be used to perform three-dimensional reconstruction of the scene to be simulated based on a set of captured images, a set of sparse point clouds, and a set of recognized labels, using a 3DGS algorithm, to generate a three-dimensional simulation scene corresponding to the scene to be simulated.

[0116] For more information on each module, please refer to [link / reference]. Figures 2 to 12 The relevant explanations will not be repeated here. It should be understood that... Figure 13 The systems and modules shown can be implemented in various ways. For example, in some embodiments, the systems and modules can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The systems and modules of this specification can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips and transistors, or programmable hardware devices such as field-programmable gate arrays and programmable logic devices, but also by software, for example, executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0117] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules without departing from these principles to form subsystems connected to other modules. Alternatively, some modules may be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0118] Some embodiments of this specification also provide a scene simulation device based on a drone, which includes a processor and a storage medium. The storage medium stores computer program instructions, and the processor is used to execute at least a portion of the computer program instructions to implement the drone-based scene simulation method provided in the foregoing embodiments of this specification.

[0119] Some embodiments of this specification also provide a computer program product, including computer instructions or a computer program, which, when at least a portion of the computer instructions or computer program is executed by a processor, can implement the UAV-based scene simulation method provided in the foregoing embodiments of this specification.

[0120] In some embodiments, the processor may be a combination of one or more of the following processors: central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), programmable logic controller (PLC), reduced instruction set computer (RISC), microprocessor, etc. In some embodiments, the processor may be processing device 120.

[0121] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) It can realize accurate three-dimensional reconstruction of the scene to be simulated based on the images captured by the UAV of the scene to be simulated. In the process of three-dimensional reconstruction, there is no need to make a large number of digital assets or manually perform a large amount of post-editing work, which improves the production efficiency of scene three-dimensional reconstruction. Moreover, the obtained three-dimensional simulation scene can be applied to the application scenarios of UAV visualization simulation, obstacle avoidance training and so on. (2) During the process of the UAV shooting the scene to be simulated, at least two surround shooting flight paths at different altitudes are set to realize all-round shooting of the scene to be simulated. (3) The blur degree of the captured image is determined, and the captured image with the blur degree deviating from the preset range is reshot, so as to avoid the blurry captured image affecting the reconstruction accuracy of the three-dimensional simulation scene. (4) The moving objects in the captured image are identified and the moving objects are masked, so as to avoid the moving objects appearing in the captured image affecting the generation quality of the subsequent three-dimensional simulation scene. It should be noted that the beneficial effects that may be produced by different embodiments are different. In different embodiments, the beneficial effects that may be produced can be any one or a combination of the above, or any other possible beneficial effects.

[0122] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. A scene simulation method based on unmanned aerial vehicles (UAVs), characterized in that, The method includes: Based on the drone's capture of the simulated scene, a set of captured images of the simulated scene is obtained; Based on the set of captured images, a sparse point cloud set of the scene to be simulated is determined; Based on the set of captured images, determine a set of identification tags for at least a portion of the captured objects in the set of captured images; Based on the set of captured images, the set of sparse point clouds, and the set of identification tags, the 3DGS algorithm is used to reconstruct the three-dimensional scene to be simulated, so as to generate a three-dimensional simulation scene corresponding to the scene to be simulated.

2. The method according to claim 1, characterized in that, The drone's flight trajectory for capturing the simulated scene includes at least two drone movement trajectories at different flight altitudes that circle and capture the simulated scene.

3. The method according to claim 1, characterized in that, The process of acquiring a set of images of the simulated scene based on drone photography includes: The drone's flight path is planned based on the simulated scenario. The drone is controlled to fly according to the shooting flight trajectory and to capture images during the flight, wherein there is at least a partial overlap between adjacent captured images.

4. The method according to claim 1, characterized in that, The process of acquiring a set of images of the simulated scene based on drone photography includes: Acquire the first image captured by the drone; Determine the degree of blur in the first captured image; When the blur level of the first captured image deviates from a preset range, the shooting position information corresponding to the first captured image is obtained; Based on the shooting location information, control the drone to re-shoot the first captured image.

5. The method according to claim 4, characterized in that, Determining the blur level of the first captured image includes: Obtain the sampled image features of the first captured image; Based on the features of the sampled image, the degree of blur in the first captured image is determined using a pre-trained shallow convolutional neural network.

6. The method according to claim 5, characterized in that, The step of extracting the sampled image features of the first captured image includes: Construct the Laplacian pyramid corresponding to the first captured image; In each layer of the Laplacian pyramid, image blocks of a preset size are sampled at intervals as the sampling blocks corresponding to the first captured image; All the sampling blocks are stitched together to obtain the feature map corresponding to the first captured image, and the Laplacian pyramid feature corresponding to the feature map is used as the feature of the sampling image.

7. The method according to claim 1, characterized in that, The process of acquiring a set of images of the simulated scene based on drone photography includes: Acquire the second image captured by the drone; Identify moving objects contained in the second captured image; The portion of the second captured image containing the moving object is masked to obtain an optimized captured image that ignores the moving object. The set of captured images includes the optimized captured images.

8. The method according to claim 7, characterized in that, The identification of moving objects contained in the second captured image includes: Based on the depth map corresponding to the second captured image, obtain depth information corresponding to multiple captured objects; Based on the second captured image and the captured images at adjacent time points, determine the motion vectors corresponding to the multiple captured objects, and determine the average motion vector of the captured objects with the same depth information; The subject whose motion vector deviates from the mean of the motion vector by a preset range is considered as the moving subject.

9. The method according to claim 1, characterized in that, The step of determining the sparse point cloud set of the scene to be simulated based on the captured image set includes: Obtain feature point information of the captured images in the captured image set; Based on the feature point information, obtain the shooting location information corresponding to the captured image; Based on the shooting location information and the feature point information, the sparse point cloud set is constructed.

10. The method according to claim 1, characterized in that, The step of determining a set of identification tags for at least a portion of the captured objects in the captured image set includes: Identify at least some of the subjects captured in the captured image and label the captured subjects with identification tags; The identification tags corresponding to the captured objects in the captured image set are standardized to unify the identification tags corresponding to the same captured objects contained in different captured images; The captured image portion corresponding to the photographed object and the identification tag are used as the identification tag set.

11. A scene simulation system based on unmanned aerial vehicles (UAVs), characterized in that, The system includes: The scene acquisition module is used to acquire a set of captured images of the scene to be simulated based on the drone's capture of the scene; The sparse reconstruction module is used to determine the sparse point cloud set of the scene to be simulated based on the set of captured images. An object recognition module is used to determine a set of recognition tags for at least a portion of the captured objects in the captured image set based on the captured image set. The reconstruction simulation module is used to perform three-dimensional reconstruction of the scene to be simulated based on the captured image set, the sparse point cloud set, and the identification tag set, using the 3DGS algorithm to generate a three-dimensional simulation scene corresponding to the scene to be simulated.

12. A computer program product, characterized in that, It includes computer instructions or computer programs, which, when executed by a processor in at least a portion, enable the implementation of the UAV-based scene simulation method as described in any one of claims 1 to 10.

Images