Three-dimensional reconstruction method based on unmanned aerial vehicle and edge calculation

Abstract

The invention discloses a three-dimensional reconstruction method based on an unmanned aerial vehicle and edge calculation, and relates to the field of unmanned aerial vehicle technology, edge calculation and three-dimensional scene reconstruction. The method comprises the following steps: an unmanned aerial vehicle carries a monocular camera to collect a standard-definition video stream from a target scene; the edge computing device receives and processes the standard-definition video stream in real time to generate preliminary low-precision three-dimensional reconstruction; the edge computing device identifies a defect area by analyzing a low-precision reconstruction result, and optimizes the flight path of the unmanned aerial vehicle; the unmanned aerial vehicle collects a high-definition video stream from the target scene according to the optimized path; and the edge computing device performs off-line high-precision three-dimensional reconstruction on the high-definition video stream so as to improve the accuracy and detail expression capability of a reconstructed scene. Through cooperative work of the unmanned aerial vehicle and the edge calculation end, real-time data processing, feedback and decision making in the collection process and high-precision three-dimensional reconstruction result storage in the off-line stage are achieved, and the real-time performance and accuracy of three-dimensional reconstruction are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) technology, edge computing, and 3D scene reconstruction, and specifically to a 3D reconstruction method based on UAVs and edge computing. Background Art

[0002] 3D reconstruction technology plays a vital role at the forefront of current science and technology. It transforms the real world into accurate digital 3D models, expanding our understanding and ability to represent reality. With advancements in computing power and algorithms, 3D reconstruction has been widely applied in fields such as digital media, gaming, cultural heritage preservation, industrial design, urban planning, topographic mapping, and disaster management.

[0003] In recent years, the rapid development of drone technology has made it an important tool for 3D reconstruction. Drones, equipped with high-definition cameras or other sensors, can fly quickly and flexibly in various environments, acquiring large-scale, high-quality image data. The application of drones improves data collection efficiency and reduces manpower and time costs, showing unique advantages, especially in hard-to-reach or dangerous areas. However, drone 3D reconstruction faces challenges such as poor real-time performance and low computing power, affecting its efficiency. Edge computing, by deploying computing tasks at the network edge, close to the data source, reduces data transmission latency, improves processing speed, and allows for real-time analysis and decision-making. In drone 3D reconstruction, this method can dynamically adjust flight paths and camera parameters, optimize data collection strategies, and achieve efficient and accurate 3D reconstruction.

[0004] Therefore, in order to address the inherent challenges of high efficiency and accuracy in UAV-based 3D reconstruction, it is necessary to provide a UAV-based 3D reconstruction method using edge computing to solve the aforementioned problems. Summary of the Invention

[0005] The purpose of this invention is to provide a 3D reconstruction method based on UAVs and edge computing, which solves the problem that existing UAV-based 3D reconstruction methods are not effective in terms of accuracy and efficiency.

[0006] To achieve the above objectives, this invention provides a 3D reconstruction method based on unmanned aerial vehicles (UAVs) and edge computing, comprising the following steps:

[0007] S1: The drone carries a monocular camera to capture standard definition video streams of the target scene;

[0008] S2: Edge computing devices receive and process standard definition video streams in real time to generate preliminary low-precision 3D reconstructions;

[0009] S3: Edge computing devices identify defect areas by analyzing low-precision reconstruction results and optimize the flight path of drones;

[0010] S4: The drone captures a high-definition video stream of the target scene according to the optimized path;

[0011] S5: Edge computing devices perform offline high-precision 3D reconstruction of the high-definition video stream to improve the accuracy and detail representation of the reconstructed scene.

[0012] Preferably, step S2 specifically includes the following steps:

[0013] S2.1: The edge computing device receives the standard definition video stream from the drone in real time. During the reception process, the edge device ensures the integrity and real-time performance of the video stream. In addition, the edge device also needs to monitor the transmission status of the video stream to ensure stable transmission of the video stream during the drone's flight and prevent reconstruction interruptions due to signal interference or loss.

[0014] S2.2: The edge computing device encodes the received standard-definition video stream, converting the original video frames into a format suitable for algorithm processing. Subsequently, the processed image data is input into the spatial memory module. The spatial memory module stores the 3D information accumulated during the previous reconstruction process. Based on this, the algorithm extracts key features from the current frame to generate query features, specifically:

[0015]

[0016] in, The query features generated at time t The features decoded by the target decoder at time t. The features obtained by the encoder at time t are used to retrieve 3D information related to the current frame from the spatial memory module.

[0017] S2.3: The memory retrieval module extracts the 3D information retrieved from the spatial memory module and fuses it with the key features of the current frame. The decoder is responsible for decoding this information, combining the retrieved historical 3D data with the features of the current frame to generate preliminary 3D scene point cloud data, specifically:

[0018]

[0019] Among them, A t-1 It is an attention map, f K and f V These are the key and value features in the spatial memory module, respectively.

[0020] S2.4: The initial 3D scene point cloud data is optimized by the optimizer to correct potential errors and improve the coherence and accuracy of the 3D scene. Specifically:

[0021]

[0022] Among them, the features output by the decoder A point plot and its confidence level are generated using the MLP header.

[0023] Preferably, step S3 specifically includes the following steps:

[0024] S3.1: For the initially reconstructed 3D scene, an image evaluation convolutional neural network is used to evaluate the quality of the generated rendered image. This network evaluates the quality of the reconstructed image based on the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM), specifically:

[0025] Q = w p ·P+w s ·S (5)

[0026] w p +w s =1 (6)

[0027] Where Q is the overall image quality score, which is a weighted average of PSNR(P) and SSIM(S), with weights w and w respectively. p and w s When the overall score of a certain area is lower than a preset threshold, the area is marked as a defective area.

[0028] S3.2: Based on the output of the image evaluation module, dynamically adjust the UAV's flight path. When the system detects an area with substandard reconstruction quality, it generates a new flight mission, causing the UAV to return to that area for supplementary photography to improve the reconstruction quality.

[0029] Preferably, step S5 specifically includes the following steps:

[0030] S5.1: Acquire and initialize each frame of the input high-definition video stream data, and create opaque and nearly transparent 3D Gaussian bodies, where the opaque Gaussian body is used to fit the surface and dominant color, while the transparent Gaussian body is used to fit the remaining colors.

[0031] S5.2: Explicitly add Gaussian bodies corresponding to three types of pixels for each frame of the image, and initiate Gaussian body optimization based on the color and depth losses between the input and rendered RGB-D images, specifically:

[0032] L = w c L color +w d Ldepth +w2L2 (7)

[0033]

[0034] Among them, L color For color loss, L depth For depth loss, the actual color image C is measured. k With rendering color images Actual depth image D k With rendering depth images The absolute error is L1, while L2 is the L2 loss used to constrain the geometric properties of the transparent Gaussian body, and w corresponds to the weights of the three.

[0035] S5.3: A real-time optimization scheme is used to optimize the Gaussian body online to adapt to environmental changes and reduce memory and computational costs. A weighted averaging method is used to merge the optimization results of the current time window with previous results to avoid forgetting issues. Specifically:

[0036] η′ o =(1-w curr )η o-1 +w curr η o (10)

[0037]

[0038] Among them, η o-1 Based on the optimization result of the previous frame, η o The weight w is the result of the optimization for the current frame. curr The weighted average value η′ is determined by reflecting the magnitude of the difference between the current frame and historical frames. o .

[0039] S5.4: Through continuous optimization and fusion, a 3D scene model containing high-precision geometric structure and color information is reconstructed and stored in the edge device.

[0040] Therefore, the present invention employs the above-mentioned 3D reconstruction method based on UAVs and edge computing, which has the following beneficial effects:

[0041] (1) This invention combines UAVs with edge computing to significantly improve the efficiency of 3D reconstruction without affecting reconstruction accuracy. UAVs can quickly acquire a large amount of image data, while edge computing can process this data in real time, reducing data transmission latency and accelerating reconstruction speed.

[0042] (2) By introducing edge computing devices, this invention can dynamically adjust the flight path of the UAV based on the preliminary reconstruction results, reduce repeated collection and invalid flight, optimize the data collection strategy, and thus improve the effectiveness and relevance of data collection.

[0043] (3) This invention reduces the computational burden on the drone itself by transmitting drone data to an edge computing device for processing, thereby extending the drone's working time and range. At the same time, edge computing reduces dependence on centralized data centers and lowers the overall system cost.

[0044] (4) This invention improves the accuracy and detail expression of the reconstructed scene by acquiring high-definition video streams and offline high-precision 3D reconstruction, ensuring that the final 3D model has high fidelity and reliability.

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0046] Figure 1 A framework diagram of a 3D reconstruction method based on UAV and edge computing provided for an embodiment of the present invention;

[0047] Figure 2 A block diagram of real-time low-precision reconstruction provided in an embodiment of the present invention;

[0048] Figure 3 This is a block diagram of offline high-precision reconstruction provided in an embodiment of the present invention; Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0051] The terms "comprising" or "including" as used in this invention mean that the element preceding the term encompasses the element listed after the term, and do not exclude the possibility of encompassing other elements. Terms such as "inner," "outer," "upper," and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In this invention, unless otherwise explicitly specified and limited, the term "attached" and similar terms should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two elements or the interaction relationship between two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0052] like Figure 1 As shown in the figure, the present invention discloses a 3D reconstruction method based on UAV and edge computing, which mainly includes the following steps:

[0053] S1: The drone carries a monocular camera to capture standard definition video streams of the target scene;

[0054] Preferably, this process involves precise calibration of the UAV and monocular camera to ensure high temporal synchronization and spatial consistency of the acquired video stream. The monocular camera provides the system with rich visual information, while the depth information of the scene is inferred by edge devices through consecutive image frames, laying the foundation for subsequent 3D reconstruction. In practice, the standard-definition video stream acquired by the UAV provides a continuous and reliable information source for the 3D reconstruction algorithm, ensuring the quality of the video stream and the reconstruction effect.

[0055] S2: Edge computing devices receive and process standard definition video streams in real time to generate preliminary low-precision 3D reconstructions;

[0056] Preferably, the edge computing device receives the standard definition video stream transmitted from the drone in real time. During the reception process, the edge device ensures the integrity and real-time performance of the video stream. In addition, the edge device also needs to monitor the transmission status of the video stream to ensure stable transmission of the video stream during the drone's flight and prevent reconstruction interruptions due to signal interference or loss.

[0057] Further, refer to Figure 2The edge computing device encodes the received standard-definition video stream, converting the original video frames into a format suitable for algorithm processing. Subsequently, the processed image data is input into the spatial memory module. The spatial memory module stores 3D information accumulated during the previous reconstruction process. Based on this, the algorithm extracts key features from the current frame to generate query features, specifically:

[0058]

[0059] in, The query features generated at time t The features decoded by the target decoder at time t. The features obtained by the encoder at time t are used to retrieve 3D information related to the current frame from the spatial memory module.

[0060] It is worth noting that the memory retrieval module extracts 3D information retrieved from the spatial memory module and fuses it with the key features of the current frame. The decoder is responsible for decoding this information, combining the retrieved historical 3D data with the features of the current frame to generate preliminary 3D scene point cloud data, specifically:

[0061]

[0062] Among them, A t-1 It is an attention map, f K and f V These are the key and value features in the spatial memory module, respectively.

[0063] Finally, the preliminary 3D scene point cloud data is optimized by an optimizer to correct potential errors and improve the coherence and accuracy of the 3D scene. Specifically:

[0064]

[0065] Among them, the features output by the decoder A point plot and its confidence level are generated using the MLP header.

[0066] S3: Edge computing devices identify defect areas by analyzing low-precision reconstruction results and optimize the flight path of drones;

[0067] Preferably, for the initially reconstructed 3D scene, an image evaluation convolutional neural network is used to evaluate the quality of the generated rendered image. This network evaluates the quality of the reconstructed image based on the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM), specifically:

[0068] Q = w p ·P+w s ·S (5)

[0069] w p +w s =1 (6)

[0070] Where Q is the overall image quality score, which is a weighted average of PSNR(P) and SSIM(S), with weights w and w respectively. p and w s When the overall score of a certain area is lower than a preset threshold, the area is marked as a defective area.

[0071] Furthermore, the drone's flight path is dynamically adjusted based on the output of the image evaluation module. When the system detects an area with substandard reconstruction quality, a new flight mission is generated, causing the drone to return to that area for supplementary photography to improve the reconstruction quality.

[0072] S4: The drone captures a high-definition video stream of the target scene according to the optimized path;

[0073] Preferably, this process involves precise optimization of the UAV's flight path and acquisition of high-definition video streams from a monocular camera. In practice, the UAV executes its flight mission according to the optimized path, ensuring full coverage of each key area while avoiding unnecessary duplicate acquisitions. During the acquisition of the high-definition video stream, the monocular camera prioritizes the quality and stability of the video stream. The high-definition video stream provides the system with rich and detailed visual information, further enhancing the 3D reconstruction results.

[0074] S5: Edge computing devices perform offline high-precision 3D reconstruction of the high-definition video stream to improve the accuracy and detail representation of the reconstructed scene.

[0075] Preferred, Reference Figure 3 The process involves taking each frame of the input high-definition video stream data and initializing it to create opaque and nearly transparent 3D Gaussian bodies. The opaque Gaussian body is used to fit the surface and dominant color, while the transparent Gaussian body is used to fit the remaining colors.

[0076] Furthermore, Gaussian bodies corresponding to three types of pixels are explicitly added to each frame of the image, and Gaussian body optimization is initiated based on the color loss and depth loss between the input and rendered RGB-D images, specifically:

[0077] L = w c L color +w d L depth +w2L2 (7)

[0078]

[0079] Among them, L color For color loss, L depth For depth loss, the actual color image C is measured. k With rendering color images Actual depth image D k With rendering depth images The absolute error is L1, while L2 is the L2 loss used to constrain the geometric properties of the transparent Gaussian body, and w corresponds to the weights of the three.

[0080] It is worth noting that a real-time optimization scheme is used to optimize the Gaussian body online to adapt to environmental changes and reduce memory and computational costs. Furthermore, a weighted averaging method is used to integrate the optimization results of the current time window with previous results to avoid forgetting issues. Specifically:

[0081] η′ o =(1-w curr )η o-1 +w curr η o (10)

[0082]

[0083] Among them, η o-1 Based on the optimization result of the previous frame, η o The weight w is the result of the optimization for the current frame. curr The weighted average value η′ is determined by reflecting the magnitude of the difference between the current frame and historical frames. o .

[0084] Finally, through continuous optimization and fusion, a 3D scene model containing high-precision geometric structure and color information is reconstructed and stored in the edge device.

[0085] Therefore, this invention employs a 3D reconstruction method based on UAVs and edge computing. By using UAVs to collect image data and combining it with the powerful processing capabilities of edge computing, efficient and accurate reconstruction of 3D scenes is achieved. This method leverages the flexibility and wide coverage of UAVs in different environments, combined with the real-time processing and intelligent decision-making functions of edge computing devices, to generate more accurate and detailed 3D models. While ensuring reconstruction quality, this invention also significantly improves the real-time performance and adaptability of the 3D reconstruction process, achieving efficient perception and understanding of dynamic and complex environments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A three-dimensional reconstruction method based on drone and edge computing, characterized in that: The following steps are involved: S1: The drone carries a monocular camera to collect standard-definition video streams of the target scene; S2: The edge computing device receives and processes the standard definition video stream in real time and generates a preliminary low-precision 3D reconstruction; S3: The edge computing device identifies defective areas by analyzing low-precision reconstruction results and optimizes the flight path of the drone; S4: The drone collects high-definition video streams of the target scene based on the optimized path; S5: The edge computing device performs offline high-precision 3D reconstruction on the high-definition video stream to improve the accuracy and detail expression of the reconstructed scene.

2. The three-dimensional reconstruction method based on drone and edge computing according to claim 1, characterized in that: The specific method of step S2 is as follows: S2.1: The edge computing device receives the standard definition video stream from the drone in real time. During the reception process, the edge device will ensure the integrity and real-time nature of the video stream. In addition, the edge device also needs to monitor the transmission status of the video stream to ensure stable transmission of the video stream during the flight of the drone and prevent reconstruction interruptions caused by signal interference or loss. S2.2: The edge computing device encodes the received standard definition video stream and converts the original video frame into a format suitable for algorithm processing. Subsequently, the processed image data is input into the spatial memory module. The spatial memory module stores the 3D information accumulated during the previous reconstruction process. On this basis, the algorithm extracts key features from the current frame and generates query features, specifically: in, is the query feature generated at time t, is the feature decoded by the target decoder at time t, are the features processed by the encoder at time t. These query features are used to retrieve 3D information related to the current frame from the spatial memory module. S2.3: The memory reading module extracts the 3D information retrieved from the spatial memory module and fuses it with the key features of the current frame. The decoder is responsible for decoding this information, combining the retrieved historical 3D data with the features of the current frame to generate preliminary 3D scene point cloud data, specifically: Among them, A t-1 is the attention map, f K and f V They are the key and value features in the spatial memory module respectively. S2.4: The preliminary 3D scene point cloud data is optimized in the optimizer to correct possible errors and improve the coherence and accuracy of the 3D scene. Specifically: Among them, the features output by the decoder are Generate a point map and its confidence through the MLP head.

3. The three-dimensional reconstruction method based on drone and edge computing according to claim 1, characterized in that: The specific method of step S3 is as follows: S3.1: For the initially reconstructed 3D scene, the quality of the generated rendered image is evaluated using an image evaluation convolutional neural network. The network evaluates the quality of the reconstructed image based on the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM), specifically: Q=w p ·P+w s ·S (5) In p +in s =1 (6) Among them, Q is the comprehensive score of image quality, which is the weighted average of PSNR (P) and SSIM (S), with weights w p and w s ,When the comprehensive score of a region is lower than the preset threshold, the region is marked as a ,defective region. S3.2: Dynamically adjust the flight path of the drone based on the output of the image evaluation module. When the system detects an area where the reconstruction quality does not meet the standard, a new flight mission is generated to make the drone return to the area for additional photography to improve the reconstruction quality of the area.

4. The three-dimensional reconstruction method based on drone and edge computing according to claim 1, characterized in that: The specific method of step S5 is as follows: S5.1: Obtain each frame of the input high-definition video stream data and initialize it, create opaque and approximately transparent 3D Gaussian bodies, where the opaque Gaussian body is used to fit the surface and dominant color, and the transparent Gaussian body is used to fit the remaining color. S5.2: Explicitly add Gaussian volumes corresponding to the three types of pixels for each frame, and start Gaussian volume optimization based on the color loss and depth loss between the input and rendered RGB-D images, specifically: L=w c L color +w d L depth +W2L2 (7) Among them, L color is the color loss, L depth is the depth loss, which measures the actual color image C k Rendering color images Actual depth image D k Rendering depth image The absolute error of , and L2 is the L2 loss used to constrain the geometric properties of the transparent Gaussian body, and w corresponds to the weights of the three. S5.3: Use a real-time optimization scheme to optimize the Gaussian volume online to adapt to environmental changes and reduce memory and computing costs, and use a weighted average method to fuse the optimization results of the current time window with the previous results to avoid the forgetting problem. Specifically: or' o =(1-w curr )or o-1 +w curr or o (10) Among them, η o-1 is the optimization result of the previous frame, η o Optimize the result for the current frame, weight w curr Reflects the difference between the current frame and the historical frame, and finally determines the weighted average value η′ o . S5.4: After continuous optimization and fusion, a three-dimensional scene model containing high-precision geometric structure and color information is reconstructed and stored in the edge device.

Images