Three-dimensional reconstruction method, apparatus and device, and readable storage medium

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method, device and equipment and a readable storage medium, and electronic equipment carries out sparse reconstruction on a look-around image sequence and wheel speed meter information of each look-around image to obtain a sparse reconstruction result of each look-around image. And then, the electronic equipment reconstructs a static scene point cloud and a dynamic object point cloud of the target scene. And then, superposing the static scene point cloud of the target scene and the dynamic object point cloud of the target scene to obtain a three-dimensional reconstruction result of the target scene. By adopting the scheme, the electronic equipment obtains the monocular depth map of each look-around image by using the monocular depth estimation network, and reconstructs the dynamic object point cloud of the target scene according to the monocular depth map of each look-around image and the dynamic object mask, thereby realizing the reconstruction of the dynamic object in the target scene. Therefore, the quality of the three-dimensional reconstruction result of the target scene is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method, apparatus, device, and readable storage medium. Background Technology

[0002] Image-based 3D reconstruction refers to using a camera to photograph real-world objects and scenes, and then processing them using computer vision technology to obtain 3D models of the objects and scenes.

[0003] Image-based 3D reconstruction can be categorized based on the number of input views: monocular depth estimation, binocular stereo matching, and multi-view reconstruction. Figure Three 3D reconstruction. Among them, multi-view Figure Three The 3D reconstruction process includes: multi-view solid geometry, depth map estimation, point cloud fusion, mesh reconstruction and optimization, and texture mapping. Figure Three 3D reconstruction is widely used in various scenarios because it does not require image correction, has low image acquisition costs, and has a wide range of applications.

[0004] However, traditional multi-view Figure Three 3D reconstruction is mainly suitable for reconstructing static objects, and the results are poor for reconstructing dynamic objects. Summary of the Invention

[0005] This application provides a three-dimensional reconstruction method, apparatus, device, and readable storage medium. By using the dynamic object mask of a semantic segmentation network, static scene point clouds and dynamic scene point clouds are obtained by using a multi-view reconstruction network and a monocular depth estimation network, respectively. This overcomes the defect of difficulty in reconstructing dynamic objects in three-dimensional reconstruction and achieves the goal of improving the quality of three-dimensional reconstruction.

[0006] In a first aspect, embodiments of this application provide a three-dimensional reconstruction method, including:

[0007] Sparse reconstruction is performed on the surround view image sequence and the wheel speed meter information of each surround view image in the surround view image sequence to obtain the sparse reconstruction result. The surround view image sequence is the image captured by the mobile device traveling in the target scene using multiple surround view cameras.

[0008] Based on the panoramic image sequence and the sparse reconstruction results, reconstruct the static scene point cloud of the target scene;

[0009] Based on the surround view image sequence and the sparse reconstruction result, determine the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence, and reconstruct the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence.

[0010] superimpose the static scene point cloud of the target scene and the dynamic object point cloud of the target scene to obtain a three-dimensional reconstruction result of the target scene.

[0011] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction device, comprising:

[0012] a sparse reconstruction module configured to perform sparse reconstruction on a loop view image sequence and wheel speed meter information of each loop view image in the loop view image sequence, the loop view image sequence being images captured by a plurality of loop view cameras while a mobile device travels in a target scene;

[0013] a static scene reconstruction module configured to reconstruct a static scene point cloud of the target scene according to the loop view image sequence and the sparse reconstruction result;

[0014] a dynamic object reconstruction module configured to determine a monocular depth map and a dynamic object mask of each loop view image in the loop view image sequence according to the loop view image sequence and the sparse reconstruction result, and reconstruct a dynamic object point cloud of the target scene according to the monocular depth map and the dynamic object mask of each loop view image in the loop view image sequence;

[0015] a superimposition module configured to superimpose the static scene point cloud of the target scene and the dynamic object point cloud of the target scene to obtain a three-dimensional reconstruction result of the target scene.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to enable the electronic device to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer instructions, and the computer instructions are used to implement the method in the first aspect or any possible implementation manner of the first aspect when executed by a processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0019] The three-dimensional reconstruction method, device, equipment and readable storage medium provided by the embodiments of the present application, the electronic device performs sparse reconstruction on the loop view image sequence and the wheel speed meter information of each loop view image to obtain the sparse reconstruction result of each loop view image. Then, the electronic device reconstructs the static scene point cloud of the target scene according to the loop view image sequence and the sparse reconstruction result. At the same time, the monocular depth map and the dynamic object mask of each loop view image in the loop view image sequence are determined according to the loop view image sequence and the sparse reconstruction result, and the dynamic object point cloud of the target scene is reconstructed according to the monocular depth map and the dynamic object mask of each loop view image in the loop view image sequence. Then, the static scene point cloud of the target scene and the dynamic object point cloud of the target scene are superimposed to obtain the three-dimensional reconstruction result of the target scene. By using this scheme, the electronic device uses the monocular depth estimation network to obtain the monocular depth map of each loop view image, and reconstructs the dynamic object point cloud of the target scene according to the monocular depth map and the dynamic object mask of each loop view image, so as to realize the reconstruction of the dynamic object in the target scene, and further realize the purpose of improving the quality of the three-dimensional reconstruction result of the target scene. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a traditional multi-view image-based 3D reconstruction flowchart; Figure Three is a three-dimensional reconstruction method provided by the embodiments of the present application;

[0022] Figure 2 is a network architecture diagram suitable for the three-dimensional reconstruction method provided by the embodiments of the present application;

[0023] Figure 3 is a flowchart of the three-dimensional reconstruction method provided by the embodiments of the present application;

[0024] Figure 4 is a three-dimensional reconstruction method provided by the embodiments of the present application;

[0025] Figure 5A is a flowchart of sparse reconstruction in the three-dimensional reconstruction method provided by the embodiments of the present application;

[0026] Figure 5B is a process diagram of sparse reconstruction in the three-dimensional reconstruction method provided by the embodiments of the present application;

[0027] Figure 6A is a flowchart of static scene point cloud reconstruction in the three-dimensional reconstruction method provided by the embodiments of the present application;

[0028] Figure 6B This is a schematic diagram of the static scene point cloud reconstruction process in the three-dimensional reconstruction method provided in this application embodiment;

[0029] Figure 7A This is a flowchart of dynamic object point cloud reconstruction in the 3D reconstruction method provided in the application embodiment;

[0030] Figure 7B This is a schematic diagram of the static scene point cloud reconstruction process in the three-dimensional reconstruction method provided in this application embodiment;

[0031] Figure 8 This is a schematic diagram of the dense point cloud overlay process in the three-dimensional reconstruction method provided in this application embodiment;

[0032] Figure 9 A schematic diagram of a three-dimensional reconstruction device provided in an embodiment of this application;

[0033] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] For a long time, 3D reconstruction technology has been one of the core issues in the field of computer vision. It has broad prospects in applications such as augmented reality, mixed reality, autonomous driving, robotics, and large-scale map building. Based on the type of sensor used, 3D reconstruction technology can be divided into active and passive 3D reconstruction. Active 3D reconstruction utilizes depth sensors to actively acquire depth information; common depth sensors include LiDAR and structured light devices. Active 3D reconstruction has certain limitations in its applicable scenarios and usually requires expensive hardware. In contrast, passive 3D reconstruction typically only requires a camera, has a wider range of applicable scenarios, and therefore receives considerable attention and favor from industry and academia. Passive 3D reconstruction, also known as image-based 3D reconstruction, refers to the technique of using a camera to photograph real-world objects and scenes, and then processing them using computer vision technology to obtain 3D models of the objects and scenes.

[0035] Traditional image-based 3D reconstruction methods use principles such as photometric consistency and geometric consistency to compute dense 3D point clouds. Under ideal Lambertian scenes, these methods have achieved high accuracy. However, traditional methods have limitations when dealing with scenes with weak textures, high reflectivity, and repetitive textures, such as difficulty in reconstruction or incomplete reconstruction results. To overcome these problems, deep learning has achieved great success in 3D reconstruction in recent years. Learning-based methods can incorporate global semantic information, such as specular priors and reflection priors, making matching more robust. By learning from large amounts of data and feature representations, deep learning methods can better handle scenes with weak textures, high reflectivity, and repetitive textures, thereby improving the quality and accuracy of reconstruction.

[0036] Clearly, these learning-based methods have brought new breakthroughs and advancements to 3D reconstruction. They not only provide more complete reconstruction results but also handle various complex scenes and materials. Therefore, learning-based methods have received widespread attention and application in the field of 3D reconstruction, providing powerful tools and techniques for achieving more accurate and comprehensive 3D reconstruction.

[0037] Image-based 3D reconstruction can be categorized based on the number of input views: monocular depth estimation, binocular stereo matching, and multi-view reconstruction. Figure Three 3D reconstruction. Monocular depth estimation methods typically use only a single camera, limiting their field of view and perspective. When faced with complex, large scenes, they cannot capture all visual information, resulting in insufficient image acquisition and inadequate field-of-view coverage, leading to poor reconstruction quality. Furthermore, if the depth map generated by monocular depth estimation does not apply geometric consistency constraints based on multiple views, it will also result in poor 3D reconstruction quality. Multi-view... Figure Three The input image for 3D reconstruction is multiple monocular images. Depth map prediction is performed through similarity search between multiple views. Figure Three 3D reconstruction, also known as multi-view stereo matching, is widely used in the reconstruction of 3D models in various scenarios because images obtained through multi-view stereo matching do not require correction, have low image acquisition costs, and are widely applicable.

[0038] Figure 1 It is a traditional image-based multiview. Figure Three The reconstruction flowchart includes the following steps:

[0039] 101. Perform dense reconstruction based on the image and pose to obtain a depth map.

[0040] 102. Perform point cloud fusion based on the depth map to obtain a dense point cloud.

[0041] 103. Perform mesh reconstruction and optimization on the dense point cloud to obtain a polygonal mesh.

[0042] 104. Apply texture mapping to the polygonal mesh to obtain a 3D model.

[0043] according to Figure 1 It can be seen that: traditional multi-view Figure Three The main technologies involved in 3D reconstruction include multi-view solid geometry, depth map estimation, point cloud fusion, mesh reconstruction and optimization, and texture mapping.

[0044] Although image-based multi-view Figure Three 3D reconstruction has a wide range of applications; however, it has been verified that due to multi-view... Figure Three3D reconstruction requires geometric consistency between multiple views, but dynamic objects do not meet this constraint, resulting in low point cloud completeness for dynamic objects, which in turn leads to problems with image-based multiview reconstruction. Figure Three 3D reconstruction yields poor results for dynamic objects.

[0045] Moreover, traditional multi-view Figure Three The sparse reconstruction technique obtains the camera pose through the motion recovery structure algorithm. However, since no real-scale constraints are applied, the size of the camera pose obtained after sparse reconstruction is inconsistent with the actual size.

[0046] In addition, multi-view based on deep learning Figure Three Multi-view reconstruction methods have certain limitations on the scene or environment. For example, different scenes, such as indoor scenes, outdoor scenes, small scenes, and large scenes, have different depth scales and image features. Therefore, many multi-view reconstruction methods based on deep learning... Figure 2 The dimensional reconstruction method requires constructing a dataset containing dense depth ground truth values ​​according to the specific scenario, in order to train the model and adjust the parameters, which greatly increases the application cost and lacks versatility and scalability.

[0047] Based on this, embodiments of this application provide a three-dimensional reconstruction method, apparatus, device, and readable storage medium. The method utilizes a monocular depth estimation network to obtain a monocular depth map of each panoramic image, and reconstructs the dynamic object point cloud of the target scene based on the monocular depth map of each panoramic image and the dynamic object mask, thereby realizing the reconstruction of dynamic objects in the target scene and improving the quality of the three-dimensional reconstruction results of the target scene.

[0048] Figure 2 This is a schematic diagram of the network architecture applicable to the three-dimensional reconstruction method provided in this application embodiment. Please refer to... Figure 2 The network architecture includes: terminal device 21 and server 22, and a network connection is established between terminal device 21 and server 22.

[0049] Terminal device 21 is, for example, a self-moving device or a vehicle. When terminal device 21 is a self-moving device, it is a mechanical device capable of highly autonomous spatial movement within a specified environmental space, such as a robot or an unmanned vehicle. The robot can be a household robot, such as a robotic vacuum cleaner or a home companion robot; or it can be a commercial robot, such as a patrol robot or a guide robot used in shopping malls, supermarkets, banks, hospitals, airports, or train stations.

[0050] In this embodiment, the terminal device 21 has a mobility device, enabling it to move autonomously within its current environment. Additionally, the terminal device 21 includes various environmental sensors and inertial navigation sensors. The environmental sensors include, but are not limited to, surround-view cameras; the inertial navigation sensors include, but are not limited to, inertial measurement units (IMUs) and wheel speedometers. The number of surround-view cameras is, for example, 3, 4, or 8, and this embodiment is not limited. These surround-view cameras can capture 360-degree images of the surrounding environment of the terminal device 21 without blind spots. For example, the terminal device 21 may have one surround-view camera at each of its front, back, left, and right sides, for a total of four surround-view cameras. Each surround-view camera has a field of view of 150 degrees, and the fields of view of adjacent cameras overlap.

[0051] Server 22 can be either hardware or software. When server 22 is hardware, it can be a single server or a distributed server cluster consisting of multiple servers. When server 22 is software, it can be multiple software modules or a single software module, etc., and the embodiments of this application are not limited thereto.

[0052] During the 3D reconstruction process, as the terminal device 21 moves within the target scene, it simultaneously acquires surround-view images using surround-view cameras and wheel speed information using wheel speed meters. That is, the information acquired at each acquisition moment includes both the surround-view image and the wheel speed meter information for that moment. Since the terminal device 21 has multiple surround-view cameras, there are multiple surround-view images at the same acquisition moment. For example, if the terminal device 21 has four surround-view cameras, four surround-view images will be acquired simultaneously.

[0053] It should be noted that, since multiple surround-view cameras may not be perfectly synchronized, in this embodiment, multiple surround-view images at the same moment refer to images captured by multiple surround-view cameras within a time period around that moment. For example, if the acquisition time is exactly 10:00 AM, allowing for a 0.1-second error, the surround-view images captured by four surround-view cameras within that time period are considered multiple surround-view images at the same moment. These surround-view images are also referred to as images synchronized by time software.

[0054] After a period of time, i.e., after multiple acquisitions, the terminal device 21 generates a surround view image sequence based on the surround view images from multiple acquisition times, and sends the surround view image sequence and wheel speed meter information from each acquisition time to the server 22. The server 22 then performs 3D reconstruction based on the surround view image sequence and wheel speed meter information. Alternatively, after each surround view image acquisition, the terminal device 21 sends the surround view image and the wheel speed meter information for that moment to the server 22, and the server 22 generates a surround view image sequence based on the surround view images from each acquisition time.

[0055] Server 22 performs sparse reconstruction based on the surround view image sequence and wheel speed measurement information at each acquisition time to obtain the sparse reconstruction result for each surround view image. Then, based on the surround view image sequence and the sparse reconstruction result, server 22 reconstructs the static scene point cloud and dynamic object point cloud of the target scene and overlays them to obtain the 3D reconstruction result of the target scene. This 3D reconstruction result is a dense point cloud capable of reproducing the virtual target scene.

[0056] It is understandable that the panoramic image sequence is a sequence generated by installing multiple acquisition times sequentially, meaning the length of the panoramic image sequence is related to the number of acquisition times. The number of panoramic images at the same acquisition time is related to the number of panoramic cameras. For example, if there are 1000 acquisition times, and 4 panoramic images are acquired at each acquisition time, then the panoramic image sequence contains 4000 panoramic images.

[0057] In this embodiment, if the terminal device 21 travels along the boundary of the target scene for one full circle, that is, the trajectory of the terminal device 21 forms a closed loop, then the entire target scene can be reconstructed in three dimensions based on the panoramic image sequence and wheel speed information obtained from one full circle. If the trajectory of the terminal device 21 is not closed, that is, the terminal device 21 travels in a local space, then the local space of the target scene is reconstructed in three dimensions.

[0058] In this embodiment, the 3D reconstruction result obtained is a dense point cloud, which can be used for 4D annotation, navigation, etc. For example, after obtaining the 3D reconstruction result, 4D annotation is performed on the 3D reconstruction result to obtain annotation data. This annotation data is used to supervise the training process of models such as object detection models, multi-view reconstruction networks, and monocular depth estimation networks. The 4D annotation includes 3D annotation and temporal dimension annotation. Furthermore, after obtaining the 3D reconstruction result, it is deployed on terminal device 21. When terminal device 21 re-enters the target scene, it uses the 3D reconstruction result for navigation, obstacle avoidance, etc.

[0059] It should be understood that, Figure 2 The number of terminal devices 21 and servers 22 shown is merely illustrative. In actual implementation, any number of terminal devices 21 and servers 22 can be deployed according to actual needs.

[0060] In addition, although Figure 3 Server 22 is deployed in the application. However, this embodiment is not limited thereto. For example, when terminal device 21 has strong data processing capabilities, the above-mentioned 3D reconstruction can also be performed by terminal device 21. In this way, even if terminal device 21 is not connected to the network, it can perform 3D reconstruction of the target scene anytime and anywhere.

[0061] Figure 4This is a flowchart of a three-dimensional reconstruction method provided in an embodiment of this application. The executing entity of this embodiment is, for example, the aforementioned terminal device or server, or other electronic equipment. This embodiment includes:

[0062] 301. Perform sparse reconstruction on the surround view image sequence and the wheel speed meter information of each surround view image in the surround view image sequence to obtain sparse reconstruction results.

[0063] The surround view image sequence consists of images captured by a mobile device as it moves through the target scene using multiple surround view cameras.

[0064] In this embodiment, the terminal device is equipped with surround-view cameras, wheel speedometers, etc. For example, one surround-view camera is installed at the front, back, left, and right of the terminal device, for a total of four surround-view cameras. Alternatively, six surround-view cameras can be installed on the terminal device, enabling 360-degree coverage of the surrounding environment.

[0065] As the terminal device moves within the target scene, it captures panoramic images using a surround-view camera and collects wheel speed data using a wheel speed meter. By acquiring panoramic images from multiple perspectives using the surround-view camera, comprehensive observation is ensured, thereby guaranteeing a more complete and dense reconstruction result.

[0066] In this step, the electronic device acquires surround view images from the terminal device at multiple acquisition times and generates a surround view image sequence. Alternatively, the terminal device generates the surround view image sequence, and the electronic device directly acquires the surround view image sequence from the terminal device.

[0067] After acquiring the panoramic image sequence and wheel speed meter information at each acquisition time, the electronic device performs sparse reconstruction based on the panoramic image sequence and wheel speed meter information to obtain the sparse reconstruction result for each panoramic image. The sparse reconstruction structure includes camera intrinsic parameters, camera pose, depth range, and image matching pair information, etc.

[0068] 302. Based on the panoramic image sequence and the sparse reconstruction result, reconstruct the static scene point cloud of the target scene.

[0069] For example, the electronic device uses the surround view image sequence and sparse reconstruction results as input to the multi-view reconstruction network, and the surround view image sequence as input to the semantic segmentation network. Based on the output of the semantic segmentation network and the output of the multi-view reconstruction network, it reconstructs the static scene point cloud of the target scene.

[0070] 303. Based on the surround view image sequence and the sparse reconstruction result, determine the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence, and reconstruct the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence.

[0071] For example, an electronic device uses a sequence of panoramic images as input to a monocular depth estimation network, and uses the output of the monocular depth estimation network and sparse reconstruction results to reconstruct a dynamic object scene point cloud of the target scene.

[0072] It should be noted that there is no strict order between steps 302 and 303 above.

[0073] 304. Superimpose the static scene point cloud and the dynamic object point cloud of the target scene to obtain the three-dimensional reconstruction result of the target scene.

[0074] For example, an electronic device integrates static scene point clouds and dynamic object point clouds through a point cloud overlay algorithm to obtain a complete dense point cloud reconstruction result of the target scene, that is, to obtain a three-dimensional reconstruction result of the target scene.

[0075] The 3D reconstruction method provided in this application involves an electronic device performing sparse reconstruction on a sequence of surround-view images and the wheel speedometer information of each surround-view image to obtain sparse reconstruction results for each surround-view image. Then, the electronic device reconstructs a static scene point cloud of the target scene based on the surround-view image sequence and the sparse reconstruction results. Simultaneously, based on the surround-view image sequence and the sparse reconstruction results, the monocular depth map and dynamic object mask of each surround-view image in the sequence are determined, and the dynamic object point cloud of the target scene is reconstructed based on the monocular depth map and dynamic object mask of each surround-view image. Finally, the static scene point cloud and the dynamic object point cloud of the target scene are superimposed to obtain the 3D reconstruction result of the target scene. Using this scheme, the electronic device utilizes a monocular depth estimation network to obtain the monocular depth map of each surround-view image, and reconstructs the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each surround-view image, thereby achieving the reconstruction of dynamic objects in the target scene and improving the quality of the 3D reconstruction result of the target scene.

[0076] The 3D reconstruction methods provided in this application include sparse reconstruction, static scene point cloud reconstruction of the target scene, dynamic object point cloud reconstruction of the target scene, and dense point cloud overlay. For example, please refer to... Figure 4 , Figure 4 This is an overall flowchart of the three-dimensional reconstruction method provided in the embodiments of this application.

[0077] Please refer to Figure 5A The entire process is divided into four parts: sparse reconstruction, static scene point cloud reconstruction, dynamic object point cloud reconstruction, and dense point cloud overlay. The semantic segmentation network is related to both static scene point cloud reconstruction and dynamic object point cloud reconstruction. For clarity, the semantic segmentation network is not categorized under either static scene point cloud reconstruction or dynamic object point cloud reconstruction.

[0078] The terminal device is equipped with multiple surround-view cameras, which acquire a sequence of surround-view images from these cameras. Simultaneously, a wheel speed meter is installed on the terminal device to obtain wheel speed information corresponding to each surround-view image.

[0079] During sparse reconstruction, the electronic device performs sparse reconstruction based on the structure-of-motion (SOG) algorithm to obtain sparse reconstruction results such as camera parameters. During sparse reconstruction, the electronic device performs scaled sparse reconstruction using the panoramic image sequence and wheel speed sensor information, outputting sparse reconstruction results such as camera intrinsic parameters, camera pose, and depth range for each panoramic image.

[0080] Simultaneously, the electronic device inputs each panoramic image in the panoramic image sequence into the semantic segmentation model, so that the semantic segmentation model outputs a dynamic object mask for each panoramic image. The semantic segmentation model can be, for example, a Segment Anything Model (SAM), etc., and this application embodiment is not limited to any particular model.

[0081] During the static scene point cloud reconstruction process, the electronic device reconstructs the static scene point cloud of the target scene based on the multi-view reconstruction network and the aforementioned dynamic object mask.

[0082] During the dynamic object point cloud reconstruction process, the electronic device reconstructs the dynamic object point cloud of the target scene based on a monocular depth estimation network and the aforementioned dynamic object mask.

[0083] During the dense point cloud overlay process, the electronic device integrates the static scene point cloud and dynamic object point cloud of the target scene through a point cloud overlay algorithm, thereby obtaining a complete dense point cloud result of the target scene.

[0084] The following sections will provide detailed explanations of the sparse reconstruction, static scene point cloud reconstruction of the target scene, dynamic object point cloud reconstruction of the target scene, and dense point cloud overlay.

[0085] First, sparse reconstruction.

[0086] Figure 5B This is a flowchart of sparse reconstruction in the 3D reconstruction method provided in the application embodiment, including the following steps:

[0087] 501. Determine the matching image for each around view image in the around view image sequence to obtain an image matching set, wherein the same around view image has at least one image matching pair.

[0088] Figure 5B This is a schematic diagram illustrating the sparse reconstruction process in the 3D reconstruction method provided in this application. Please refer to... Figure 1The electronic device acquires a sequence of panoramic images using a surround-view camera and obtains the robot pose corresponding to each panoramic image using a wheel speed sensor. Then, the electronic device inputs the panoramic image sequence and the robot pose of each panoramic image into a global retrieval network, enabling the global retrieval network to determine the matching image for each panoramic image in the sequence, thus obtaining an image matching set. The global retrieval network can be, for example, an EigenPlaces network.

[0089] In this embodiment, the same round-view image has at least one image matching pair. For example, the round-view image sequence contains 20,000 round-view images, numbered sequentially as follows: Figure 2 , Figure 1 ...Figure 20000. Figure 2 and Figure 1 , Figure 3 and Figure 1 , Figure 1 And Figure 1000, Figure 1 And Figure 1001, Figure 5B If both of them match Figure 1002, then these 5 pairs of images form 5 image matching pairs.

[0090] 502. Determine the key points of each panoramic image in the panoramic image sequence.

[0091] Please refer to the following at the same time Figure 5B The electronic device inputs each panoramic image in the panoramic image sequence into a feature extraction network to obtain key points for each panoramic image. The feature extraction network is also known as a feature extraction model, such as the ALIKED model, and this application embodiment is not limited to this.

[0092] 503. For each image matching pair in the image matching set, a key point matching map is determined based on the key points of each panoramic image in the image matching pair to obtain a key point matching map set.

[0093] Please refer to the following at the same time Figure 1 For each image matching pair in the image matching set, the electronic device generates a key point matching map based on the key points of the two panoramic images in the image matching pair, and then generates a key point matching map set.

[0094] Continuing with the example above, Figure 2 and Figure 1 , Figure 3 and Figure 1 , Figure 1 And Figure 1000, Figure 1 And Figure 1001, Figure 1 Both match Figure 1002. Figure 3 and Figure 1Taking this image matching pair as an example, the electronic device inputs the key points of the image matching pair into a feature matching network to obtain a key point matching map. Clearly, for... Figure 5B Since there are 5 image matching pairs, 5 keypoint matching maps can be obtained. The keypoint matching maps of all the panoramic images form a keypoint matching map set. The feature matching network is, for example, a deep matcher (Light Glue), etc., and this application embodiment is not limited.

[0095] 504. Based on the key point matching map set of each panoramic image in the panoramic image sequence and the wheel speed meter information, determine the sparse reconstruction result of the corresponding panoramic image.

[0096] Please refer to the following at the same time Figure 1 The electronic device obtains camera intrinsics, robot pose, depth range, and image matching pair information for each panoramic image through keypoint matching atlases, robot pose, and structure-of-motion (SOP) algorithms. Examples of SOP algorithms include the COLMAP algorithm. Robot pose can be obtained from wheel speed readings.

[0097] Continuing with the above example, regarding Figure 1 There are a total of 5 keypoint matching images. Therefore, the electronic device takes pictures based on these 5 keypoint matching images. Figure 1 The camera pose and motion recovery structure algorithm of the panoramic image are used to obtain... Figure 1 The sparse reconstruction results.

[0098] In this embodiment, the robot pose is used to recover the true scale of the camera pose, and a translation vector magnitude ||T| is obtained through wheel speed meter information. odom The magnitude of another translation vector, ||T, is obtained through the motion recovery structure algorithm. sfm The pose scale factor s can be determined according to the following formula (1):

[0099]

[0100] This approach utilizes wheel velocity sensor information to acquire the robot's pose, ensuring that the camera pose dimensions obtained through the structure-of-motion (SOP) algorithm are consistent with the actual values, thus achieving denser reconstruction results with better scale consistency. Furthermore, by acquiring surround view image sequences from cameras in multiple orientations, the completeness of the 3D reconstruction results is improved.

[0101] Optionally, in the above embodiments, after obtaining the image matching set, the electronic device further determines the camera pose for capturing the panoramic images based on the wheel speed meter information of each panoramic image in the panoramic image sequence. Then, based on the camera pose of each panoramic image in the panoramic image sequence, the electronic device identifies and deletes incorrect matching pairs from multiple image matching pairs of the panoramic images, wherein the difference in camera pose between the two panoramic images in the incorrect matching pair is greater than a preset difference.

[0102] For example, camera pose is used to indicate the position, orientation, etc., of the self-moving device when capturing a surround view image. Two mutually matching surround view images have similar camera poses, meaning they are close in distance and have the same orientation. If the distance between the self-moving device determined from the two camera poses is greater than a preset threshold, it means the two matching surround view images do not match and are an incorrect match pair. For example, different locations in a garage may look similar. Figure 1 Match with Figure 98 and take a picture. Figure 2 Location and shooting of mobile devices Figure 1 If the distance between the mobile device and the target location is greater than 5 meters, for example, 8 meters, then the image matching pair is considered a mismatch. For example, Figure 1 Match with Figure 200, take a picture Figure 2 The angle and shooting of the mobile device Figure 10 If the difference in the angles of the mobile device is greater than the preset angle, it means that the two images were taken by the panoramic camera when the mobile device was facing different directions, and this image matching pair is a mismatch pair.

[0103] Furthermore, because wheel speed sensors can drift during extended travel of the mobile device, relying solely on camera pose to determine whether an image pair is an incorrect match could easily lead to misjudgments. Therefore, surrounding images can also be considered. For example, Figure 10 It matches Figure 980, and the two shooting positions are very close together. At this point, the electronic device further confirms... Figure 9 Nearby images, i.e. Figure 10 Check whether Figure 11, etc., matches Figure 980. If one or more surrounding images do not match Figure 980, it indicates... Figure 6A Figure 980 is a mismatch pair.

[0104] This approach allows electronic devices to identify and remove incorrect matching pairs from the image matching set before extracting key points, thereby improving the quality of the image matching set and ultimately enhancing the accuracy of 3D reconstruction.

[0105] Secondly, static scene point cloud reconstruction of the target scene.

[0106] Figure 6B This is a flowchart of static scene point cloud reconstruction in the 3D reconstruction method provided in the application embodiment, including the following steps:

[0107] 601. Based on the panorama image sequence and the sparse reconstruction result, determine the depth map of each panorama image in the panorama image sequence.

[0108] Figure 6B This is a schematic diagram illustrating the static scene point cloud reconstruction process in the 3D reconstruction method provided in this application embodiment. Please refer to... Figure 6B The electronic device inputs the panoramic images and their reconstruction results from the panoramic image sequence into the multi-view reconstruction network, so that the multi-view reconstruction network outputs a depth map D for each panoramic image. mvs And confidence map C. The multi-view reconstruction network can be, for example, IGEV-MVS, etc., but this application embodiment is not limited to this. The reconstruction results of the panoramic image include camera intrinsic parameters, camera pose, depth range, and image pairing information, etc.

[0109] 602. Determine the dynamic object mask for each panoramic image in the panoramic image sequence.

[0110] Please install simultaneously Figure 6B The electronic device inputs the round-view images from the round-view image sequence into a semantic segmentation network to obtain the dynamic object mask M for each round-view image. dynamic As shown in formula (2) below:

[0111]

[0112] 603. Generate a static scene depth map for each panoramic image based on the depth map and dynamic object mask of each panoramic image in the panoramic image sequence.

[0113] Please install simultaneously Figure 6B For each panoramic image, the electronic device uses the dynamic object mask M of that panoramic image. dynamic Perform a logical NOT operation, then compare it with the depth map D of the panoramic image. mvs Perform a logical AND operation to remove depth map D. mvs The depth of the dynamic object region in the panoramic image is used to obtain the static scene depth map D. static As shown in formula (3) below:

[0114]

[0115] 604. Based on the static scene depth map of each panoramic image in the panoramic image sequence, reconstruct the single-frame static scene point cloud of the corresponding panoramic image.

[0116] 605. Fuse the single-frame static scene point cloud of each panoramic image in the panoramic image sequence to generate the static scene point cloud of the target scene.

[0117] For each panoramic image, the static scene depth map D static The electronic device will display the static scene depth map D. static A single-frame static scene point cloud is generated by transforming the camera coordinate system to the world coordinate system. Then, the single-frame static scene point clouds of all the panoramic images are superimposed to obtain the static scene point cloud PC of the target scene. static .

[0118] This approach allows electronic devices to quickly and accurately reconstruct static scene point clouds by combining dynamic object masks obtained through semantic segmentation networks with depth maps obtained through multi-view reconstruction networks.

[0119] Optional, please also refer to Figure 1 For each panoramic image, the static scene depth map D static The electronic device also performs at least one of the following checks: geometric consistency, near-field consistency, and photometric consistency. Based on these checks, a geometric mask M is generated. geo Near-field mask M close and luminance mask M photo Then, these masks are used to filter out outliers, resulting in the static scene depth map D after outlier filtering. static_filter Outer pixels, also known as bad pixels, abnormal pixels, or abnormal pixels, are not limited in the embodiments of this application. Electronic devices utilize the static scene depth map D... static A logical AND operation is performed with each mask to filter out outliers with inconsistent depths. This is shown in formula (4) below.

[0120] D static_filter =D static ∩M geo ∩M close ∩M photo Formula (4)

[0121] After various consistency checks, only valid points in the static scene depth map are retained. Taking a 720×1080 panoramic image as an example, after geometric consistency checks, near-field consistency checks, and / or photometric consistency checks, if one or more consistency check results are poor, only 5% of the pixels may be retained. If the various check results are good, 50%, 70%, or even higher percentages of pixels may be retained.

[0122] The geometric consistency test, near-field consistency test, and photometric consistency test described above will be explained in detail below.

[0123] A. Geometric consistency check.

[0124] In this embodiment of the application, for each panoramic image in the panoramic image sequence, the electronic device performs a geometric consistency check by projecting it twice. The geometric consistency check includes the following steps:

[0125] Step a: For each aroundview image in the aroundview image sequence, determine multiple reference images based on the matching images of the aroundview images.

[0126] For example, a roundabout image sequence contains 20,000 roundabout images, numbered sequentially as follows: Figure 2 , Figure 1 ...Figure 20000. Figure 2 and Figure 1 , Figure 3 and Figure 1 , Figure 1 And Figure 1000, Figure 1 And Figure 1001, Figure 1 If all five pairs of images match Figure 1002, then these five pairs of images form five image matching pairs. Therefore, for... Figure 2 In other words, Figure 3 , Figure 2 Figures 1000, 1001, and 1002 are all reference images.

[0127] Step b: The electronic device determines the first depth value of the first pixel on the panoramic image based on the static scene depth map of the panoramic image, and determines the first 3D point based on the first depth value.

[0128] For each pixel in the panoramic image, the electronic device performs a reprojection. The panoramic image is also called the source image. For example, the first pixel is the pixel with pixel coordinates (100, 100) in the panoramic image. The electronic device indexes the static scene depth map of the panoramic image based on this pixel coordinate to obtain the first depth value. Then, based on the sparse reconstruction results such as the camera intrinsic parameters corresponding to the panoramic image, the electronic device transforms the first depth value to the world coordinate system to obtain the first 3D point.

[0129] Step c: Project the first 3D point onto each of the plurality of reference images to obtain a plurality of second pixels, wherein each of the plurality of reference images is a matching image of the panorama image. This step is also known as projection.

[0130] Continuing with the example above, the electronic device projects the first 3D point onto... Figure 3 , Figure 2 In Figures 1000, 1001, and 1002, five second pixels are obtained.

[0131] Step d: Determine the second depth value of each second pixel point based on the static scene depth map of the first reference image, and determine the second 3D point based on the second depth value of each second pixel point to obtain multiple second 3D points.

[0132] Continuing with the example above, for Figure 2 The second pixel on the screen is indexed by the electronic device based on the pixel coordinates of the second pixel. Figure 2 The static scene depth map is used to obtain the second depth value. Then, the electronic device uses this data... Figure 3 The corresponding sparse reconstruction results, such as camera intrinsic parameters, are used to transform the second depth value into the world coordinate system to obtain the second 3D point.

[0133] Similarly, electronic devices Figure 2 The same operation was performed on Figures 1000, 1001, and 1002, resulting in a total of 5 second 3D points.

[0134] Step e: Project multiple second 3D points onto the panoramic image to obtain multiple third pixels. This step is also called back projection, and the two projections, projection and back projection, are also called reprojection.

[0135] In this step, the electronic device will Figure 3 , Figure 7A The second 3D points corresponding to Figures 1000, 1001, and 1002 are projected onto the panoramic image to obtain five third pixel points.

[0136] Step f: When the relationship between more than a preset number of third pixels and the first pixel satisfies a preset condition, the depth value corresponding to the first pixel in the static scene depth map of the panoramic image is retained.

[0137] In this embodiment, the preset conditions include, for example, a pixel offset of less than one pixel and a depth value difference of less than 1%. The electronic device calculates the distance between each third pixel and the first pixel, and calculates the difference between the depth value corresponding to each third pixel and the depth value corresponding to the first pixel. If more than a preset number of third pixels satisfy the preset conditions in relation to the first pixel, the depth value corresponding to the first pixel in the static scene depth map of the panoramic image is retained. If less than a preset number of third pixels satisfy the preset conditions in relation to the first pixel, the first pixel is filtered out from the static scene depth map. The preset number may be, for example, 3, 4, or 60% of the number of reference images, and is not limited in this embodiment.

[0138] In actual implementation, a geometric mask can be generated based on whether the relationship between the third pixel and the first pixel meets the preset conditions. The geometric mask can then be used to filter out inconsistent pixels in the static scene depth map.

[0139] This approach uses reprojection to perform geometric consistency checks on the depth map of a static scene, offering high reliability and speed.

[0140] B. Near-field consistency test.

[0141] In this embodiment, for the static scene depth map of each panoramic image in the panoramic image sequence, the electronic device generates a near-field mask based on the depth value of each pixel in the static scene depth map. Then, the electronic device uses the near-field mask to filter out pixels in the static scene depth map whose depth values ​​are greater than a preset depth value.

[0142] For example, since the depth values ​​of regions with large depth values ​​have low reliability, a near-field consistency check is applied to remove far-field regions with large depth values. During the filtering process, for each pixel in the static scene depth map, the electronic device determines whether the depth value of the pixel is greater than or equal to a preset depth value. If the depth value of the pixel is greater than or equal to the preset depth value, it indicates that the pixel is an outlier and needs to be filtered out. The preset depth value is, for example, 80 meters, 75 meters, etc., and this embodiment of the application is not limited.

[0143] The electronic device compares the depth value of each pixel with a preset depth value and generates a geometric mask. This geometric mask is used to filter out pixels with abnormal depth values ​​in the static scene depth map, thereby completing the near-field consistency check.

[0144] By adopting this approach, the electronic device filters out the far-field depth in the static scene depth map through near-field consistency testing, so that unreliable far-field depth will not form point cloud noise. This effectively reduces the noise rate of dense point clouds while improving the accuracy of 3D reconstruction.

[0145] C. Photometric consistency test.

[0146] In this embodiment, the electronic device generates a depth map for each panoramic image using a multi-view reconstruction network, and also generates a confidence map corresponding to the depth map. This confidence map indicates the confidence value of the depth value of each pixel in the depth map. The confidence value ranges from, for example, 0 to 1. Then, the electronic device generates a photometric mask based on the confidence value of each pixel in the confidence map, and uses the photometric mask to filter out pixels in the static scene depth map whose confidence value of the depth value is lower than a preset confidence value.

[0147] For example, the preset confidence value may be 0.7, 0.8, etc., and this application embodiment is not limited to this. If the confidence value of a pixel in the static scene depth map is less than or equal to the preset confidence value, it indicates that the pixel is unreliable and needs to be filtered out. The electronic device compares the confidence value of each pixel in the static scene depth map with the preset confidence value, and generates a photometric mask based on the comparison result. This photometric mask is used to filter out pixels in the static scene depth map whose depth value confidence value is lower than the preset confidence value, thereby completing the photometric consistency check.

[0148] Using this approach, electronic devices filter out pixels with low confidence in the static scene depth map through photometric consistency testing, thereby improving the reliability of 3D reconstruction results.

[0149] Next, dynamic object point cloud reconstruction of the target scene.

[0150] Figure 7B This is a flowchart of the dynamic object point cloud reconstruction method in the 3D reconstruction method provided in the application embodiment, including the following steps:

[0151] 701. Based on the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence, determine the dynamic object depth map of the corresponding panoramic image.

[0152] Figure 7B This is a schematic diagram illustrating the static scene point cloud reconstruction process in the 3D reconstruction method provided in this application embodiment. Please refer to... Figure 6A The electronic device inputs the panoramic images from the panoramic image sequence into a monocular depth estimation (Depth Anything) network to obtain a monocular depth map D for each panoramic image. mono Simultaneously, the electronic device determines the dynamic object mask M for each panoramic image. dynamic For details, please participate Figure 7B The description of step 602 will not be repeated here.

[0153] Subsequently, the electronic device generates a monocular depth map D for each panoramic image. mono and dynamic object mask M dynamic Perform a logical AND operation to obtain the dynamic object depth map D of the panoramic image. dynamic As shown in formula (5) below:

[0154] D dynamic =D mono ∩M dynamic Formula (5)

[0155] 702. Based on the dynamic object depth map of each panoramic image in the panoramic image sequence, obtain the single-frame dynamic object point cloud of the corresponding panoramic image.

[0156] Please refer to the following at the same timeFigure 8 The electronic device uses sparse reconstruction results, such as camera intrinsic parameters, to generate a depth map D of the dynamic object. dynamic Projecting onto the world coordinate system yields a single-frame dynamic object point cloud.

[0157] 703. Fuse the single-frame dynamic object point cloud of each panoramic image in the panoramic image sequence to generate the dynamic object point cloud of the target scene.

[0158] In this step, the electronic device overlays the single-frame dynamic object point clouds of all the panoramic images to obtain the dynamic object point cloud PC of the target scene. dynamic .

[0159] This approach allows electronic devices to obtain dynamic object masks through semantic segmentation networks. By using the dynamic object masks and monocular depth estimation networks, they can obtain dynamic object point clouds of the target scene, thus overcoming the limitation of traditional 3D reconstruction methods that cannot reconstruct dynamic objects. At the same time, they can obtain static scene point clouds of the target scene by using dynamic object masks and multi-view reconstruction networks, thereby obtaining 3D reconstruction results of the target scene. These 3D reconstruction results simultaneously include dynamic object point clouds and static scene maps, thereby improving the 3D reconstruction results.

[0160] Finally, dense point clouds are superimposed.

[0161] Figure 8 This is a schematic diagram illustrating the dense point cloud overlay process in the 3D reconstruction method provided in this application. Please refer to... Figure 9 Electronic devices will generate static scene point clouds of the target scene from PCs. static PC with dynamic object point cloud of target scene dynamic A dense point cloud is obtained by overlaying points using a point cloud overlay algorithm. Then, a statistical filtering denoising algorithm is used to remove noise from the point cloud. This noise-removed point cloud is the 3D reconstruction result of the target scene, which is a dense point cloud PC. The noise point cloud removal process is shown in formula (6):

[0162] PC = statis_filter(PC) static +PC dynamic )Formula (6)

[0163] Among them, statis_filter represents the statistical filtering point cloud denoising algorithm.

[0164] This approach uses a statistical filtering denoising algorithm to remove noisy point clouds, thereby improving the quality of 3D reconstruction.

[0165] Optionally, currently available multi-view reconstruction networks and monocular depth estimation networks are all small-scale, applicable only to small scenes and not to large scenes. Therefore, when the target scene is a large scene, before static scene point cloud reconstruction and dynamic object point cloud reconstruction, the electronic device trains a multi-view reconstruction network and a monocular depth estimation network using datasets of multiple scales. The multi-view reconstruction network is used to acquire the static scene point cloud of the target scene, and the monocular depth estimation network is used to acquire the dynamic object point cloud of the target scene.

[0166] For example, current 3D reconstruction models such as multi-view reconstruction networks and monocular depth estimation networks cannot be transferred to large scenes. If the target scene is one that multi-view reconstruction networks and monocular depth estimation networks are not suitable for, a training dataset with multiple scales and modalities is obtained before executing the 3D reconstruction method provided in this application. The multi-view reconstruction network and monocular depth estimation network are trained in stages. For example, the first stage is trained for small scenes. The second stage is trained for large scenes, and the third stage is trained for even larger scenes. Through a progressive training strategy, a 3D reconstruction model such as a multi-view reconstruction network and monocular depth estimation network that can adapt to small scenes, large scenes, indoor scenes, and outdoor scenes is obtained.

[0167] This approach trains a multi-view reconstruction network and a monocular depth estimation network by mixing datasets of different depth dimensions. This improves the robustness of both networks and allows them to adapt to different depth scenarios, such as indoor and outdoor environments.

[0168] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0169] Figure 9 This is a schematic diagram of a three-dimensional reconstruction device provided in an embodiment of this application. The three-dimensional reconstruction device 900 includes: a sparse reconstruction module 91, a static scene reconstruction module 92, a dynamic object reconstruction module 93, and an overlay module 94.

[0170] The sparse reconstruction module 91 is used to perform sparse reconstruction on the surround view image sequence and the wheel speed meter information of each surround view image in the surround view image sequence to obtain the sparse reconstruction result. The surround view image sequence is the image captured by the self-moving device in the target scene using multiple surround view cameras.

[0171] The static scene reconstruction module 92 is used to reconstruct the static scene point cloud of the target scene based on the surround view image sequence and the sparse reconstruction result;

[0172] The dynamic object reconstruction module 93 is used to determine the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence based on the panoramic image sequence and the sparse reconstruction result, and to reconstruct the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence.

[0173] The overlay module 94 is used to overlay the static scene point cloud and the dynamic object point cloud of the target scene to obtain the three-dimensional reconstruction result of the target scene.

[0174] In one feasible implementation, the dynamic object reconstruction module 93 is used to determine the dynamic object depth map of the corresponding panoramic image based on the monocular depth map of each panoramic image in the panoramic image sequence and the dynamic object mask; obtain the single-frame dynamic object point cloud of the corresponding panoramic image based on the dynamic object depth map of each panoramic image in the panoramic image sequence; and fuse the single-frame dynamic object point clouds of each panoramic image in the panoramic image sequence to generate the dynamic object point cloud of the target scene.

[0175] In one feasible implementation, the sparse reconstruction module 91 is used to determine the matching image of each around view image in the around view image sequence to obtain an image matching set, wherein the same around view image has at least one image matching pair; determine the key points of each around view image in the around view image sequence; for each image matching pair in the image matching set, determine a key point matching map based on the key points of each around view image in the image matching pair to obtain a key point matching map set; and determine the sparse reconstruction result of the corresponding around view image based on the key point matching map set of each around view image in the around view image sequence and the wheel speed meter information.

[0176] In one feasible implementation, before the sparse reconstruction module 91 determines the key points of each panoramic image in the panoramic image sequence, it is further configured to determine the camera pose of the panoramic image based on the wheel speed meter information of each panoramic image in the panoramic image sequence; based on the camera pose of each panoramic image in the panoramic image sequence, determine incorrect matching pairs from multiple image matching pairs of the panoramic images, wherein the difference between the camera poses of the two panoramic images in the incorrect matching pair is greater than a preset difference; and delete the incorrect matching pair from the multiple image matching pairs of the panoramic images.

[0177] In one feasible implementation, the static scene reconstruction module 92 is used to: determine the depth map of each panoramic image in the panoramic image sequence based on the panoramic image sequence and the sparse reconstruction result; determine the dynamic object mask of each panoramic image in the panoramic image sequence; generate a static scene depth map of the corresponding panoramic image based on the depth map and the dynamic object mask of each panoramic image in the panoramic image sequence; reconstruct a single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence; and fuse the single-frame static scene point clouds of each panoramic image in the panoramic image sequence to generate a static scene point cloud of the target scene.

[0178] In one feasible implementation, before the static scene reconstruction module 92 reconstructs the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, it is further used to generate a near-field mask based on the depth value of each pixel in the static scene depth map; and use the near-field mask to filter out pixels in the static scene depth map whose depth value is greater than a preset depth value.

[0179] In one feasible implementation, before the static scene reconstruction module 92 reconstructs the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, it is further configured to determine multiple reference images for each panoramic image in the panoramic image sequence based on the matching image of the panoramic image; determine the first depth value of the first pixel point on the panoramic image based on the static scene depth map of the panoramic image; determine the first 3D point based on the first depth value; and project the first 3D point onto each of the multiple reference images to obtain... Multiple second pixels are obtained, and each of the multiple reference images is a matching image of the panoramic image. A second depth value is determined based on the static scene depth map of each reference image, and a second 3D point is determined based on the second depth value of each second pixel to obtain multiple second 3D points. These multiple second 3D points are projected onto the panoramic image to obtain multiple third pixels. When the relationship between more than a preset number of third pixels and the first pixel satisfies a preset condition, the depth value corresponding to the first pixel in the static scene depth map of the panoramic image is retained.

[0180] In one feasible implementation, before the static scene reconstruction module 92 reconstructs the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, it is further used to determine the confidence map corresponding to the depth map of each panoramic image in the panoramic image sequence. The confidence map is used to indicate the reliability of the depth value of each pixel in the depth map; a photometric mask is generated based on the reliability of each pixel in the confidence map; and the photometric mask is used to filter out pixels in the static scene depth map whose depth value reliability is lower than a preset reliability.

[0181] Determine the confidence map corresponding to the depth map of each panoramic image in the panoramic image sequence, wherein the confidence map is used to indicate the reliability of the depth value of each pixel in the depth map;

[0182] A photometric mask is generated based on the reliability of each pixel in the confidence map;

[0183] The photometric mask is used to filter out pixels whose depth values ​​in the static scene depth map have a reliability lower than a preset reliability.

[0184] Optional, please refer to Figure 10 In one feasible implementation, the aforementioned three-dimensional reconstruction device 900 further includes:

[0185] Training module 95 is used to train a multi-view reconstruction network and a monocular depth estimation network by mixing datasets of multiple scales. The multi-view reconstruction network is used to obtain the static scene point cloud of the target scene, and the monocular depth estimation network is used to obtain the dynamic object point cloud of the target scene.

[0186] The three-dimensional reconstruction device provided in this application embodiment can perform the actions of the electronic device in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0187] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Please refer to... Figure 10 The electronic device 1000 described in this application includes: at least one processor 1001, at least one communication bus 1002, a user interface 1003, at least one network interface 1004, and a memory 1005.

[0188] The communication bus 1002 is used to realize the connection and communication between these components.

[0189] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. The display screen is used to display the editing interface, roaming interface, etc.

[0190] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0191] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the panoramic sphere required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 1001.

[0192] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure OneAs shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and operating applications for electronic devices.

[0193] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the three-dimensional reconstruction method implemented by the above-mentioned electronic device.

[0194] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the three-dimensional reconstruction method implemented by the above-mentioned electronic device.

[0195] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure One One or more processes and / or boxes Figure One A device that provides the functions specified in one or more boxes.

[0197] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The function specified in one or more boxes.

[0198] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure One One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0199] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0200] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0201] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0202] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0203] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Sparse reconstruction is performed on the surround view image sequence and the wheel speed meter information of each surround view image in the surround view image sequence to obtain the sparse reconstruction result. The surround view image sequence is the image captured by the mobile device traveling in the target scene using multiple surround view cameras. Based on the panoramic image sequence and the sparse reconstruction results, reconstruct the static scene point cloud of the target scene; Based on the surround view image sequence and the sparse reconstruction result, determine the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence, and reconstruct the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each surround view image in the surround view image sequence. The static scene point cloud and the dynamic object point cloud of the target scene are superimposed to obtain the three-dimensional reconstruction result of the target scene.

2. The method according to claim 1, characterized in that, The step of reconstructing the dynamic object point cloud of the target scene based on the monocular depth map of each panoramic image in the panoramic image sequence and the dynamic object mask includes: Based on the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence, determine the dynamic object depth map of the corresponding panoramic image. Based on the dynamic object depth map of each panoramic image in the panoramic image sequence, the single-frame dynamic object point cloud of the corresponding panoramic image is obtained. The dynamic object point cloud of the target scene is generated by fusing the single-frame dynamic object point cloud of each panoramic image in the panoramic image sequence.

3. The method according to claim 1, characterized in that, The sparse reconstruction of the surround view image sequence and the wheel speedometer information of each surround view image in the surround view image sequence to obtain the sparse reconstruction result includes: The matching image for each circumferential image in the circumferential image sequence is determined to obtain an image matching set, wherein the same circumferential image has at least one image matching pair; Identify the key points of each panoramic image in the panoramic image sequence; For each image matching pair in the image matching set, a keypoint matching map is determined based on the keypoints of each panoramic image in the image matching pair to obtain a keypoint matching map set; Based on the key point matching atlas of each panoramic image in the panoramic image sequence and the wheel speed meter information, the sparse reconstruction result of the corresponding panoramic image is determined.

4. The method according to claim 3, characterized in that, Before determining the key points of each panoramic image in the panoramic image sequence, the method further includes: The camera pose for capturing the panoramic images is determined based on the wheel speed meter information of each panoramic image in the panoramic image sequence. Based on the camera pose of each panoramic image in the panoramic image sequence, incorrect matching pairs are determined from multiple image matching pairs of the panoramic images, wherein the difference between the camera poses of the two panoramic images in the incorrect matching pair is greater than a preset difference. The incorrect matching pairs are deleted from the multiple image matching pairs of the panoramic image.

5. The method according to any one of claims 1 to 4, characterized in that, The step of reconstructing the static scene point cloud of the target scene based on the surround view image sequence and the sparse reconstruction result includes: Based on the surround view image sequence and the sparse reconstruction results, determine the depth map of each surround view image in the surround view image sequence; Determine the dynamic object mask for each panoramic image in the panoramic image sequence; Based on the depth map and dynamic object mask of each panoramic image in the panoramic image sequence, a static scene depth map of the corresponding panoramic image is generated. Based on the static scene depth map of each panoramic image in the panoramic image sequence, the single-frame static scene point cloud of the corresponding panoramic image is reconstructed. The static scene point cloud of the target scene is generated by fusing the single-frame static scene point cloud of each panoramic image in the panoramic image sequence.

6. The method according to claim 5, characterized in that, Before reconstructing the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, the method further includes: A near-field mask is generated based on the depth value of each pixel in the static scene depth map; The near-field mask is used to filter out pixels in the static scene depth map whose depth value is greater than a preset depth value.

7. The method according to claim 5, characterized in that, Before reconstructing the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, the method further includes: For each circumferential image in the circumferential image sequence, multiple reference images are determined based on the matching images of the circumferential images; Based on the static scene depth map of the panoramic image, determine the first depth value of the first pixel on the panoramic image, and determine the first 3D point based on the first depth value; The first 3D point is projected onto each of the multiple reference images to obtain multiple second pixels, wherein each of the multiple reference images is a matching image of the panoramic image; Based on the static scene depth map of each reference image, determine the second depth value of the corresponding second pixel, and determine the second 3D point based on the second depth value of each second pixel to obtain multiple second 3D points; Multiple second 3D points are projected onto the panoramic image to obtain multiple third pixel points; If the relationship between more than a preset number of third pixels and the first pixel satisfies a preset condition, then the depth value corresponding to the first pixel in the static scene depth map of the panoramic image is retained.

8. The method according to claim 5, characterized in that, Before reconstructing the single-frame static scene point cloud of the corresponding panoramic image based on the static scene depth map of each panoramic image in the panoramic image sequence, the method further includes: Determine the confidence map corresponding to the depth map of each panoramic image in the panoramic image sequence, wherein the confidence map is used to indicate the reliability of the depth value of each pixel in the depth map; A photometric mask is generated based on the reliability of each pixel in the confidence map; The photometric mask is used to filter out pixels whose depth values ​​in the static scene depth map have a reliability lower than a preset reliability.

9. The method according to any one of claims 1 to 4, characterized in that, Also includes: A multi-view reconstruction network and a monocular depth estimation network are trained using datasets of multiple scales. The multi-view reconstruction network is used to acquire the static scene point cloud of the target scene, and the monocular depth estimation network is used to acquire the dynamic object point cloud of the target scene.

10. A three-dimensional reconstruction device, characterized in that, include: The sparse reconstruction module is used to perform sparse reconstruction on the surround view image sequence and the wheel speed meter information of each surround view image in the surround view image sequence to obtain the sparse reconstruction result. The surround view image sequence is the image captured by the mobile device traveling in the target scene using multiple surround view cameras. The static scene reconstruction module is used to reconstruct the static scene point cloud of the target scene based on the surround view image sequence and the sparse reconstruction result; The dynamic object reconstruction module is used to determine the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence and the sparse reconstruction result, and to reconstruct the dynamic object point cloud of the target scene based on the monocular depth map and dynamic object mask of each panoramic image in the panoramic image sequence. The overlay module is used to overlay the static scene point cloud and the dynamic object point cloud of the target scene to obtain the three-dimensional reconstruction result of the target scene.

11. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

Images