A method and apparatus for recovering three-dimensional information

Abstract

This application discloses a method and apparatus for restoring three-dimensional information. The technical solution provided in this application involves identifying a lost frame and a successfully reconstructed reference frame adjacent to the lost frame when reconstruction failure is determined in the reconstructed area; upsampling the reference frame and the lost frame; determining a first feature matching relationship between the reference frame and the lost frame based on the sampling results; and estimating the relative motion between the reference frame and the lost frame based on the first feature matching relationship to restore the pose and point cloud of the lost frame. By employing the above technical means, the pose and point cloud of the lost frame can be recovered using the reference frame, thus achieving the restoration of a lost scene. Furthermore, the upsampling method eliminates the need for re-measuring the image online, thereby improving the real-time performance of three-dimensional reconstruction, efficiently restoring lost scenes in the event of scene loss, and enhancing the user's experience with three-dimensional reconstruction information.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for recovering three-dimensional information. Background Technology

[0002] 3D reconstruction refers to the process of reconstructing 3D information from single-view or multi-view images. Currently, in real-time 3D reconstruction scenarios, when the reconstructed scene has severe occlusion or poor texture, incomplete reconstruction of some areas is common. Therefore, offline methods are typically used to reconstruct all areas completely, or the images are remeasured online to avoid incomplete 3D reconstruction.

[0003] However, both offline operations and re-measuring images online will extend the operation time and reduce the efficiency of 3D reconstruction. Summary of the Invention

[0004] This application provides a method and apparatus for restoring 3D information, which can restore 3D data in scenarios where reconstruction has failed, while ensuring the real-time performance of 3D reconstruction, thus solving the technical problem of incomplete 3D reconstruction. This achieves the technical effect of improving the efficiency and real-time performance of 3D reconstruction and optimizing the user experience.

[0005] In a first aspect, embodiments of this application provide a method for restoring three-dimensional information, comprising:

[0006] If it is determined that there is a reconstruction failure in the reconstruction area, the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed are identified.

[0007] Upsample the reference frame and the lost frame, and determine a first feature matching relationship between the reference frame and the lost frame based on the sampling results;

[0008] Based on the first feature matching relationship, the relative motion between the reference frame and the lost frame is estimated to recover the pose and point cloud of the lost frame.

[0009] In a second aspect, embodiments of this application provide a three-dimensional information recovery apparatus, comprising:

[0010] The determination module is used to determine the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed when it is determined that there is a reconstruction failure in the reconstruction area;

[0011] The matching module is used to upsample the reference frame and the lost frame, and determine the first feature matching relationship between the reference frame and the lost frame based on the sampling result;

[0012] The estimation module is used to estimate the relative motion between the reference frame and the lost frame based on the first feature matching relationship, and to recover the pose and point cloud of the lost frame.

[0013] In a third aspect, embodiments of this application provide an electronic device, including:

[0014] Memory and one or more processors;

[0015] The memory is used to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for restoring three-dimensional information as described in the first aspect.

[0017] In a fourth aspect, embodiments of this application provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the method for restoring three-dimensional information as described in the first aspect.

[0018] This application embodiment identifies a lost frame and a successfully reconstructed reference frame adjacent to the lost frame when reconstruction failure is determined in the reconstructed area. The reference frame and the lost frame are upsampled, and a first feature matching relationship is determined between them based on the sampling results. Relative motion estimation between the reference frame and the lost frame is performed based on the first feature matching relationship to recover the pose and point cloud of the lost frame. By employing the above techniques, through upsampling the lost frame and reference frame and performing feature matching and relative motion estimation, the pose and point cloud of the lost frame can be recovered using the reference frame, thus achieving the recovery of the lost scene. Furthermore, by using upsampling, there is no need to remeasure the image online, thereby improving the real-time performance of 3D reconstruction, efficiently recovering the lost scene in the event of scene loss, enhancing the user's experience with 3D reconstruction information, and increasing the success rate of 3D scene reconstruction. Attached Figure Description

[0019] Figure 1 This is a flowchart of a three-dimensional information recovery method provided in Embodiment 1 of this application;

[0020] Figure 2 This is a flowchart illustrating the determination of lost frames and reference frames in Embodiment 1 of this application;

[0021] Figure 3 This is a schematic diagram of the loss scenario in Embodiment 1 of this application;

[0022] Figure 4 This is a schematic diagram of a reference frame in Embodiment 1 of this application;

[0023] Figure 5This is a flowchart of the three-dimensional feature point determination of the reference frame in Embodiment 1 of this application;

[0024] Figure 6 This is a flowchart of the pose and point cloud recovery process for lost frames in Embodiment 1 of this application;

[0025] Figure 7 This is a flowchart illustrating the recovery process after reconstruction failure in the reconstruction area in Embodiment 1 of this application.

[0026] Figure 8 This is a schematic diagram of the structure of a three-dimensional information recovery device provided in Embodiment 2 of this application;

[0027] Figure 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] The 3D information recovery method provided in this application aims to recover lost scenes after 3D reconstruction by identifying images adjacent to the lost scene, upsampling them, and then using feature matching and relative motion estimation between the images. This ensures real-time performance of 3D reconstruction while achieving efficient recovery of lost scenes. Traditional lost scene recovery schemes typically employ offline methods to reconstruct all regions or remeasure the images online. If reconstruction still fails, it indicates that the algorithm cannot handle this scenario. Obviously, using offline methods to reconstruct all regions or remeasure the images online increases user waiting time and user costs. Furthermore, with the original image resolution and number of feature points, reconstructing the image does not guarantee successful reconstruction of the entire scene. If reconstruction still fails after the user has waited for a complete cycle, it further impacts the user experience. Therefore, this application provides a 3D information recovery method to recover missing scenes while ensuring real-time performance of 3D reconstruction, solving the technical problem of incomplete 3D reconstruction.

[0030] Example 1:

[0031] Figure 1 A flowchart of a three-dimensional information recovery method according to Embodiment 1 of this application is provided. The three-dimensional information recovery method provided in this embodiment can be executed by a three-dimensional information recovery device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Generally, the three-dimensional information recovery device can be a server host, computer, or other computing device.

[0032] The following description uses the 3D information restoration device as an example to illustrate the 3D information restoration method. (Refer to...) Figure 1 The method for restoring this three-dimensional information specifically includes:

[0033] S110. If it is determined that there is a reconstruction failure in the reconstruction area, identify the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed.

[0034] After completing the reconstruction of the 3D scene, this application determines whether the reconstructed 3D scene has been lost, and if it is determined that there is a reconstruction failure in the reconstruction area, it performs a recovery operation for the lost scene in the reconstruction area.

[0035] It can be understood that during the three-dimensional reconstruction process, due to the weak texture of some scenes, the inter-frame matching relationship may be poor, and it is impossible to calculate its pose and point cloud, resulting in the loss of some scenes. Therefore, in order to obtain the complete three-dimensional scene information, it is necessary to restore the pose and point cloud of this part of the lost scene to ensure that the complete three-dimensional scene information can be presented to the user.

[0036] Specifically, after completing the three-dimensional scene reconstruction, the embodiment of the present application determines the number of successfully reconstructed poses and the number of images used for three-dimensional scene reconstruction. When the number of poses is less than the number of images, it is determined that the reconstruction area of the three-dimensional scene has failed.

[0037] It can be understood that during the three-dimensional reconstruction process, it is necessary to reconstruct the pose and point cloud of each frame of image according to the collected images and RTK data to construct the three-dimensional scene of the acquisition target. Therefore, for a completely reconstructed scene, the number of reconstructed image poses should be the same as the number of images participating in the three-dimensional scene reconstruction. If the number of poses is less than the number of images, it indicates that there is a scene loss.

[0038] Based on this, after completing the three-dimensional scene reconstruction process, the present application first judges the ratio between the number of restored poses n_poses and the number of images n_images. When n_poses < n_images, it means that the scene restoration is incomplete at this time, which may affect subsequent operations. It is necessary to enter the post-processing process of three-dimensional scene reconstruction to restore the lost poses and three-dimensional scene point clouds. If n_poses = n_images, it means that the three-dimensional scene reconstruction is complete and subsequent operations can be directly carried out.

[0039] Furthermore, when it is determined that there is a reconstruction failure in the reconstruction area, the present application determines each image in the lost scene, defines it as a lost frame, and determines the image adjacent to the lost frame and successfully reconstructed, defines it as a reference frame. To perform feature matching and relative motion estimation based on the reference frame and the lost frame to restore the pose and point cloud of the lost frame.

[0040] Refer to Figure 2 , the determination process of the lost frame and the reference frame includes:

[0041] S1101. Traverse all the images used for three-dimensional scene reconstruction, and determine the image with an invalid pose flag bit as the lost frame;

[0042] S1102. Determine the image whose distance from the lost frame is less than the set threshold and has been successfully reconstructed as the reference frame corresponding to the lost frame.

[0043] Based on all the images acquired during the 3D reconstruction process, the ID number of each image is first obtained, denoted as `view_id`. Simultaneously, the ID number of the image whose pose has been recovered is denoted as `pose_id`. That is, the ID numbers of all images and the ID numbers of the images whose poses have been recovered are respectively:

[0044] view_id = {0, 1, 2, ..., N}

[0045] pose_id = {0, 1, 2, ..., M}

[0046] If N≠M, it indicates that the scene reconstruction is incomplete and post-processing is needed to restore the pose and point cloud of the lost scene. This step requires counting the number of image frames in the lost scene, denoted as `lost_id`.

[0047] lost_id = view_id - pose_id

[0048] Then, based on the number of image frames in the missing scene, the missing frames are determined frame by frame. Since the recovered pose images from the first 3D reconstruction contain pose flag information, and the pose flag information of the unrecovered images has invalid values, the images with invalid pose flag values ​​are identified as the missing frames by traversing all images.

[0049] Furthermore, the lost frame is expanded to find an image adjacent to the lost frame whose pose has been recovered as a reference frame. This reference frame is then used for feature matching and relative motion estimation to recover the pose and point cloud of the lost frame.

[0050] Specifically, based on the absolute geographic coordinates provided by the RTK information corresponding to each frame, images with pose within the neighborhood of all lost frames are identified as reference frames. This is done by traversing all lost frames, where the RTK coordinates of any lost frame are R. i =(x i ,y i ,z i ) T Calculate the distance between the lost frame and all images whose pose has been recovered using RTK coordinates. Let R be the RTK coordinate of any view whose pose has been recovered. j =(x j ,y j ,z j ) T Then R i With R j The distance s between them is:

[0051]

[0052] By configuring a set threshold T, when the distance s between RTK coordinates satisfies s < T, it is regarded as the neighborhood of the lost frame, that is, the corresponding image is the reference frame of the lost frame. In this way, by determining the reference frame, it helps to recover the pose and point cloud of the subsequent lost frames.

[0053] Exemplarily, referring to Figure 3 , through the three-dimensional scene 1 reconstructed from the scene images captured by the UAV flight trajectory 11, when it is determined that there is a scene loss. First, all lost frames in the lost scene 12 are determined, and then the reference frames with recovered poses within the lost frame area are determined. Referring to Figure 4 , for each lost frame in the lost scene 12, referring to the above reference frame determination method, the reference frame 13 corresponding to the lost frame can be found on the periphery (i.e., the neighborhood) of the lost scene.

[0054] Optionally, in practical applications, there may be multiple lost frames corresponding to one reference frame. When recovering the poses and point clouds of these lost frames, a unified reference frame is used for calculation to save the number of operation requirements and improve the operation efficiency, thereby improving the real-time performance of the three-dimensional scene reconstruction.

[0055] S120. Upsample the reference frame and the lost frame, and determine the first feature matching relationship between the reference frame and the lost frame based on the sampling results.

[0056] Furthermore, based on the determined lost frames and reference frames, first upsample the reference frame and the lost frame to obtain more feature points in the lost frame and the reference frame.

[0057] It should be noted that during the three-dimensional reconstruction process, in order to ensure the real-time performance of the three-dimensional reconstruction and improve the three-dimensional reconstruction efficiency, for the captured scene images, usually the downsampling method is adopted, using images with a lower resolution and controlling a certain number of feature points for the three-dimensional scene reconstruction. And after the scene is lost, if the original lower-resolution images are still used and fewer feature points are controlled for the recovery of the lost three-dimensional scene, it is very likely that the scene recovery will fail. Therefore, during the process of recovering the point cloud and pose of the lost scene, the reference frame and the lost frame are upsampled, using a higher image resolution than the initial operation, and the default feature extractor is used to extract more feature points for calculation.

[0058] After upsampling, feature matching is performed based on the sampling results to determine the corresponding feature points between the lost frame and the reference frame. A matching relationship between these feature points is then established, defined as the first feature matching relationship, to determine the data association between the two frames. Feature points are points with significant image features (such as points where image grayscale values ​​change drastically or points with significant curvature at image edges (i.e., the intersection of two edges)). Image feature points reflect the essential features of an image and can identify target objects within it. Since adjacent images contain overlapping shooting regions, feature point matching enables the matching of adjacent images, establishing a data association between them.

[0059] Specifically, inter-frame matching between the reference frame and the lost frame is performed based on feature points from the upsampled lost frame and the reference frame to determine the first feature matching relationship between the 3D feature points of the reference frame and the 2D feature points of the lost frame. Before performing feature matching between the reference frame and the lost frame, this embodiment constructs an initial 3D scene of the reference frame to determine the 3D feature points in the reference frame, so as to match the 3D feature points of the reference frame with the 2D feature points of the lost frame. This facilitates subsequent relative motion estimation based on the 3D-2D matching relationship of the feature points, thereby recovering the pose and point cloud of the lost frame.

[0060] Reference Figure 5 The process for determining the 3D feature points of the reference frame includes:

[0061] S1201. Determine the second feature matching relationship between two adjacent reference frames, and determine the relative motion relationship based on the pose of the two adjacent reference frames;

[0062] S1202. Determine the three-dimensional feature points of the reference frame based on the second feature matching relationship and the relative motion relationship.

[0063] Specifically, since the pose of the reference frame has been recovered, the two adjacent reference frames are determined using RTK coordinates. Inter-frame matching is performed between the two adjacent reference frames to identify matching feature points, constructing a feature matching relationship between them, defined as the second feature matching relationship. Simultaneously, based on the poses of the two reference frames, their relative motion relationship can be determined. Then, based on this feature matching relationship and relative motion relationship, combined with the pixel coordinates of the feature points and camera intrinsic parameters, joint optimization is performed to obtain the initial 3D scene and determine the 3D coordinates of the feature points in the reference frames.

[0064] For example, refer to Figure 4 For multiple reference frames corresponding to lost frames identified outside the lost scene, determine the second feature matching relationship and relative motion relationship between any two adjacent reference frames. Assume the two reference frames are I... i and Ij The attitudes are T i and T j Based on the second feature matching relationship and the relative motion relationship, the initial 3D scene corresponding to the outer reference frame of the lost scene can be obtained, and this initial 3D scene is denoted as:

[0065]

[0066] P represents the pose of the i-th frame in the reference frame relative to the world coordinate system, where i ∈ i1, ii; j This represents the j-th map point in the reconstructed initial 3D scene.

[0067] It should be noted that in practical applications, there are many ways to construct a 3D scene based on the feature matching relationship and relative motion relationship between images and to determine the 3D coordinates of feature points in the image. The embodiments of this application do not impose fixed restrictions on the specific 3D construction method, and will not be elaborated here.

[0068] After completing the initial 3D scene construction, feature matching is performed based on the 3D coordinates of the reference frame feature points in the 3D scene and the 2D image coordinates of the corresponding lost frame to construct the 3D-2D matching relationship of the feature points, which is used for subsequent relative motion estimation to determine the pose and point cloud of the lost frame.

[0069] S130. Based on the first feature matching relationship, perform relative motion estimation between the reference frame and the lost frame to recover the pose and point cloud of the lost frame.

[0070] Based on the established first feature matching relationship, the initial pose of the lost frame is determined through relative motion estimation, and then the initial point cloud of the lost frame is determined based on the initial pose. Finally, through joint optimization, the pose and point cloud of the lost frame are determined.

[0071] Reference Figure 6 The pose and point cloud recovery process for lost frames includes:

[0072] S1301. Based on the first feature matching relationship, perform relative motion estimation between the reference frame and the lost frame to determine the initial pose of the lost frame;

[0073] S1302. Determine the initial point cloud of the lost frame based on the initial attitude;

[0074] S1303. Perform joint optimization based on the initial pose, initial point cloud, and pose and point cloud of the reference frame to obtain the pose and point cloud of the lost frame.

[0075] Based on the 3D-2D matching relationship between the reference frame's 3D feature points and the lost frame's 2D feature points provided by the first feature matching relationship, the initial pose of the corresponding lost frame can be obtained by constructing an optimization method using reprojection error and estimating the inter-frame relative motion using the PNP algorithm.

[0076] The formula for calculating the initial attitude is:

[0077]

[0078] Where J represents minimizing the cost function, n represents the number of feature points between the reference frame and the lost frame that satisfy the first feature matching relationship, and Z m P represents the m-th two-dimensional feature point in the lost frame. m Let R0 represent the m-th 3D feature point in the reference frame, K represent the camera intrinsic parameters, (R0, t0) represent the initial pose, R0 is the initial rotation parameter, and t0 is the initial translation parameter.

[0079] For each lost frame for which a corresponding reference frame has been found, the initial pose is determined using the method described above. Then, based on this initial pose, referring to the method described above for constructing the initial 3D scene, the relative motion relationship between the reference frame and the lost frame is first determined. Combining the first feature matching relationship between the reference frame and the lost frame, the pixel coordinates of the feature points, and the camera's intrinsic parameters, the 3D coordinates of each feature point in the lost frame are calculated, thereby obtaining the initial point cloud of the lost frame.

[0080] Then, for each lost frame and its corresponding reference frame obtained from the current initial pose and initial point cloud, joint optimization is performed. The calculation formula for joint optimization is as follows:

[0081]

[0082] The set of image poses for the reference frame and the lost frame is denoted as T = {T1, T2, T3, ... T}. N},in Let the point cloud set of the reference frame and the lost frame be denoted as L={P1,P2,P3,...P M}, Represents the j-th feature point in the i-th image of the image set, (R i ,t i P represents the pose of the i-th image in the image set. j Let J represent the j-th point cloud in the image set, where J represents the cost function to be minimized, and K represents the camera intrinsic parameters.

[0083] Based on the above joint optimization formula, by minimizing the cost function J, the state variables {T,L} can be obtained through joint optimization, thereby obtaining the pose and point cloud of each lost frame.

[0084] Furthermore, the lost frames whose pose and point cloud recovery has been completed are used as reference frames to iteratively reconstruct the pose and point cloud of adjacent lost frames whose pose and point cloud recovery has not been completed.

[0085] Reference Figure 4 For a lost scene 12 in a given region, the lost frames at the edges of the lost scene can be referenced by images surrounding the lost scene 12. However, apart from the edges of the lost scene, the images inward from the edges do not currently have corresponding reference frames (the neighborhood of these images is the lost frame). Therefore, after restoring the pose and point cloud of the lost frames at the edges of the lost scene 12, the pose and point cloud of other lost frames can be further restored from the edges inward. Since the image at the edges of the lost scene has already had its point cloud and pose restored, the lost frames in the neighborhood of these images can be selected as reference frames based on RTK coordinates. Then, based on steps S110-S130 above, the pose and point cloud of the lost frames are restored. This process continues, using the restored lost frames as reference frames, iteratively restoring the pose and point cloud of the corresponding lost frames from the edges of the lost scene 12 inward until all lost frames are fully restored. By iteratively restoring the point cloud and pose of the lost frames, complete 3D scene information is obtained.

[0086] Thus, by upsampling images to increase the number of feature points in scenes where 3D reconstruction failed, the success rate of data association was improved. Furthermore, a layer-by-layer diffusion method was used to fill in the missing parts of the reconstructed scene, thereby improving both the real-time performance and the success rate and completeness of 3D reconstruction.

[0087] For example, such as Figure 7 As shown, taking a plant protection operation scenario as an example, during the plant protection operation, it is necessary to acquire three-dimensional scene information of the operation area in order to perform subsequent plant protection operations based on the three-dimensional scene information. This is achieved by using an unmanned aerial vehicle (UAV) to capture images of the operation area according to a set flight trajectory, determining the RTK information corresponding to each image, and then using the acquired images and RTK information as input into the three-dimensional information restoration device of this embodiment for three-dimensional reconstruction. Here, each image is a single image captured by a single camera, and the RTK information, after conversion, determines the current world coordinates (X, Y, Z) of the UAV.

[0088] Furthermore, to ensure real-time performance, 3D reconstruction typically employs sampling lower-resolution images and controlling a certain number of feature points for scene reconstruction. A fixed number of feature points are extracted from the downsampled image, and the absolute coordinates of the UAV-acquired image are determined based on RTK information. Then, feature matching is performed on several frames that are close to the currently acquired image location to establish data associations. Subsequently, the feature matching relationships between the RTK data and all image pairs are fused for joint optimization, thereby estimating the UAV pose and the 3D scene point cloud.

[0089] During 3D reconstruction, incomplete scene reconstructions may occur due to weak textures in some areas. To ensure the smooth operation of plant protection work and improve user experience, when scene loss is identified in the reconstructed 3D scene, the lost frames and their corresponding reference frames are determined. Using existing pose reference frames as constraints, the pose and point cloud of the lost frames in the lost scene are gradually restored. Using existing pose reference frames and lost frames, upsampling is employed to increase the number of feature points and construct an initial 3D scene structure. The PNP algorithm is then used to gradually restore the pose and point cloud of the lost frames from the edges of the lost scene inwards, thus achieving complete 3D scene reconstruction in a faster and more successful manner.

[0090] The above describes a process where, in cases where reconstruction failures are identified in the reconstructed area, a lost frame and a successfully reconstructed reference frame adjacent to the lost frame are determined. Both the reference frame and the lost frame are upsampled, and a first feature matching relationship is determined between them based on the sampling results. Relative motion estimation between the reference frame and the lost frame is then performed based on this first feature matching relationship to recover the pose and point cloud of the lost frame. By employing this technique, through upsampling of the lost frame and reference frame, and performing feature matching and relative motion estimation, the pose and point cloud of the lost frame can be recovered using the reference frame, thus achieving the restoration of the lost scene. Furthermore, the upsampling method eliminates the need for re-measuring the image online, thereby improving the real-time performance of 3D reconstruction, efficiently restoring lost scenes in cases of scene loss, enhancing the user experience of 3D reconstruction information, and increasing the success rate of 3D scene reconstruction.

[0091] Example 2:

[0092] Based on the above embodiments, Figure 8 This is a schematic diagram of a three-dimensional information recovery device provided in Embodiment 2 of this application. (Reference) Figure 8 The three-dimensional information recovery device provided in this embodiment specifically includes: a determination module 21, a matching module 22, and an estimation module 23.

[0093] The determining module 21 is used to determine the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed when the reconstruction area is determined to have reconstruction failure.

[0094] Matching module 22 is used to upsample the reference frame and the lost frame, and determine the first feature matching relationship between the reference frame and the lost frame based on the sampling results;

[0095] The estimation module 23 is used to estimate the relative motion between the reference frame and the lost frame based on the first feature matching relationship, and to recover the pose and point cloud of the lost frame.

[0096] Specifically, the determination module 21 determines the reconstruction failure of the 3D scene reconstruction area by determining the number of reconstructed poses and the number of images for 3D scene reconstruction.

[0097] By traversing all images for 3D scene reconstruction, images with invalid pose flags are identified as lost frames. Images that are less than a set threshold in distance from the lost frames and have been successfully reconstructed are identified as reference frames corresponding to the lost frames. Lost frames that have completed pose and point cloud recovery are used as reference frames to iteratively reconstruct the pose and point cloud of adjacent lost frames that have not completed pose and point cloud recovery.

[0098] Specifically, the matching module 22 performs inter-frame matching between the reference frame and the lost frame based on the feature points of the upsampled lost frame and the reference frame, and determines the first feature matching relationship between the three-dimensional feature points of the reference frame and the two-dimensional feature points of the lost frame. Then, by determining the second feature matching relationship between two adjacent reference frames, and determining the relative motion relationship based on the pose of the two adjacent reference frames, the three-dimensional feature points of the reference frame are determined based on the second feature matching relationship and the relative motion relationship.

[0099] Specifically, the estimation module 23 estimates the relative motion between the reference frame and the lost frame based on the first feature matching relationship to determine the initial pose of the lost frame; determines the initial point cloud of the lost frame based on the initial pose; and performs joint optimization based on the initial pose, the initial point cloud, and the pose and point cloud of the reference frame to obtain the pose and point cloud of the lost frame.

[0100] The formula for calculating the initial attitude is as follows:

[0101]

[0102] Where J represents minimizing the cost function, n represents the number of feature points between the reference frame and the lost frame that satisfy the first feature matching relationship, and Z m P represents the m-th two-dimensional feature point in the lost frame. mLet R0 represent the m-th 3D feature point in the reference frame, K represent the camera intrinsic parameters, (R0, t0) represent the initial pose, R0 is the initial rotation parameter, and t0 is the initial translation parameter.

[0103] The formula for calculating joint optimization is:

[0104]

[0105] The set of images of the reference frame and the lost frame is denoted as T = {T1, T2, T3, ... T}. N},in Let the point cloud set of the reference frame and the lost frame be denoted as L={P1,P2,P3,...P M}, Represents the j-th feature point in the i-th image of the image set, (R i ,t i P represents the pose of the i-th image in the image set. j Let J represent the j-th point cloud in the image set, where J represents the cost function to be minimized, and K represents the camera intrinsic parameters.

[0106] The above describes a process where, in cases where reconstruction failures are identified in the reconstructed area, a lost frame and a successfully reconstructed reference frame adjacent to the lost frame are determined. Both the reference frame and the lost frame are upsampled, and a first feature matching relationship is determined between them based on the sampling results. Relative motion estimation between the reference frame and the lost frame is then performed based on this first feature matching relationship to recover the pose and point cloud of the lost frame. By employing this technique, through upsampling of the lost frame and reference frame, and performing feature matching and relative motion estimation, the pose and point cloud of the lost frame can be recovered using the reference frame, thus achieving the restoration of the lost scene. Furthermore, the upsampling method eliminates the need for re-measuring the image online, thereby improving the real-time performance of 3D reconstruction, efficiently restoring lost scenes in cases of scene loss, enhancing the user experience of 3D reconstruction information, and increasing the success rate of 3D scene reconstruction.

[0107] The three-dimensional information restoration device provided in Embodiment 2 of this application can be used to execute the three-dimensional information restoration method provided in Embodiment 1 above, and has corresponding functions and beneficial effects.

[0108] Example 3:

[0109] This application provides an electronic device in embodiment three, referring to... Figure 9The electronic device includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The electronic device may have one or more processors and one or more memories. The processor 31, memory 32, communication module 33, input device 34, and output device 35 can be connected via a bus or other means.

[0110] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the three-dimensional information recovery method described in any embodiment of this application (e.g., the determination module, matching module, and estimation module in a three-dimensional information recovery device). The memory may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The communication module 33 is used for data transmission.

[0112] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory, thereby realizing the above-mentioned method for restoring three-dimensional information.

[0113] Input device 34 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 35 may include display devices such as a display screen.

[0114] The electronic device provided above can be used to perform the three-dimensional information recovery method provided in Embodiment 1 above, and has corresponding functions and beneficial effects.

[0115] Example 4:

[0116] This application embodiment also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for restoring three-dimensional information. The method for restoring three-dimensional information includes: determining a lost frame and a reference frame that is adjacent to the lost frame and has been reconstructed when it is determined that there is a reconstruction failure in the reconstruction area; upsampling the reference frame and the lost frame, and determining a first feature matching relationship between the reference frame and the lost frame based on the sampling result; and estimating the relative motion between the reference frame and the lost frame based on the first feature matching relationship to restore the pose and point cloud of the lost frame.

[0117] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0118] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the three-dimensional information recovery method described above, but can also perform related operations in the three-dimensional information recovery method provided in any embodiment of this application.

[0119] The three-dimensional information recovery device, storage medium, and electronic device provided in the above embodiments can execute the three-dimensional information recovery method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the three-dimensional information recovery method provided in any embodiment of this application.

[0120] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for restoring three-dimensional information, characterized in that, include: If it is determined that there is a reconstruction failure in the reconstruction area, the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed are identified. Upsampling is performed on the reference frame and the lost frame, and a first feature matching relationship between the reference frame and the lost frame is determined based on the sampling results. The determination of the first feature matching relationship between the reference frame and the lost frame based on the sampling results includes: performing inter-frame matching between the reference frame and the lost frame based on the feature points of the upsampled lost frame and the reference frame, and determining a first feature matching relationship between the three-dimensional feature points of the reference frame and the two-dimensional feature points of the lost frame. Based on the first feature matching relationship, the relative motion between the reference frame and the lost frame is estimated to recover the pose and point cloud of the lost frame.

2. The method for restoring three-dimensional information according to claim 1, characterized in that, Before determining the first feature matching relationship between the three-dimensional feature points of the reference frame and the two-dimensional feature points of the lost frame, the method further includes: Determine the second feature matching relationship between two adjacent reference frames, and determine the relative motion relationship based on the pose of the two adjacent reference frames; The three-dimensional feature points of the reference frame are determined based on the second feature matching relationship and the relative motion relationship.

3. The method for restoring three-dimensional information according to claim 1, characterized in that, The step of estimating the relative motion between the reference frame and the lost frame based on the first feature matching relationship, and recovering the pose and point cloud of the lost frame, includes: Based on the first feature matching relationship, the relative motion between the reference frame and the lost frame is estimated to determine the initial pose of the lost frame; The initial point cloud of the lost frame is determined based on the initial pose; The pose and point cloud of the lost frame are obtained by jointly optimizing the initial pose, the initial point cloud, and the pose and point cloud of the reference frame.

4. The method for restoring three-dimensional information according to claim 3, characterized in that, The formula for calculating the initial attitude is: Where J represents minimizing the cost function, n represents the number of feature points between the reference frame and the lost frame that satisfy the first feature matching relationship, Zm represents the m-th two-dimensional feature point in the lost frame, Pm represents the m-th three-dimensional feature point in the reference frame, K represents the camera intrinsic parameters, (R0, t0) represents the initial pose, R0 is the initial rotation parameter, and t0 is the initial translation parameter.

5. The method for restoring three-dimensional information according to claim 3, characterized in that, The calculation formula for the joint optimization is as follows: The image set of the reference frame and the lost frame is denoted as . ,in The point cloud set of the reference frame and the lost frame is denoted as... , This represents the j-th feature point in the i-th image of the image set. Let Pj represent the pose of the i-th image in the image set, Pj represent the j-th point cloud in the image set, J represent the cost function to be minimized, and K represent the camera intrinsic parameters.

6. The method for restoring three-dimensional information according to claim 1, characterized in that, The determination that the reconstruction area has reconstruction failure situations includes: The number of reconstructed poses and the number of images used for 3D scene reconstruction are determined. If the number of poses is less than the number of images, it is determined that the reconstruction of the 3D scene has failed.

7. The method for restoring three-dimensional information according to claim 1, characterized in that, The determination of the lost frame and the adjacent, successfully reconstructed reference frame includes: Iterate through all images for 3D scene reconstruction and identify images with invalid pose flags as lost frames. An image that is less than a set threshold distance from the lost frame and has been successfully reconstructed is identified as the reference frame corresponding to the lost frame.

8. The method for restoring three-dimensional information according to claim 7, characterized in that, The determination of the lost frame and the reference frame adjacent to the lost frame that has been successfully reconstructed also includes: The lost frame whose attitude and point cloud recovery has been completed is used as the reference frame to iteratively reconstruct the attitude and point cloud of adjacent lost frames whose attitude and point cloud recovery has not been completed.

9. A device for restoring three-dimensional information, characterized in that, include: The determination module is used to determine the lost frame and the reference frame that is adjacent to the lost frame and has been successfully reconstructed when it is determined that there is a reconstruction failure in the reconstruction area; A matching module is used to upsample the reference frame and the lost frame, and determine a first feature matching relationship between the reference frame and the lost frame based on the sampling results. The step of determining the first feature matching relationship between the reference frame and the lost frame based on the sampling results includes: performing inter-frame matching between the reference frame and the lost frame based on the feature points of the upsampled lost frame and the reference frame, and determining a first feature matching relationship between the three-dimensional feature points of the reference frame and the two-dimensional feature points of the lost frame. The estimation module is used to estimate the relative motion between the reference frame and the lost frame based on the first feature matching relationship, and to recover the pose and point cloud of the lost frame.

10. An electronic device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for restoring three-dimensional information as described in any one of claims 1-8.

11. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the method for restoring three-dimensional information as described in any one of claims 1-8.

Images