Three-dimensional scene reconstruction method, device, electronic equipment and program product
Patent Information
- Application Number
- CN202611160033.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-10-23
AI Technical Summary
[0003]然而,可移动设备的下游任务通常仅需可移动设备的可视区域与三维场景模型对应的几何信息,而不需要非可视区域的几何信息,因此,对三维场景模型的全场景稠密重建,不仅造成了不必要的采集与计算资源浪费,并且引入的大量冗余几何信息也额外增加了可移动设备下游任务的训练开销,降低了训练效率
[0019]本申请公开了一种三维场景重建方法、装置、电子设备及程序产品,根据参考图像集合,构建几何骨架,并基于几何骨架,获取目标视角对应的几何图像,再根据几何图像,在参考图像集合中确定与几何图像匹配的目标参考图像,随后以目标参考图像为参考,在目标视角下对几何骨架进行渲染,生成目标视角对应的渲染图像,使得可以通过目标参考图像与几何图像在空间位姿和内容上的高一致性,将目标参考图像的纹理、色彩等外观信息作为渲染约束,确保渲染生成的内容不仅外观合理,而且还满足真实场景的几何、遮挡、轮廓和自由空间的约束,随后根据多个目标视角分别对应的渲染图像及所述参考图像集合,构建三维场景模型,不仅可以有效填补原始稀疏视角下的观测盲区,提高三维场景模型重建的准确性,并且全程仅围绕可视区域进行针对性的三维场景模型重建,避免了在非可视区域进行冗余建模与无效计算导致的资源浪费,还提升了重建的三维场景模型的有效信息占比,还提升了下游任务的训练效率。
Smart Images

Figure CN122966857A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a three-dimensional scene reconstruction method, apparatus, electronic device, and program product. Background Technology
[0002] Currently, Real-to-Sim (R2S) technology typically relies on dense, global real-scene acquisition and scene-by-scene reconstruction to generate renderable 3D scene models, thereby providing rich data support for training downstream tasks on mobile devices.
[0003] However, downstream tasks of mobile devices typically only require the geometric information corresponding to the visible area of the mobile device and the 3D scene model, and do not require the geometric information of the non-visible area. Therefore, dense reconstruction of the entire 3D scene model not only causes unnecessary waste of acquisition and computing resources, but also introduces a large amount of redundant geometric information, which further increases the training overhead of downstream tasks of mobile devices and reduces training efficiency. Summary of the Invention
[0004] This application discloses a three-dimensional scene reconstruction method, apparatus, electronic device, and program product, which can perform targeted three-dimensional scene model reconstruction only around the visible area. While expanding the sparse reference image set and improving the accuracy of three-dimensional scene model reconstruction, it also improves the training efficiency of downstream tasks.
[0005] The first aspect of this application discloses a three-dimensional scene reconstruction method, including: Obtain a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image; Construct a geometric skeleton based on the reference image set; Based on the geometric skeleton, obtain the geometric image corresponding to the target viewpoint; Based on the geometric image, a target reference image matching the geometric image is determined from the set of reference images; Based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, a rendered image corresponding to the target viewpoint is generated; A three-dimensional scene model is constructed based on the rendered images corresponding to multiple target viewpoints and the reference image set.
[0006] In some possible embodiments, generating a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, a reference viewpoint corresponding to the target reference image, and the target viewpoint includes: Based on the geometric skeleton, a movement trajectory is determined from the reference viewpoint corresponding to the target reference image to the target viewpoint; Based on the movement trajectory, determine the geometric constraint information; The target reference image, the movement trajectory, and the geometric constraint information are input into the video generation model, and the geometric skeleton is rendered to obtain a rendered image from the target's perspective.
[0007] In some possible embodiments, determining the geometric constraint information based on the movement trajectory includes: Along the movement trajectory, interpolation is performed on the reference viewpoint corresponding to the target reference image and the pose parameters corresponding to the target viewpoint to obtain multiple pose parameters; Based on the plurality of pose parameters, determine the movement viewpoint corresponding to each of the plurality of pose parameters; Based on the geometric skeleton, generate geometric images corresponding to each moving viewpoint; The geometric constraint information is determined based on the geometric images corresponding to each moving viewpoint.
[0008] In some possible embodiments, the step of inputting the target reference image, the movement trajectory, and the geometric constraint information into the video generation model, and performing appearance rendering on the geometric skeleton to obtain a rendered image from the target's perspective includes: The target reference image, the movement trajectory, and the geometric constraint information are input into the video generation model. Under the constraint of the geometric constraint information, the video generation model renders the appearance of the geometric skeleton along the movement trajectory with the target reference image as the starting frame, and generates video data corresponding to the movement trajectory. The last frame of the video data is determined to be the rendered image from the target viewpoint.
[0009] In some possible embodiments, acquiring the reference image set includes: Obtain the set of reference images corresponding to the current iteration cycle; The method further includes: After generating the rendering images of multiple target perspectives corresponding to the current iteration cycle, the rendering images of multiple target perspectives corresponding to the current iteration cycle are added to the reference image set corresponding to the current iteration cycle to obtain the reference image set corresponding to the next iteration cycle. The reference image set corresponding to the next iteration cycle is used to generate the rendering images of multiple target perspectives corresponding to the next iteration cycle. The step of constructing a 3D scene model based on the rendered images corresponding to multiple target viewpoints and the reference image set includes: A 3D scene model is constructed based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the set of reference images corresponding to the last iteration cycle.
[0010] In some possible embodiments, adding the rendered images of multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle includes: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle; Based on the rendered image corresponding to the first target viewpoint, determine the second point cloud information corresponding to the first target viewpoint, wherein the first target viewpoint is any one of the multiple target views corresponding to the current iteration cycle; Calculate the first distance information between the first point cloud information and the second point cloud information; If the first distance information is less than or equal to the distance threshold, then the rendered image corresponding to the first target viewpoint is added to the reference image set.
[0011] In some possible embodiments, before adding the rendered images of the multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle, the method further includes: By repeatedly executing the step of generating a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, multiple rendered images corresponding to the first target viewpoint are obtained. The step of adding the rendered images of multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle includes: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle; Based on the multiple rendered images corresponding to the first target viewpoint, determine the second point cloud information of the multiple rendered images corresponding to the first target viewpoint; Calculate the first distance information between the first point cloud information and the second point cloud information corresponding to the multiple rendered images respectively; From the multiple rendered images, the rendered image with the smallest first distance information is selected and added to the reference image set.
[0012] In some possible embodiments, constructing the geometric skeleton based on the reference image set includes: Feature point matching is performed on the multiple reference images, and multiple three-dimensional spatial points are determined based on the multiple feature points obtained by matching, wherein the three-dimensional spatial points correspond to feature points in at least one reference image; Based on the multiple three-dimensional spatial points, construct the observation geometry; The observed geometry is input into the prior model, and the missing structures in the observed geometry are predicted and filled in according to the prior model to obtain the predicted geometry. The observed geometry and the predicted geometry are fused to obtain the geometric skeleton.
[0013] In some possible embodiments, constructing the observation geometry based on the plurality of three-dimensional spatial points includes: The confidence level and reprojection residual corresponding to each of the three-dimensional spatial points are determined; the confidence level corresponding to each three-dimensional spatial point is related to the number of reference images corresponding to the three-dimensional spatial point; the reprojection residual is used to characterize the deviation between the mapping position of the three-dimensional spatial point in the corresponding reference image and the feature point; From the plurality of three-dimensional spatial points, multiple target three-dimensional spatial points are selected with a confidence level greater than or equal to the confidence level threshold and a reprojection residual greater than or equal to the residual threshold. The observation geometry is constructed based on the multiple target three-dimensional spatial points.
[0014] In some possible embodiments, before obtaining the geometric image corresponding to the target viewpoint based on the geometric skeleton, the method further includes: Based on the geometric skeleton, multiple first initial viewpoints are determined, and geometric images corresponding to the multiple first initial viewpoints are generated respectively; Based on the geometric images corresponding to the plurality of first initial viewpoints, determine the viewpoint evaluation values corresponding to the plurality of first initial viewpoints respectively; The target viewpoints are sorted in descending order of their evaluation values, and the top K viewpoints are selected from the plurality of first initial viewpoints, where K is a positive integer.
[0015] In some possible embodiments, the step of sorting the target views from the plurality of first initial views in descending order according to the view evaluation values includes: The plurality of first initial viewpoints are sorted in descending order of the viewpoint evaluation values to obtain a sorted viewpoint sequence. The sorted view sequence is divided into multiple view groups, wherein the similarity between the view evaluation values of two adjacent first initial views in each view group is greater than or equal to a similarity threshold. Determine the first K view groups, and take the first initial view in each of the first K view groups as the target view.
[0016] The second aspect of this application discloses a three-dimensional scene reconstruction apparatus, comprising: The data acquisition module is used to acquire a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image; The scene construction module is used to construct a geometric skeleton based on the reference image set; The first generation module is used to obtain a geometric image corresponding to the target viewpoint based on the geometric skeleton; An image matching module is used to determine a target reference image that matches the geometric image in the set of reference images, based on the geometric image. The second generation module is used to generate a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint. The scene construction module is also used to construct a three-dimensional scene model based on the rendered images corresponding to multiple target viewpoints and the set of reference images.
[0017] A third aspect of this application discloses an electronic device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method described in any of the above embodiments.
[0018] The fourth aspect of this application discloses a computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any of the above embodiments.
[0019] This application discloses a method, apparatus, electronic device, and program product for 3D scene reconstruction. Based on a set of reference images, a geometric skeleton is constructed. Based on the geometric skeleton, a geometric image corresponding to the target viewpoint is obtained. Then, based on the geometric image, a target reference image matching the geometric image is determined from the set of reference images. Subsequently, using the target reference image as a reference, the geometric skeleton is rendered from the target viewpoint to generate a rendered image corresponding to the target viewpoint. This allows for high consistency between the target reference image and the geometric image in spatial pose and content. The texture, color, and other appearance information of the target reference image are used as rendering constraints to ensure that the rendered content not only has a reasonable appearance but also meets the constraints of geometry, occlusion, contour, and free space in the real scene. Then, based on the rendered images corresponding to multiple target viewpoints and the set of reference images, a 3D scene model is constructed. This not only effectively fills the observation blind spots under the original sparse viewpoints and improves the accuracy of 3D scene model reconstruction, but also focuses on the visible area for targeted 3D scene model reconstruction throughout the process, avoiding resource waste caused by redundant modeling and ineffective calculations in non-visible areas. It also increases the effective information ratio of the reconstructed 3D scene model and improves the training efficiency of downstream tasks. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1A This is an application scenario diagram of a 3D scene reconstruction method. Figure 1B An application scenario diagram for another 3D scene reconstruction method; Figure 2 A flowchart of a three-dimensional scene reconstruction method provided in the embodiments of this application; Figure 3 A flowchart for generating a rendered image corresponding to a target viewpoint based on geometric constraint information, provided in an embodiment of this application; Figure 4 A flowchart for determining geometric constraint information based on a movement trajectory, provided in an embodiment of this application; Figure 5 A flowchart illustrating the iterative acquisition process provided in this application embodiment; Figure 6 A flowchart for filtering rendered images provided in this application embodiment; Figure 7 A flowchart for filtering multiple rendered images corresponding to a first target viewpoint, provided as an embodiment of this application; Figure 8 A flowchart for constructing a geometric skeleton is provided for an embodiment of this application; Figure 9 A flowchart for constructing the observation geometry provided in this application embodiment; Figure 10 A flowchart for determining multiple target perspectives provided in embodiments of this application; Figure 11 A flowchart illustrating the grouping and filtering target perspective provided in this application embodiment; Figure 12 A schematic diagram illustrating the reconstruction of a 3D scene model from a robot's perspective, as provided in an embodiment of this application. Figure 13 This is a schematic diagram of a three-dimensional scene reconstruction device provided in an embodiment of this application; Figure 14 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0024] Figure 1A This is an application scenario diagram of a 3D scene reconstruction method. For example... Figure 1A As shown, the 3D scene reconstruction method can be applied to terminal device 100. Terminal device 100 is communicatively connected to camera device 101, which is deployed on mobile device 102. Camera device 101 sends reference images captured from multiple perspectives to terminal device 100. Terminal device 100 performs 3D scene reconstruction based on the received multiple reference images. After obtaining the 3D scene model, it trains various tasks of mobile device 102 based on the geometric and rendering information of the 3D scene model.
[0025] Terminal device 100 may include, but is not limited to, personal computers, tablets, servers, edge computing units or cloud devices, etc., without specific limitations.
[0026] The mobile device 102 may include devices such as robots, vehicles, or drones. Robots may include body-worn robots, inspection robots, delivery robots, or sweeping robots. Optionally, robots may be legged robots, wheeled robots, tracked robots, or hybrid robots, etc., without specific limitations.
[0027] Figure 1B This is an application scenario diagram for another 3D scene reconstruction method. For example... Figure 1B As shown, the 3D scene reconstruction method can be applied to a mobile device 102. A camera device 101 and a controller 103 can be deployed on the mobile device 102, with the camera device 101 connected to the controller 103. The camera device 101 sends reference images captured from multiple perspectives to the controller 103. The controller 103 reconstructs the 3D scene based on the received reference images, obtaining a 3D scene model. Then, based on the geometric and rendering information of the 3D scene model, the controller trains the mobile device 102 for various tasks.
[0028] The controller 103 may be an embedded computing unit or a system-on-a-chip, etc., without specific limitations.
[0029] For example, the mobile device 102 can be an intelligent inspection robot equipped with a camera device 101 and an embedded computing unit. The mobile device 102 can autonomously complete the 3D scene reconstruction of the area where the robot is located after acquiring multiple reference images from multiple perspectives.
[0030] Unless otherwise stated, this application uses the mobile device 102 for 3D scene reconstruction.
[0031] In some embodiments, the mobile device 102 reconstructs a 3D scene based on multiple reference images using Real-to-Sim (R2S) technology.
[0032] The scene representation schemes used in the scene reconstruction stage of R2S technology may include, but are not limited to, Three-Dimensional Gaussian Splatting (3DGS), Neural Radiance Field (NeRF), 3D mesh models, or other explicit, implicit, and hybrid scene representation schemes.
[0033] Currently, R2S technology typically relies on dense multi-view images as input, which can fully cover the geometric and texture information of the scene from various angles. Deep fusion is then performed through multi-view consistency constraints, resulting in a reconstructed 3D scene model with higher texture and geometric accuracy, more closely resembling the real physical environment. This provides richer and more accurate data support for training downstream tasks of the mobile device 102, eliminating the need for additional image acquisition to supplement training data. For example, if the mobile device 102 is a robot, the robot's downstream tasks may include, but are not limited to, navigation tasks, operational tasks, or motion control tasks.
[0034] However, because R2S technology generally adopts a dense reconstruction mode across the entire scene, it generates a large amount of redundant geometric data outside the field of view of the mobile device 102. Downstream tasks of the mobile device 102 typically only require the geometric information corresponding to the visible area and the 3D scene model, and do not need geometric information of non-visible areas. For example, a robot's autonomous navigation task only needs to perceive the visible scene structure around its path, without needing to pay attention to the geometric information of occluded areas or behind walls. Therefore, dense reconstruction of the entire 3D scene model not only causes unnecessary waste of acquisition and computing resources, but also introduces a large amount of redundant geometric information, further increasing the training overhead of the downstream tasks of the mobile device 102 and reducing training efficiency. In summary, how to adapt the reconstructed 3D scene model to the actual observation needs of the downstream tasks of the mobile device 102 is one of the main problems that urgently need to be solved.
[0035] Figure 2 A flowchart illustrating the three-dimensional scene reconstruction method provided in this application embodiment. Figure 2 As shown, the method may include the following steps: Step 202: Obtain a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image.
[0036] The reference image may include an image actually captured by the camera device, and / or a generated high-fidelity virtual image. The reference viewpoint may refer to the six-degree-of-freedom pose parameters of the camera device in the world coordinate system when acquiring the reference image.
[0037] Step 204: Construct a geometric skeleton based on the reference image set.
[0038] In some embodiments, the mobile device can input multiple reference images contained in the reference image set into a prior model, and then use the prior model to reconstruct the 3D scene based on the multiple reference images to obtain a geometric skeleton.
[0039] The mobile device can construct observation geometry based on multiple reference images contained in the reference image set, input the observation geometry into the prior model, and then use the prior model to reconstruct the 3D scene based on the observation geometry to obtain the geometric skeleton.
[0040] Observational geometry refers to the 3D geometric structure reconstructed from real data of a reference image. Geometric skeleton refers to the complete 3D scene reconstruction result obtained by a prior model using observational geometry as input, through geometric completion and prediction. The geometric skeleton includes both observational geometry based on real observations and predictive geometry generated through prior model predictions.
[0041] Step 206: Based on the geometric skeleton, obtain the geometric image corresponding to the target viewpoint.
[0042] The mobile device can determine multiple initial viewpoints based on a geometric skeleton and generate geometric images corresponding to each of the multiple initial viewpoints. The mobile device can then determine a target viewpoint from the multiple initial viewpoints and determine the corresponding geometric image.
[0043] In some embodiments, the mobile device may determine multiple first initial viewpoints of the mobile device based on a geometric skeleton and behavioral information of the mobile device.
[0044] The behavioral information of a mobile device may include, but is not limited to, one or more of the following: the installation location of the camera device on the mobile device, the task, the reachable configuration, the permissible movement area, and the body-related area. The task may refer to the specific action or procedure that the mobile device needs to perform; the reachable configuration may refer to all the configuration combinations that the various joints of the mobile device can achieve within their degrees of freedom; the permissible movement area may refer to the physical space range in which the mobile device can move safely; and the body-related area may refer to the spatial area occupied by the main structure of the mobile device, including the geometrically enclosed volume of components such as the base, arm, and end effector.
[0045] Understandably, the installation position of the camera device limits the reference offset of the viewing angle relative to the mobile device itself, the task defines the spatial path range that the viewing angle needs to cover, the reachable configuration and the allowed movement area jointly constrain the feasible boundary of the viewing angle pose, and the body-related area is used to exclude invalid viewing angles that are occluded by the mobile device itself. Therefore, by using the behavioral information of the mobile device, multiple first initial viewing angles that match the actual working scenario of the mobile device can be selected from any viewing angle that can be generated in the geometric skeleton theory. This ensures that multiple first initial viewing angles are all valid viewing angles that the mobile device can actually reach under real working conditions and that have clear observational value for the current task.
[0046] Step 208: Based on the geometric image, determine the target reference image that matches the geometric image in the reference image set.
[0047] In some embodiments, the mobile device can calculate the image similarity between the geometric image and multiple reference images respectively, and determine the reference image with the highest image similarity from the multiple reference images as the target reference image to match the geometric image.
[0048] Image similarity can include one or more of the following: structural similarity, peak signal-to-noise ratio, mean square error, feature point matching degree, and perceptual image patch similarity.
[0049] Optionally, the mobile device can also determine the target reference image that matches the geometric image by comparing the geometric consistency or semantic consistency of the geometric image with multiple reference images respectively. Step 210: Generate a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint.
[0050] Mobile devices can use a target reference image as a reference to render the geometric skeleton from the target's perspective, thus obtaining a rendered image corresponding to the target's perspective.
[0051] In some embodiments, the mobile device can input a target reference image, a geometric skeleton, a reference viewpoint corresponding to the target reference image, and a target viewpoint into a video generation model. The video generation model uses the target reference image as a reference to render the appearance of the geometric skeleton from the target viewpoint, thereby generating a rendered image corresponding to the target viewpoint.
[0052] Step 212: Construct a 3D scene model based on the rendered images and reference image sets corresponding to multiple target viewpoints.
[0053] Mobile devices can directly construct 3D scene models based on rendered images and reference image sets corresponding to multiple target viewpoints, without first constructing the observation geometry and then using a prior model to predict and complete the observation geometry.
[0054] Since the reference image set includes multiple reference images corresponding to reference viewpoints and multiple target viewpoints, all of which are viewpoints achievable by mobile devices, constructing a 3D scene model based on the rendered images corresponding to multiple target viewpoints and the reference image set not only allows the reconstruction range of the 3D scene model to match the visible area of the mobile device, ensuring that the 3D scene model does not have geometric structures outside the visible area, thus avoiding unnecessary waste of acquisition and computing resources, but also, due to the high fidelity of the rendered images corresponding to multiple target viewpoints, the geometric appearance of the 3D scene model is more realistic and the detail restoration is higher.
[0055] In some embodiments, a mobile device can construct a 3D scene model based on rendered images and reference image sets corresponding to multiple target viewpoints using the 3DGS algorithm. Optionally, the mobile device can also construct a 3D scene model based on rendered images and reference image sets corresponding to multiple target viewpoints using other scene representation schemes such as NeRF.
[0056] In this embodiment, a geometric skeleton is constructed based on a set of reference images. Based on the geometric skeleton, a geometric image corresponding to the target viewpoint is obtained. Then, based on the geometric image, a target reference image matching the geometric image is determined from the set of reference images. Subsequently, using the target reference image as a reference, the geometric skeleton is rendered from the target viewpoint to generate a rendered image corresponding to the target viewpoint. This allows for high consistency between the target reference image and the geometric image in spatial pose and content. The texture, color, and other appearance information of the target reference image are used as rendering constraints to ensure that the rendered content not only has a reasonable appearance but also meets the constraints of geometry, occlusion, contour, and free space in the real scene. Then, based on the rendered images and reference image sets corresponding to multiple target viewpoints, a 3D scene model is constructed. This not only effectively fills the observation blind spots under the original sparse viewpoints and improves the accuracy of 3D scene model reconstruction, but also focuses on the visible area for targeted 3D scene model reconstruction throughout the process, avoiding resource waste caused by redundant modeling and ineffective computation in non-visible areas. Furthermore, it increases the proportion of effective information in the reconstructed 3D scene model, improving the training efficiency of downstream tasks.
[0057] In some embodiments, the mobile device can determine geometric constraint information based on a geometric skeleton, according to a reference viewpoint corresponding to the target reference image and the target viewpoint, and generate a rendered image corresponding to the target viewpoint based on the target reference image and the geometric constraint information. Figure 3 This is a flowchart illustrating the generation of a rendered image corresponding to a target viewpoint based on geometric constraint information, provided in an embodiment of this application. Figure 3 As shown, the steps, based on the target reference image, geometric skeleton, reference viewpoint corresponding to the target reference image, and target viewpoint, may include the following steps: Step 301: Based on the geometric skeleton, determine the movement trajectory from the reference viewpoint corresponding to the target reference image to the target viewpoint.
[0058] In one implementation, the movement trajectory can be a straight path or a curved path. The curved path can be determined by the mobile device using a fitting method such as a Bézier curve or a B-spline curve.
[0059] As one implementation method, the mobile device can determine a constrained feasible trajectory based on the geometric skeleton, using its behavioral information, to move from the pose of the reference viewpoint corresponding to the target reference image to the pose corresponding to the target viewpoint. By imposing constraints on the movement trajectory from the reference viewpoint corresponding to the target reference image to the target viewpoint using the mobile device's behavioral information, it can be ensured that each pose on the movement trajectory is within the actual reachable range of the mobile device, avoiding the generation of virtual viewpoint paths that are physically unexecutable by the mobile device.
[0060] Step 303: Determine the geometric constraint information based on the movement trajectory.
[0061] Figure 4 This is a flowchart illustrating the determination of geometric constraint information based on a movement trajectory, provided as an embodiment of this application. Figure 4 As shown, step 303 may include steps 402 to 408.
[0062] Step 402: Along the movement trajectory, interpolate the pose parameters corresponding to the reference viewpoint and the target viewpoint of the target reference image to obtain multiple pose parameters.
[0063] Pose parameters can include translation parameters and rotation parameters. Translation parameters can be the position coordinates of the viewpoint in three-dimensional space; rotation parameters can be the Euler angles of the viewpoint in three-dimensional space.
[0064] The mobile device can perform linear interpolation on the translation parameters of the reference viewpoint corresponding to the target reference image and the translation parameters corresponding to the target viewpoint along the movement trajectory to obtain multiple translation interpolations; the mobile device can also perform linear interpolation on the rotation parameters of the reference viewpoint corresponding to the target reference image and the rotation parameters corresponding to the target viewpoint along the movement trajectory to obtain multiple rotation interpolations. The number of rotation interpolations is the same as the number of translation interpolations, and the multiple rotation interpolations correspond one-to-one with the multiple translation interpolations.
[0065] Step 404: Determine the movement viewpoints corresponding to the multiple pose parameters based on the multiple pose parameters.
[0066] Since one moving viewpoint corresponds to one translation interpolation and one rotation interpolation, the movable device can determine multiple moving views based on multiple translation interpolations and their corresponding rotation interpolations.
[0067] By interpolating along the movement trajectory to generate multiple continuous moving viewpoints, it is possible to fit the movement path of the mobile device and ensure that the visible area of the mobile device is smoothly connected and continuously covered along the movement trajectory. This provides a coherent and reliable viewpoint constraint for 3D scene reconstruction, which helps to determine more accurate geometric constraint information based on the geometric images corresponding to each moving viewpoint. This ensures that the rendered content not only looks reasonable, but also meets the geometric, occlusion, contour and free space constraints of the real scene.
[0068] Step 406: Based on the geometric skeleton, generate geometric images corresponding to each moving viewpoint.
[0069] Step 408: Determine geometric constraint information based on the geometric images corresponding to each moving viewpoint.
[0070] Geometric constraint information includes one or more of the following: point cloud information, depth information, normal information, contour information, and color information of the geometric image corresponding to each moving viewpoint.
[0071] Through the above implementation method, by generating continuous moving viewpoints along the moving trajectory and generating corresponding geometric images, continuous geometric constraint information along the motion path of the mobile device can be formed, improving the spatial coherence of geometric constraints, providing sufficient and reliable geometric constraints for subsequent reconstruction of the 3D scene, and improving the rationality of the content generated by the geometric skeleton rendering.
[0072] Step 305: Input the target reference image, movement trajectory and geometric constraint information into the video generation model, perform appearance rendering on the geometric skeleton, and obtain the rendered image from the target's perspective.
[0073] The mobile device can input the target reference image, movement trajectory, and geometric constraint information into the video generation model. Under the constraints of the geometric constraint information, the video generation model renders the appearance of the geometric skeleton along the movement trajectory with the target reference image as the starting frame, and generates video data corresponding to the movement trajectory. The mobile device can determine that the last frame of the video data is the rendered image from the target's perspective.
[0074] The video data may include multiple frames of images, each corresponding to a different moving viewpoint and a target viewpoint, with the last frame of the video data corresponding to the target viewpoint.
[0075] Through the above implementation method, continuous video rendering is performed along the movement trajectory with the reference image as the starting frame. The smooth transition between frames can ensure the continuity of the scene appearance and geometric structure of the geometric skeleton, avoid texture jumps and geometric distortions caused by single-frame jump rendering, and select the last frame of the video data as the rendering image of the target viewpoint to ensure that it is consistent with the reference image in terms of visual features and geometric scale, effectively improve the realism and reliability of the rendering results, and ensure the high fidelity of the geometric skeleton.
[0076] In this embodiment, the movement trajectory from the reference viewpoint to the target viewpoint and the corresponding geometric constraint information are first determined based on the geometric skeleton. Then, the video generation model is guided by the geometric constraints to complete the appearance rendering of the target viewpoint. This can constrain the accuracy of the geometric structure of the rendering process according to the target reference image, avoid the geometric drift and texture misalignment problems that are easy to occur in unconstrained rendering, and thus output a rendered image of the target viewpoint with both geometric accuracy and visual consistency. This expands the sparse reference image set and improves the accuracy of the 3D scene model reconstruction.
[0077] In some embodiments, the mobile device can iteratively acquire rendered images corresponding to each target viewpoint multiple times, thereby expanding the sparse set of reference images and improving the accuracy of 3D scene model reconstruction. Figure 5 A flowchart illustrating the iterative acquisition process provided in this application's embodiments. For example... Figure 5 As shown, the method may further include the following steps: Step 501: Obtain the set of reference images corresponding to the current iteration cycle.
[0078] In some embodiments, the mobile device can determine whether the current iteration cycle is less than the cycle threshold. If not, step 503 is executed. If so, the current iteration cycle is determined to be the last iteration cycle, and step 505 is executed.
[0079] The period threshold can be set according to actual needs. When the period threshold is set to 5 iteration periods, if the current iteration period is the 5th iteration period, the mobile device can directly execute step 505.
[0080] Step 503: After generating the rendered images of multiple target perspectives corresponding to the current iteration cycle, add the rendered images of multiple target perspectives corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle to obtain the reference image set corresponding to the next iteration cycle.
[0081] The reference image set corresponding to the next iteration cycle is used to generate the rendered images of multiple target views corresponding to the next iteration cycle, and the multiple target views corresponding to the current iteration cycle are at least partially different from the multiple target views corresponding to the next iteration cycle.
[0082] Step 505: Construct a 3D scene model based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the set of reference images corresponding to the last iteration cycle.
[0083] For example, when the period threshold is set to 5 iteration periods and the current iteration period is the 5th iteration period, the mobile device can construct a 3D scene model based on the rendered images of multiple target perspectives corresponding to the 5th iteration period and the reference image set corresponding to the 5th iteration period.
[0084] Through the above implementation method, by iteratively expanding the reference image set, multiple rendered images from different perspectives can be gradually accumulated, thereby enriching the reference image set. Finally, a 3D scene model is constructed based on the iteratively complete reference image set and the rendered images from multiple target perspectives corresponding to the last iteration cycle. This can effectively improve the integrity and geometric accuracy of the reconstruction results of the 3D scene model, ensuring reliable geometric information coverage and observation constraints within the visible area of the mobile device, and thus providing richer and more accurate data support for the training of downstream tasks of the mobile device.
[0085] In some embodiments, the mobile device can filter the rendered images of multiple target perspectives corresponding to the current iteration cycle, so as to add the high-fidelity rendered images to the reference image set corresponding to the current iteration cycle. Figure 6 This is a flowchart illustrating the filtering of rendered images provided in an embodiment of this application. Figure 6 As shown, the step of adding the rendered images of multiple target perspectives corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle may include the following steps: Step 602: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle.
[0086] A point cloud can refer to a collection of multiple three-dimensional spatial points corresponding to a geometric skeleton, used to represent the three-dimensional geometric structure of the geometric skeleton. The first point cloud information may include the position coordinates of the multiple three-dimensional spatial points corresponding to the geometric skeleton.
[0087] Optionally, the point cloud information may also include one or more of depth information, normal information, and color information.
[0088] Step 604: Determine the second point cloud information corresponding to the first target viewpoint based on the rendered image corresponding to the first target viewpoint.
[0089] The first target perspective is any one of the multiple target perspectives corresponding to the current iteration cycle.
[0090] The mobile device can map the rendered image corresponding to the first target viewpoint to the geometric skeleton through the pose parameters corresponding to the first target viewpoint, thereby determining the second point cloud information corresponding to the first target viewpoint.
[0091] The second point cloud information may include the position coordinates of multiple three-dimensional spatial points corresponding to the first target viewpoint and the geometric skeleton.
[0092] Step 606: Calculate the first distance information between the first point cloud information and the second point cloud information.
[0093] The first distance information is used to reflect the geometric consistency between the rendered image from the target's perspective and the geometric skeleton. The smaller the first distance information, the higher the geometric consistency.
[0094] As one implementation, the first distance information can be the Chamfer distance, which is used to measure the similarity between the point cloud corresponding to the first point cloud information and the point cloud corresponding to the second point cloud information.
[0095] Step 608: Determine if the first distance information is less than or equal to the distance threshold. If not, proceed to step 610; if yes, proceed to step 612.
[0096] Step 610: Do not add the rendered image corresponding to the first target viewpoint to the reference image set.
[0097] If the rendered images corresponding to multiple target viewpoints cannot be added to the reference image set, the mobile device can re-execute the step of generating the rendered image corresponding to the target viewpoint based on the target reference image, geometric skeleton, reference viewpoint corresponding to the target reference image, and target viewpoint, to obtain new rendered images corresponding to multiple target viewpoints, and then execute step 604 again.
[0098] Step 612: Add the rendered image corresponding to the first target viewpoint to the reference image set.
[0099] In some embodiments, if the current iteration cycle is not the last iteration cycle, the mobile device may add the rendered image corresponding to the first target viewpoint to the reference image set.
[0100] In some embodiments, when the current iteration cycle is the last iteration cycle, the mobile device can construct a 3D scene model based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the reference image set corresponding to the last iteration cycle, without adding the rendered image corresponding to the first target perspective to the reference image set.
[0101] Through the above implementation method, only rendered images that meet the geometric consistency standard with the geometric skeleton of the current iteration cycle can be included in the reference image set, avoiding the generation of incorrect geometric structures by rendered images with poor geometric consistency, thereby affecting the reconstruction accuracy of the geometric skeleton.
[0102] In some embodiments, the mobile device can generate multiple rendered images corresponding to each target viewpoint in the current iteration cycle, and filter out high-fidelity rendered images for each target viewpoint. Figure 7 This is a flowchart illustrating the filtering of multiple rendered images corresponding to a first target viewpoint, provided as an embodiment of this application. Figure 7 As shown, the method may include the following steps: Step 701 involves repeatedly executing the step of generating a rendered image corresponding to the target viewpoint based on the target reference image, geometric skeleton, reference viewpoint corresponding to the target reference image, and target viewpoint, to obtain multiple rendered images corresponding to the target viewpoint.
[0103] In the current iteration cycle, after determining the target reference image that matches the geometric image from the set of reference images based on the geometric image, the mobile device can repeatedly execute the steps of generating a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, to obtain multiple rendered images corresponding to the target viewpoint.
[0104] Step 703: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle.
[0105] Step 705: Determine the second point cloud information of the multiple rendered images corresponding to the first target viewpoint based on the multiple rendered images corresponding to the first target viewpoint.
[0106] Step 707: Calculate the first distance information between the first point cloud information and the second point cloud information corresponding to the multiple rendered images.
[0107] Step 709: Select the rendering image with the smallest first distance information from multiple rendered images and add it to the reference image set.
[0108] The mobile device can compare the first distance information of multiple rendered images corresponding to the first target viewpoint, determine the rendered image with the smallest first distance information, and add the rendered image with the smallest first distance information to the reference image set.
[0109] In some embodiments, the mobile device can select the rendering image with the smallest first distance information from multiple rendering images corresponding to the first target viewpoint, and determine whether the first distance information of the rendering image is less than or equal to a distance threshold. If so, the rendering image with the smallest first distance information in the first target viewpoint is added to the reference image set.
[0110] In some embodiments, when the current iteration cycle is the last iteration cycle, the mobile device can construct a 3D scene model based on the rendered image with the smallest first distance information among multiple target perspectives corresponding to the last iteration cycle, and the reference image set corresponding to the last iteration cycle.
[0111] Through the above implementation method, it can be ensured that the rendered image with the smallest first distance information added to the reference image set has the best geometric consistency and the highest fidelity with the geometric skeleton of the current iteration cycle. This allows for the continuous supplementation of reliable observation constraints during the iterative reconstruction process, avoiding the introduction of geometric errors by rendered images with larger first distance information. Consequently, the overall quality of the reference image set is effectively improved, and the reconstruction accuracy and reconstruction range of the geometric skeleton during the iterative optimization process are enhanced.
[0112] In some embodiments, the mobile device can construct a geometric skeleton based on a set of reference images using a prior model. Figure 8 A flowchart illustrating the construction of a geometric skeleton provided for an embodiment of this application. (See attached diagram.) Figure 8 As shown, the step of constructing a geometric skeleton based on a set of reference images may include the following steps: Step 802: Perform feature point matching on multiple reference images, and determine multiple three-dimensional spatial points based on the multiple feature points obtained from the matching.
[0113] Each point in three-dimensional space corresponds to a feature point in at least one reference image.
[0114] In some embodiments, the mobile device can perform feature point matching on multiple reference images using the Structure from Motion (SfM) algorithm, and determine multiple three-dimensional spatial points based on the matched feature points.
[0115] Step 804: Construct the observation geometry based on multiple three-dimensional spatial points.
[0116] Figure 9 A flowchart illustrating the construction of observation geometry provided in this application embodiment. For example... Figure 9 As shown, step 804 may include steps 901 to 905.
[0117] Step 901: Determine the confidence level and reprojection residual corresponding to each three-dimensional spatial point.
[0118] The confidence level of a 3D spatial point is related to the number of reference images corresponding to that 3D spatial point, and the higher the number of reference images corresponding to the 3D spatial point, the higher the confidence level of the 3D spatial point; the reprojection residual is used to characterize the deviation between the mapping position of the 3D spatial point in the corresponding reference image and the feature point.
[0119] Step 903: From multiple three-dimensional spatial points, select multiple target three-dimensional spatial points whose corresponding confidence level is greater than or equal to the confidence level threshold and whose corresponding reprojection residual is greater than or equal to the residual threshold.
[0120] Step 905: Construct the observation geometry based on multiple target three-dimensional spatial points.
[0121] It is understandable that the confidence level of a 3D spatial point represents the credibility of the real scene corresponding to that point, and the reprojection residual represents the geometric fitting accuracy of the geometric skeleton to that point. Therefore, by filtering out 3D spatial points with low credibility and low geometric fitting accuracy through confidence level and reprojection residual, it helps to ensure that the constructed observation geometry has sufficient credible geometric data support, thereby improving the reconstruction accuracy of the observation geometry.
[0122] In some embodiments, the mobile device constructs observation geometry based on multiple target three-dimensional spatial points using a 3DGS algorithm.
[0123] Step 806: Input the observed geometry into the prior model, and predict and complete the missing structures in the observed geometry based on the prior model to obtain the predicted geometry.
[0124] In some embodiments, the prior model may be a SAM 3D model. The mobile device can input the observed geometry into the SAM 3D model and predict and complete the missing structures in the observed geometry based on the SAM 3D model to obtain the predicted geometry.
[0125] Step 808: The observed geometry and the predicted geometry are fused to obtain the geometric skeleton.
[0126] In some embodiments, the mobile device can fuse multiple three-dimensional spatial points corresponding to the observed geometry and multiple three-dimensional spatial points corresponding to the predicted geometry to obtain a geometric skeleton.
[0127] In this embodiment, an observation geometry with real observation basis is first constructed by feature point matching. Then, the missing structure of the observation geometry is supplemented and geometric fusion is completed by using a prior model. This not only preserves the geometric accuracy and reliability of the real observation data, but also effectively fills the structural gaps in the observation blind spots. Compared with the method of directly inputting the reference image into the prior model for end-to-end reconstruction, it can avoid the problems of geometric drift, detail distortion and insufficient overall consistency that are prone to occur in end-to-end reconstruction models, thereby improving the integrity and robustness of 3D scene reconstruction under sparse observation conditions.
[0128] In some embodiments, the mobile device may perform a viewpoint evaluation on a plurality of first initial viewpoints to filter out a plurality of target viewpoints from the plurality of first initial viewpoints of the mobile device. Figure 10 A flowchart illustrating the determination of multiple target perspectives provided in embodiments of this application. For example... Figure 10 As shown, the method may include the following steps: Step 1002: Based on the geometric skeleton, determine multiple first initial viewpoints and generate geometric images corresponding to the multiple first initial viewpoints respectively.
[0129] Step 1004: Determine the view evaluation value corresponding to each of the multiple first initial viewpoints based on the geometric images corresponding to the multiple first initial viewpoints.
[0130] The viewpoint evaluation value includes one or more of the first support ratio, the second support ratio, and the viewpoint dispersion. Each pixel in the geometric image has a target label, which is used to indicate whether the three-dimensional space point corresponding to the pixel belongs to the observed geometry or the predicted geometry.
[0131] As one implementation, the mobile device can determine the first support ratio corresponding to each first initial viewpoint based on the proportion of the number of pixels belonging to the observed geometry indicated by the target label in the geometric image corresponding to each first initial viewpoint to the total number of pixels in each geometric image.
[0132] By selecting the first support ratio, the first initial viewpoint with sufficient support from the reference image can be prioritized, which helps to reduce the proportion of purely predicted structures during the rendering process and ensures the geometric credibility of the rendering results.
[0133] As one implementation, the mobile device can determine the second support ratio corresponding to each first initial viewpoint based on the ratio of the sum of the number of pixels in the geometric image corresponding to each first initial viewpoint that the target label indicates belongs to the observed geometry and the number of pixels that the target label indicates belongs to the predicted geometry to the total number of pixels in each geometric image.
[0134] By using the second support ratio to filter perspectives, invalid perspectives with large areas of no effective geometric information can be avoided, thereby increasing the scene information density of the supplementary rendered image.
[0135] As one implementation method, the mobile device can calculate the second distance information between the pose corresponding to each first initial viewpoint and the poses corresponding to other first initial viewpoints to obtain the viewpoint discreteness corresponding to each first initial viewpoint.
[0136] The second distance information can be Euclidean distance. The mobile device can calculate the Euclidean distance between the pose corresponding to each first initial viewpoint and the poses corresponding to other first initial viewspoints, and determine the viewpoint dispersion corresponding to each first initial viewpoint based on the average or weighted average of multiple Euclidean distances.
[0137] The smaller the average or weighted average of multiple second distance information, the smaller the view dispersion corresponding to the first initial viewpoint. It can be understood that a smaller average or weighted average of multiple second distance information indicates that the first initial viewpoint is closer to other viewpoints. Therefore, to reduce the redundancy of the selected target viewpoint and ensure that the rendered image of the newly added target viewpoint can carry more differentiated scene information, it is necessary to select a first initial viewpoint with higher view dispersion as the target viewpoint. This helps increase the probability of filling in the blind spots of the reference image set that are not covered by the reference images, thereby improving the modeling accuracy of the geometric skeleton.
[0138] Step 1006: Sort the target viewpoints in descending order of their viewpoint evaluation values, and select the top K viewpoints from the multiple initial viewpoints, where K is a positive integer.
[0139] In this embodiment, by sorting the viewpoint evaluation values from largest to smallest, the top K target viewpoints are selected from multiple first initial viewpoints. This can effectively improve the accuracy of viewpoint selection and scene adaptability, provide high-quality observation constraints for subsequent 3D scene reconstruction, and reduce computational redundancy caused by invalid viewpoints.
[0140] In some embodiments, to avoid using multiple similar first initial perspectives as target perspectives, the mobile device may select one of multiple similar first initial perspectives as the target perspective to ensure the degree of differentiation between different target perspectives. Figure 11 A flowchart illustrating the grouping and filtering target perspective provided in this application embodiment. For example... Figure 11 As shown, the steps are sorted in descending order of the viewpoint evaluation values, and the top K target viewpoints are selected from multiple initial viewpoints. This may include the following steps: Step 1101: Sort the multiple first initial viewpoints in descending order of viewpoint evaluation values to obtain the sorted viewpoint sequence.
[0141] Step 1103: Divide each of the first initial viewpoints contained in the sorted viewpoint sequence into multiple viewpoint groups.
[0142] The similarity between the view evaluation values corresponding to two adjacent first initial viewpoints in each viewpoint group is greater than or equal to the similarity threshold.
[0143] The mobile device can calculate the similarity based on the view evaluation values corresponding to two adjacent first initial viewpoints in the sorted viewpoint sequence, and determine whether the similarity between the view evaluation values corresponding to the two first initial viewpoints is less than the similarity threshold. If so, the two first initial viewpoints are divided into the same viewpoint group.
[0144] Step 1105: Determine the first K view groups, and take the first initial view in each of the first K view groups as the target view.
[0145] In this embodiment, since two adjacent first initial viewpoints after sorting usually have similar observation poses and high scene overlap, by grouping viewpoints and selecting only the first first initial viewpoint in each viewpoint group, it is possible to effectively avoid similar first initial viewpoints being selected as target viewpoints while prioritizing the retention of first initial viewpoints with high viewpoint evaluation values. This ensures that the target viewpoints have good spatial dispersion and angular differences while filling the observation blind zone, thereby effectively expanding the reconstruction range of the geometric skeleton and improving the reconstruction accuracy of the geometric skeleton, thus providing richer and more accurate data support for the training of downstream tasks of mobile devices.
[0146] In some embodiments, the mobile device can sort multiple first initial views in lexicographical order according to the view dispersion, a first value and a second support rate to obtain a sorted view sequence. The sorted view sequence contains each first initial view into multiple view groups, and the view groups arranged in the first K are determined. The first initial view arranged first in each of the first K view groups is taken as the target view. The first value is the difference between 1 and the first support rate.
[0147] Lexicographical sorting can refer to sorting multiple initial perspectives sequentially according to the magnitude of perspective dispersion, first value, and second support rate.
[0148] The mobile device can sort the multiple first initial viewpoints in descending order based on the magnitude of their respective viewpoint dispersion. If two or more first initial viewpoints have the same viewpoint dispersion, they can be sorted in descending order based on the first value of the multiple first initial viewpoints with the same viewpoint dispersion. The second support rate is sorted in the same way, thus completing the lexicographical sorting of the multiple first initial viewpoints.
[0149] Furthermore, mobile devices can divide the various first initial perspectives contained in the same sorting level into multiple perspective groups. Being in the same sorting level can refer to the fact that, in a lexicographical multi-index priority system, the preceding high-priority index values of multiple first initial perspectives are the same.
[0150] For example, multiple initial first views with the same view dispersion but different first values are placed in the same sorting level, while multiple initial first views with the same view dispersion and the same first value but different second support rates are placed in the next sorting level. This prevents initial first views with weaker performance of high-priority indicators from crossing levels and mixing into high-level view groups, thus affecting the accuracy of view group division.
[0151] By sorting by lexicographical order, the first initial viewpoint can be selected in a hierarchical manner based on the priority of viewpoint dispersion, first value, and second support rate. This allows for prioritizing spatial differentiation while simultaneously considering geometric reliability and effective coverage, resulting in a more balanced and reasonable selection of K target viewpoints.
[0152] In some embodiments, when the mobile device is a legged robot, the legged robot can construct a 3D scene model optimized for the visible area of the legged robot based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the reference image set corresponding to the last iteration cycle.
[0153] Figure 12 This is a schematic diagram illustrating the reconstruction of a 3D scene model from the perspective of a legged robot, as provided in an embodiment of this application. Figure 12 As shown, during the iteration process, the legged robot can acquire the reference image set of the current iteration cycle and construct the observation geometry based on the reference image set of the current iteration cycle using the 3DGS algorithm. Then, the fragmented geometry corresponding to the observation geometry is input into the SAM 3D model. The SAM 3D model predicts and completes the missing structures in the observation geometry to obtain the predicted geometry. The legged robot fuses the observation geometry and the predicted geometry to obtain the geometric skeleton.
[0154] Based on a constructed geometric skeleton, the legged robot determines multiple initial viewpoints according to its behavioral information and generates corresponding geometric images for each viewpoint. Based on these images, the robot determines the first support ratio, second support ratio, and viewpoint dispersion for each initial viewpoint. The robot then sorts these initial viewpoints lexicographically based on the viewpoint dispersion, first support ratio, and second support ratio, selecting the top K target viewpoints.
[0155] Based on a constructed geometric skeleton, the legged robot generates a geometric image corresponding to a first target viewpoint. Using this geometric image, it determines a matching target reference image from a set of reference images. Based on the geometric skeleton, it then determines a movement trajectory from the reference viewpoint corresponding to the target reference image to the first target viewpoint. The legged robot determines geometric constraints based on this trajectory and inputs the target reference image, the movement trajectory from the reference viewpoint to the first target viewpoint, and the geometric constraints into a video generation model. Under the constraints of the geometric constraints, the video generation model renders the geometric skeleton along the movement trajectory, starting with the target reference image, and generates video data corresponding to the movement trajectory. The legged robot uses the last frame of the video data as the rendered image for the first target viewpoint.
[0156] The legged robot can determine the Chamfer distance between the first point cloud information and the second point cloud information based on the first point cloud information corresponding to the geometric skeleton and the second point cloud information corresponding to the rendered image of the first target view.
[0157] If the current iteration cycle is not the last iteration cycle and the Chamfer distance is less than or equal to the distance threshold, the legged robot can add the rendered image corresponding to the first target viewpoint to the reference image set. If the rendered images of multiple target views corresponding to the current iteration cycle cannot be added to the reference image set, the legged robot can again input the target reference image corresponding to the first target viewpoint, the movement trajectory of the reference viewpoint corresponding to the target reference image to the first target viewpoint, and the geometric constraint information into the video generation model to obtain new rendered images corresponding to multiple target views respectively.
[0158] When the current iteration cycle is the last iteration cycle, during the generation process, the legged robot can use the 3DGS algorithm to construct a 3D scene model optimized for the legged robot's visual area based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the reference image set corresponding to the last iteration cycle, thereby providing rich data support for the training of the legged robot's downstream tasks.
[0159] The three-dimensional scene reconstruction method described in the above embodiments Figure 13 This is a schematic diagram of a three-dimensional scene reconstruction device provided in an embodiment of this application. Figure 13 As shown, the three-dimensional scene reconstruction device 1300 includes a data acquisition module 1301, a scene construction module 1302, a first generation module 1303, an image matching module 1304, and a second generation module 1305.
[0160] The data acquisition module 1301 is used to acquire a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image.
[0161] Scene construction module 1302 is used to construct a geometric skeleton based on a set of reference images.
[0162] The first generation module 1303 is used to obtain the geometric image corresponding to the target viewpoint based on the geometric skeleton.
[0163] The image matching module 1304 is used to determine a target reference image that matches the geometric image in a set of reference images based on the geometric image.
[0164] The second generation module 1305 is used to generate a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint.
[0165] The scene construction module 1302 is also used to construct a three-dimensional scene model based on the rendered images and reference image sets corresponding to multiple target viewpoints.
[0166] In some embodiments, the three-dimensional scene reconstruction apparatus 1300 includes a computing module.
[0167] The calculation module is used to determine the movement trajectory from the reference viewpoint corresponding to the target reference image to the target viewpoint based on the geometric skeleton.
[0168] The calculation module is also used to determine geometric constraint information based on the movement trajectory.
[0169] The second generation module 1305 is also used to input the target reference image, the movement trajectory and geometric constraint information into the video generation model, and to perform appearance rendering on the geometric skeleton based on the geometric constraint information, the target reference image and the movement trajectory to obtain the rendered image from the target perspective.
[0170] In some embodiments, the three-dimensional scene reconstruction apparatus 1300 includes an image generation module.
[0171] The calculation module is also used to interpolate the pose parameters corresponding to the reference viewpoint and the target viewpoint of the target reference image along the movement trajectory to obtain multiple pose parameters; and to determine the movement viewpoint corresponding to the multiple pose parameters based on the multiple pose parameters.
[0172] The image generation module is used to generate geometric images corresponding to various moving viewpoints based on the geometric skeleton.
[0173] The calculation module is also used to determine geometric constraint information based on the geometric images corresponding to each moving viewpoint.
[0174] In some embodiments, the second generation module 1305 is further configured to input the target reference image, the movement trajectory, and geometric constraint information into the video generation model, and under the constraint of the geometric constraint information, render the appearance of the geometric skeleton along the movement trajectory with the target reference image as the starting frame, and generate video data corresponding to the movement trajectory; and determine the last frame of the video data as the rendered image from the target perspective.
[0175] In some embodiments, the data acquisition module 1301 is further configured to acquire a set of reference images corresponding to the current iteration cycle.
[0176] The data acquisition module 1301 is also used to add the rendered images of multiple target perspectives corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle after generating the rendered images of multiple target perspectives corresponding to the current iteration cycle, so as to obtain the reference image set corresponding to the next iteration cycle. The reference image set corresponding to the next iteration cycle is used to generate the rendered images of multiple target perspectives corresponding to the next iteration cycle.
[0177] The scene construction module 1302 is also used to construct a three-dimensional scene model based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the set of reference images corresponding to the last iteration cycle.
[0178] In some embodiments, the calculation module is further configured to obtain first point cloud information corresponding to the geometric skeleton corresponding to the current iteration cycle; determine second point cloud information corresponding to the first target viewpoint based on the rendered image corresponding to the first target viewpoint, wherein the first target viewpoint is any one of the multiple target viewpoints corresponding to the current iteration cycle; and calculate first distance information between the first point cloud information and the second point cloud information.
[0179] The data acquisition module 1301 is also used to add the rendered image corresponding to the first target viewpoint to the reference image set if the first distance information is less than or equal to the distance threshold.
[0180] In some embodiments, the second generation module 1305 is further configured to perform the step of generating a rendered image corresponding to the target viewpoint multiple times based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, to obtain multiple rendered images corresponding to the first target viewpoint.
[0181] The calculation module is also used to obtain the first point cloud information corresponding to the geometric skeleton corresponding to the current iteration cycle; determine the second point cloud information of the multiple rendered images corresponding to the first target viewpoint based on the multiple rendered images corresponding to the first target viewpoint; and calculate the first distance information between the first point cloud information and the second point cloud information corresponding to the multiple rendered images respectively.
[0182] The data acquisition module 1301 is also used to select the rendering image with the smallest first distance information from multiple rendering images and add it to the reference image set.
[0183] In some embodiments, the scene construction module 1302 is further configured to perform feature point matching on multiple reference images, determine multiple three-dimensional spatial points based on the multiple matched feature points, wherein the three-dimensional spatial points correspond to feature points in at least one reference image; construct observation geometry based on the multiple three-dimensional spatial points; input the observation geometry into a prior model, predict and complete the missing structures in the observation geometry based on the prior model to obtain the predicted geometry; and fuse the observation geometry and the predicted geometry to obtain the geometric skeleton.
[0184] In some embodiments, the calculation module is further configured to determine the confidence level and reprojection residual corresponding to each three-dimensional spatial point; the confidence level corresponding to the three-dimensional spatial point is related to the number of reference images corresponding to the three-dimensional spatial point; the reprojection residual is used to characterize the deviation between the mapping position of the three-dimensional spatial point in the corresponding reference image and the feature point.
[0185] The calculation module is also used to filter out multiple target three-dimensional spatial points from multiple three-dimensional spatial points, where the corresponding confidence level is greater than or equal to the confidence level threshold and the corresponding reprojection residual is greater than or equal to the residual threshold.
[0186] The scene construction module 1302 is also used to construct observation geometry based on multiple target 3D spatial points.
[0187] In some embodiments, the calculation module is further configured to determine multiple first initial viewpoints based on the geometric skeleton and the behavior information of the mobile device, and generate geometric images corresponding to the multiple first initial viewpoints respectively; determine viewpoint evaluation values corresponding to the multiple first initial viewpoints respectively based on the geometric images corresponding to the multiple first initial viewpoints respectively; sort the viewpoint evaluation values in descending order, and select the top K target viewpoints from the multiple first initial viewpoints, where K is a positive integer.
[0188] In some embodiments, the calculation module is further configured to sort multiple first initial perspectives in descending order of perspective evaluation values to obtain a sorted perspective sequence; divide each first initial perspective contained in the sorted perspective sequence into multiple perspective groups; wherein the similarity between the perspective evaluation values corresponding to two adjacent first initial perspectives in each perspective group is greater than or equal to a similarity threshold; determine the first K perspective groups, and take the first first initial perspective in each of the first K perspective groups as the target perspective.
[0189] In some embodiments, the viewpoint evaluation value includes one or more of a first support ratio, a second support ratio, and the viewpoint evaluation value. Each pixel in the geometric image has a target label, which is used to indicate whether the three-dimensional spatial point corresponding to the pixel belongs to the observed geometry or the predicted geometry.
[0190] The calculation module is also used to determine the first support ratio corresponding to each first initial viewpoint based on the proportion of the number of pixels in the geometric image corresponding to each first initial viewpoint that the target label indicates belongs to the observed geometry to the total number of pixels in each geometric image.
[0191] The calculation module is also used to determine the second support rate corresponding to each first initial viewpoint based on the sum of the number of pixels in the geometric image corresponding to each first initial viewpoint that the target label indicates belongs to the observed geometry and the number of pixels that the target label indicates belongs to the predicted geometry, as a proportion of the total number of pixels in each geometric image.
[0192] The calculation module is also used to calculate the second distance information between the pose corresponding to each first initial viewpoint and the poses corresponding to other first initial viewspoints, so as to obtain the viewpoint evaluation value corresponding to each first initial viewpoint.
[0193] Figure 14 This is a structural block diagram of an electronic device provided in an embodiment of this application. Figure 14 As shown, the electronic device 1400 may include a memory 1402 and a processor 1401. The memory 1402 stores a computer program. When the computer program is executed by the processor 1401, the electronic device 1400 implements the three-dimensional scene reconstruction method as described in the above embodiments.
[0194] Processor 1401 may include one or more processing cores. Processor 1401 connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, processor 1401 may be implemented using at least one hardware form of digital signal processing, field-programmable gate array, or programmable logic array. Processor 1401 may integrate one or more of the following: Central Processing Unit (CIU), Graphics Processing Unit (GIU), and modem. The CIU primarily handles the operating system, user interface, and applications; the GIU is responsible for rendering and drawing displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1401 and may be implemented separately through a communication chip.
[0195] The memory 1402 may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the computer device.
[0196] This application discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the three-dimensional scene reconstruction method as described in the above embodiments.
[0197] This application discloses a computer program product, which includes a computer program, and when the computer program is executed by a processor, the processor implements the three-dimensional scene reconstruction method as described in the above embodiments.
[0198] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, etc.
[0199] The above description is merely a specific example of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for reconstructing a three-dimensional scene, characterized in that, include: Obtain a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image; Construct a geometric skeleton based on the reference image set; Based on the geometric skeleton, obtain the geometric image corresponding to the target viewpoint; Based on the geometric image, a target reference image matching the geometric image is determined from the set of reference images; Based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, a rendered image corresponding to the target viewpoint is generated; A three-dimensional scene model is constructed based on the rendered images corresponding to multiple target viewpoints and the set of reference images.
2. The method according to claim 1, characterized in that, The step of generating a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, a reference viewpoint corresponding to the target reference image, and the target viewpoint includes: Based on the geometric skeleton, a movement trajectory is determined from the reference viewpoint corresponding to the target reference image to the target viewpoint; Based on the movement trajectory, determine the geometric constraint information; The target reference image, the movement trajectory, and the geometric constraint information are input into the video generation model, and the geometric skeleton is rendered to obtain a rendered image from the target's perspective.
3. The method according to claim 2, characterized in that, The step of determining geometric constraint information based on the movement trajectory includes: Along the movement trajectory, interpolation is performed on the reference viewpoint corresponding to the target reference image and the pose parameters corresponding to the target viewpoint to obtain multiple pose parameters; Based on the plurality of pose parameters, determine the movement viewpoint corresponding to each of the plurality of pose parameters; Based on the geometric skeleton, generate geometric images corresponding to each moving viewpoint; The geometric constraint information is determined based on the geometric images corresponding to each moving viewpoint.
4. The method according to claim 2, characterized in that, The step of inputting the target reference image, the movement trajectory, and the geometric constraint information into the video generation model, and performing appearance rendering on the geometric skeleton to obtain a rendered image from the target's perspective includes: The target reference image, the movement trajectory, and the geometric constraint information are input into the video generation model. Under the constraint of the geometric constraint information, the video generation model renders the appearance of the geometric skeleton along the movement trajectory with the target reference image as the starting frame, and generates video data corresponding to the movement trajectory. The last frame of the video data is determined to be the rendered image from the target viewpoint.
5. The method according to claim 1, characterized in that, The acquisition of the reference image set includes: Obtain the set of reference images corresponding to the current iteration cycle; The method further includes: After generating the rendering images of multiple target perspectives corresponding to the current iteration cycle, the rendering images of multiple target perspectives corresponding to the current iteration cycle are added to the reference image set corresponding to the current iteration cycle to obtain the reference image set corresponding to the next iteration cycle. The reference image set corresponding to the next iteration cycle is used to generate the rendering images of multiple target perspectives corresponding to the next iteration cycle. The step of constructing a 3D scene model based on the rendered images corresponding to multiple target viewpoints and the reference image set includes: A 3D scene model is constructed based on the rendered images of multiple target perspectives corresponding to the last iteration cycle and the set of reference images corresponding to the last iteration cycle.
6. The method according to claim 5, characterized in that, The step of adding the rendered images of multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle includes: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle; Based on the rendered image corresponding to the first target viewpoint, determine the second point cloud information corresponding to the first target viewpoint, wherein the first target viewpoint is any one of the multiple target views corresponding to the current iteration cycle; Calculate the first distance information between the first point cloud information and the second point cloud information; If the first distance information is less than or equal to the distance threshold, then the rendered image corresponding to the first target viewpoint is added to the reference image set.
7. The method according to claim 5, characterized in that, Before adding the rendered images of the multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle, the method further includes: By repeatedly executing the step of generating a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint, multiple rendered images corresponding to the first target viewpoint are obtained. The step of adding the rendered images of multiple target viewpoints corresponding to the current iteration cycle to the reference image set corresponding to the current iteration cycle includes: Obtain the first point cloud information corresponding to the geometric skeleton of the current iteration cycle; Based on the multiple rendered images corresponding to the first target viewpoint, determine the second point cloud information of the multiple rendered images corresponding to the first target viewpoint; Calculate the first distance information between the first point cloud information and the second point cloud information corresponding to the multiple rendered images respectively; From the multiple rendered images, the rendered image with the smallest first distance information is selected and added to the reference image set.
8. The method according to claim 1, characterized in that, The step of constructing a geometric skeleton based on the reference image set includes: Feature point matching is performed on the multiple reference images, and multiple three-dimensional spatial points are determined based on the multiple feature points obtained by matching, wherein the three-dimensional spatial points correspond to feature points in at least one reference image; Based on the multiple three-dimensional spatial points, construct the observation geometry; The observed geometry is input into the prior model, and the missing structures in the observed geometry are predicted and filled in according to the prior model to obtain the predicted geometry. The observed geometry and the predicted geometry are fused to obtain the geometric skeleton.
9. The method according to claim 8, characterized in that, The construction of the observation geometry based on the plurality of three-dimensional spatial points includes: The confidence level and reprojection residual corresponding to each of the three-dimensional spatial points are determined; the confidence level corresponding to each three-dimensional spatial point is related to the number of reference images corresponding to the three-dimensional spatial point; the reprojection residual is used to characterize the deviation between the mapping position of the three-dimensional spatial point in the corresponding reference image and the feature point; From the plurality of three-dimensional spatial points, multiple target three-dimensional spatial points are selected with a confidence level greater than or equal to the confidence level threshold and a reprojection residual greater than or equal to the residual threshold. The observation geometry is constructed based on the multiple target three-dimensional spatial points.
10. The method according to claim 1, characterized in that, Before obtaining the geometric image corresponding to the target viewpoint based on the geometric skeleton, the method further includes: Based on the geometric skeleton, multiple first initial viewpoints are determined, and geometric images corresponding to the multiple first initial viewpoints are generated respectively; Based on the geometric images corresponding to the plurality of first initial viewpoints, determine the viewpoint evaluation values corresponding to the plurality of first initial viewpoints respectively; The target viewpoints are sorted in descending order of their evaluation values, and the top K viewpoints are selected from the plurality of first initial viewpoints, where K is a positive integer.
11. The method according to claim 10, characterized in that, The step of sorting the target viewpoints according to their evaluation values from largest to smallest, and selecting the top K target viewpoints from the plurality of first initial viewpoints, includes: The plurality of first initial viewpoints are sorted in descending order of the viewpoint evaluation values to obtain a sorted viewpoint sequence. The sorted view sequence is divided into multiple view groups, wherein the similarity between the view evaluation values of two adjacent first initial views in each view group is greater than or equal to a similarity threshold. Determine the first K view groups, and take the first initial view in each of the first K view groups as the target view.
12. A three-dimensional scene reconstruction device, characterized in that, include: The data acquisition module is used to acquire a set of reference images, which includes multiple reference images and a reference viewpoint corresponding to each reference image; The scene construction module is used to construct a geometric skeleton based on the reference image set; The first generation module is used to obtain a geometric image corresponding to the target viewpoint based on the geometric skeleton; An image matching module is used to determine a target reference image that matches the geometric image in the set of reference images, based on the geometric image. The second generation module is used to generate a rendered image corresponding to the target viewpoint based on the target reference image, the geometric skeleton, the reference viewpoint corresponding to the target reference image, and the target viewpoint. The scene construction module is also used to construct a three-dimensional scene model based on the rendered images corresponding to multiple target viewpoints and the set of reference images.
13. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 11.
14. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 11.
