Three-dimensional scene reconstruction method and device and medium

Abstract

The invention discloses a three-dimensional scene reconstruction method and device and a medium. The method comprises the following steps: collecting a multi-view image, global point cloud data and pose information of a three-dimensional scene; generating an object-level initial grid of each target object in the image according to the multi-view image; according to the pose information, performing preliminary registration on the global point cloud data and the object-level initial grid; performing smooth deformation on the preliminarily registered grid by using an optimization objective function combined with geometric constraint, view constraint, Laplacian smooth constraint and Jacobian rigid constraint to obtain an optimized grid; and by taking the global point cloud data as geometric priori and taking the minimum layout optimization loss as a target, carrying out scene-level position, orientation and scale joint optimization on each optimization grid, and outputting an optimized three-dimensional scene model. According to the method, the geometric accuracy and visual quality of three-dimensional scene reconstruction are remarkably improved, and the method is suitable for virtual reality, robot navigation, indoor digital twinning and other applications.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D digital modeling technology, and more specifically, relates to a 3D scene reconstruction method, device and medium. Background Technology

[0002] Indoor 3D reconstruction has long relied on two technical approaches. One approach is the image-based method, represented by multi-view stereo, which obtains dense geometry through feature matching and depth estimation fusion. The other approach is the point cloud method, represented by depth sensors or LiDAR, which directly reconstructs surfaces from spatial sampling. The former is prone to holes, drift, and scale instability in areas with occlusion, viewpoint bias, and weak texture; while the latter, although possessing realistic scale and geometric robustness, is limited by occlusion, specular reflection, and accessibility, often resulting in incomplete data, structural breaks, and noise accumulation.

[0003] In recent years, 3D image generation methods relying on large-scale generative models have made significant progress, enabling the generation of high-fidelity object meshes from single or small numbers of images. However, their priors are usually derived from standardized viewpoint datasets, making them highly sensitive to non-orthogonal viewpoints, cropping, and occlusion in real-world images, leading to geometric distortion and insufficient editability. More critically, the generated results often lack consistency with the true scale of the physical world, making it difficult to align with measured point clouds, resulting in scale errors, pose drift, and scene layout distortion.

[0004] Existing methods also attempt to improve mesh quality through direct vertex optimization or Laplacian smoothing, but when there are significant differences between the initial geometry and the observed mesh, or when there is topological inconsistency, they are prone to over-smoothing, local collapse, or falling into suboptimal conditions. Furthermore, the lack of global fusion and collision constraints for multi-view generation results makes it difficult to fundamentally address issues such as object interpenetration, misalignment, and scale inconsistency. Therefore, it is of significant research importance to find a way to couple the generative advantages of 3D image reconstruction with physical prior depth to obtain 3D scene reconstruction results with high geometric accuracy and high visual quality. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a three-dimensional scene reconstruction method, device and medium, the purpose of which is to improve the geometric accuracy and visual quality of three-dimensional scene reconstruction.

[0006] Based on the above objectives, a first aspect of the present invention provides a method for reconstructing a three-dimensional scene, the method comprising: acquiring multi-view images, global point cloud data, and pose information of a three-dimensional scene; generating an object-level initial mesh for each target object in the images based on the multi-view images; performing preliminary registration of the global point cloud data and the object-level initial mesh based on the pose information; performing smooth deformation of the preliminarily registered mesh using an optimization objective function that combines geometric constraints, view constraints, Laplacian smoothing constraints, and Jacobian rigidity constraints to obtain an optimized mesh; and performing joint optimization of the scene-level position, orientation, and scale of each optimized mesh, with the global point cloud data as the geometric prior and the goal of minimizing layout optimization loss, to output an optimized three-dimensional scene model.

[0007] Preferably, the acquisition of multi-view images, global point cloud data, and pose information of the 3D scene specifically includes: using a mobile scanning system to synchronously acquire multi-view images, global point cloud data, and pose information of the 3D scene during movement; and / or, the 3D scene is an indoor 3D scene; and / or, the multi-view images are multi-view color images.

[0008] Preferably, the step of generating an object-level initial mesh for each target object in the multi-view image specifically includes: for each target object, selecting the image with the highest visual quality score for the target object from the multi-view images, and generating an object-level initial mesh for the target object using the selected image.

[0009] Preferably, the visual quality score includes: sharpness and size score, integrity score, and orthographic score.

[0010] Preferably, based on the pose information, preliminary registration is performed between the global point cloud data and the object-level initial mesh, specifically including: detecting target objects in the global point cloud data and extracting point cloud clusters corresponding to each target object in the global point cloud data; and projecting the point cloud clusters onto the image based on the pose information to perform preliminary registration between the global point cloud data and the object-level initial mesh.

[0011] Preferably, the optimization objective function is: ; in, The optimization objective function is... For the optimized mesh vertex coordinates, For a local Jacobian field, For rendering parameters, For multi-view image supervision loss, The Chamfer distance is used for geometric alignment loss. For regularization terms, For coloring consistency loss, For contour consistency loss, For Laplace regularization weights, For Jacobi regularization weights, For Laplace smoothing constraints, It is a Jacobi rigid constraint.

[0012] Preferred, , , They are respectively: ; in, For use in rendering a collection of views for consistency evaluation; In the first In each view, based on the current grid With parameters The image obtained by differentiable rendering; For the first A real image of a view. It is an L1 norm. For the set of views used for contour consistency assessment, For the current grid In the The outline image rendered from the view. For the first A true outline of each view. For grid The point set obtained by surface sampling For the target point cloud cluster corresponding to this object, x for Sampling points in p for The point in the middle, It is a Euclidean second norm.

[0013] Preferably, the layout optimization loss is: ; in, The layout optimization loss is... The number of objects participating in layout optimization, For objects Surface sampling point set, For object The corresponding point cloud clusters, The Chamfer distance; , Each is an object Object The surrounding box, The intersection-union ratio in three dimensions serves as a collision penalty.

[0014] A second aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional scene reconstruction method as described above.

[0015] A third aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the three-dimensional scene reconstruction method as described above.

[0016] Compared with existing technologies, the advantages of this invention include: providing a three-dimensional scene reconstruction method, device, and medium. During the three-dimensional scene reconstruction process, based on the registration of the global point cloud and the initial mesh at the object level, the scale and layout of the mesh are constrained by the global point cloud to eliminate generation and reconstruction faults. Local Jacobian is used as deformation constraint, and with the cooperation of image and geometric dual supervision, the scene-level position, orientation, and scale of the mesh are jointly optimized, taking into account both the stability and detail fidelity of the three-dimensional reconstruction results. This achieves a high-precision, high-stability, editable, and engineering-usable three-dimensional reconstruction method, which is especially suitable for integrated indoor reconstruction. Attached Figure Description

[0017] Figure 1 A flowchart of a three-dimensional scene reconstruction method provided in an embodiment of the present invention.

[0018] Figure 2 A flowchart illustrating the implementation process of the three-dimensional scene reconstruction method provided in this embodiment of the invention.

[0019] Figure 3 This is a schematic diagram of a data acquisition platform.

[0020] Figure 4 This is a schematic diagram of a vision-language model-guided perspective evaluation provided in an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram illustrating the global layout optimization effect provided in an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The following will further explain and illustrate this technical solution, its implementation process, and its principles.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0025] Furthermore, in the description of this invention, it should be understood that the terms "upper," "lower," "inner," "outer," "horizontal," "vertical," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0026] In the description of this specification, the references to terms such as "an embodiment," "a particular embodiment," or "the embodiment" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0027] This invention provides a method for reconstructing a three-dimensional scene. (See also...) Figure 1 The three-dimensional scene reconstruction method includes steps S100-S400.

[0028] Step S100: Collect multi-view images, global point cloud data, and pose information of the 3D scene.

[0029] In a preferred embodiment, step S100 specifically includes: using a mobile scanning system to simultaneously acquire multi-view images, global point cloud data, and pose information of the three-dimensional scene during the movement.

[0030] In a preferred embodiment, the multi-view image is a multi-view color image.

[0031] Mobile scanning systems, for example, Figure 3 The mobile data acquisition platform shown comprises two LiDARs, one inertial measurement unit, and four cameras. The two LiDARs are mounted horizontally at the top and vertically at the front, respectively. The four cameras are, for example, 5K resolution, synchronously exposed, and arranged in a surround configuration. The mobile scanning system, for example, simultaneously acquires multi-view color images, high-density global point cloud data (e.g., global point cloud data within a 100m effective ranging radius of the LiDAR sensors centered on the acquisition platform's location), and multi-view pose information during indoor data acquisition.

[0032] In a preferred embodiment, the 3D scene is an indoor 3D scene. This method is particularly effective and produces higher quality reconstructions when used for indoor 3D scene reconstruction. This method is also applicable to other 3D scenes, such as outdoor 3D scenes.

[0033] Step S200: Generate an object-level initial mesh for each target object in the image based on the multi-view image; perform preliminary registration of the global point cloud data and the object-level initial mesh based on the pose information.

[0034] In a preferred embodiment, generating an object-level initial mesh for each target object in the multi-view images specifically includes: for each target object, selecting the image with the highest visual quality score from the multi-view images, and generating an object-level initial mesh for the target object using the selected image.

[0035] In a preferred embodiment, the visual quality score includes: sharpness and size score, integrity score, and orthographic score.

[0036] Taking the use of a vision-language model as an example, this section explains the specific process of generating an initial mesh at the object level. For the acquired multi-view images (sets) ), using a vision-language model to detect the main object set A primary object refers to an object that is visually salient, functionally core, or semantically dominant in an image scene, and possesses appropriate visibility and integrity. A primary object is the target object, such as a bed, sofa, dining table, or television set, and is distinguished from objects that serve only as secondary background or decorative elements.

[0037] For any target object in any image, calculate the sharpness and size score of the target object. Integrity score Positive perspective rating The range of values ​​is, for example, The visual quality score of the target object in this image is, for example, the average of these three scores.

[0038] The mean visual quality score for all target objects in the multi-view image is: ; The above scoring process is executed in batches, and the results are stored in JSON format. This enables automatic sorting, removal of abnormal views, and selection of the top performers for each object or scene area. k A high-quality view is used as the input for generation. Preferably, k=1, selecting the highest quality view for each object or scene region as the generation input. Further, using the object labels returned by the vision-language model, Grounded-SAM is guided to perform instance segmentation on the selected image, generating object-level masks, thereby constructing a structured indoor scene asset library. Then, the selected optimal view is input into the 3D generation model to generate the initial 3D mesh of the object (i.e., the initial object-level mesh of the target object).

[0039] In a preferred embodiment, preliminary registration of global point cloud data and object-level initial mesh is performed based on pose information. Specifically, this includes: detecting target objects in global point cloud data and extracting point cloud clusters corresponding to each target object in global point cloud data; and projecting the point cloud clusters onto an image based on pose information to perform preliminary registration of global point cloud data and object-level initial mesh.

[0040] The specific process of preliminary registration of global point cloud data with object-level initial mesh is as follows: First, use an object detection model to detect target objects in the global point cloud data, and extract the corresponding point cloud clusters in the global point cloud data. Then, based on the known pose information, the point cloud clusters are projected onto the image to establish the correspondence between the mesh and the real point cloud. The initial scale alignment adopts a coarse registration method based on feature matching, and the precise alignment is optimized by iterative nearest point (ICP) from point to plane.

[0041] Step S300: Using an optimization objective function that combines geometric constraints, view constraints, Laplace smoothing constraints, and Jacobi rigid constraints, the pre-registered mesh is smoothed and deformed to obtain an optimized mesh.

[0042] In a preferred embodiment, the objective function is: ; in, To optimize the objective function, To optimize the grid vertex coordinates, For a local Jacobian field, These are rendering parameters, i.e. appearance parameters, such as neural shader network parameters, material or lighting parameters; For multi-view image supervision loss, The Chamfer distance is used for geometric alignment loss. For regularization terms, This refers to the loss of color consistency, which is the pixel difference / appearance difference between the rendered image and the actual image. For contour / mask consistency loss, For Laplace regularization weights, For Jacobi regularization weights, For Laplace smoothing constraints, It is a Jacobi rigid constraint.

[0043] In a preferred embodiment, , , They are respectively: ; in, For use in rendering a collection of views for consistency evaluation; In the first In each view, based on the current grid With parameters The image obtained by differentiable rendering; For the first A real image of a view. It is an L1 norm. A collection of views used for contour / mask consistency evaluation. For the current grid In the The binary mask / contour image rendered from the view. For the first The actual mask / contour map of each view (obtained by segmentation / annotation) For grid The point set obtained by surface sampling For the target point cloud cluster corresponding to this object, x for Sampling points in p for The point in the middle, It is a Euclidean second norm.

[0044] In this embodiment, based on registration, the differential mesh deformation combines point cloud geometric constraints and multi-view image constraints. It achieves high-degree-of-freedom and stable deformation capabilities through a local Jacobian field representation based on triangular facets, thereby jointly targeting... Optimization is being performed. This includes the image item. Composed of shading consistency and contour consistency obtained from differentiable rendering; geometric terms A bidirectional Chamfer distance is used to measure the fit between the mesh surface and the target point cloud; the regularization term consists of Laplacian smoothing and Jacobian rigidity constraints to suppress noise gradients and preserve local near-rigid bodies.

[0045] To avoid optimizing displacement directly in the mesh vertex space, this invention learns a local Jacobian field at the mesh triangular facet level. For a set of mesh triangles, for each triangle... Learn one Local Jacobian matrix This is used to encode the desired deformation gradient of the local edge vectors of the triangular facet. Then, the local gradient field is consistently integrated to the vertex positions by solving the following global Poisson equation: ; in, Let be the coordinates of the transformed vertex. To make each triangular facet Local Jacobian field formed by vertical stacking; For coche Laplacian operator, used to express global smoothness and uniformity in the vertex domain; Let be the area-mass matrix based on triangular facets. This energy term makes the solution... The local Jacobian representation is globally consistent with the predicted differential coordinate field (Jacobi field), resulting in smooth and globally consistent mesh deformation. Compared to directly optimizing vertex displacement, the local Jacobian representation allows for larger deformations while better suppressing local wrinkles and distortions, balancing global consistency with local details.

[0046] Step S400: Using global point cloud data as geometric prior and aiming to minimize layout optimization loss, perform joint optimization of scene-level position, orientation, and scale on each optimization grid, and output the optimized 3D scene model.

[0047] In a preferred embodiment, the layout optimization loss is: ; in, To optimize layout and minimize losses, The number of objects participating in layout optimization, For objects Surface sampling point set, For object The corresponding point cloud clusters, The Chamfer distance; , Each is an object Object The surrounding box, The intersection-union ratio in three dimensions serves as a collision penalty.

[0048] In this embodiment, multi-view fusion and layout consistency optimization based on global geometric priors are carried out. Using the global point cloud as a constraint, object-level geometric alignment terms and layout collision constraints are used to globally align each instance mesh with the physical scan data at the real scale, achieving collision-free spatial arrangement and scale correction. Specifically, after object-level alignment and deformation, the global point cloud is used as a geometric prior to jointly optimize the scene-level position, orientation, and scale of each instance mesh, establishing the correspondence between the object mesh and the global point cloud cluster. Collision-free placement is achieved by minimizing global alignment loss and collision loss, ultimately obtaining a complete 3D scene model that is consistent with the real scale, has a reasonable structure, and is editable.

[0049] Taking an indoor 3D scene as an example, combined with Figure 2 The schematic diagram shown illustrates the three-dimensional image reconstruction method provided by this invention. The method is implemented in the following order. (See attached diagram.) Figure 2 The method is divided into three parts: optimal view selection for 3D generation, differentiable mesh deformation stage, and scene consistency layout optimization, which can be further subdivided into the following five stages.

[0050] The first phase involved mobile data acquisition and data storage. In indoor scenes, a mobile LiDAR scanning system was used to simultaneously acquire multi-view color images, global point cloud data, and pose information. The system includes a dual-beam LiDAR, an inertial measurement unit, and four high-resolution cameras, featuring synchronized exposure and a unified time base to ensure cross-modal temporal and spatial alignment.

[0051] The second stage involves viewpoint quality assessment and structured screening. Candidate images are input into a visual-language model, which scores the main objects within the image based on three criteria: sharpness and size, completeness, and orthographic perspective. This results in a viewpoint quality score, which is output in batches using a JSON structure. The model supports automated sorting, outlier removal, and optimal view selection by object or region. The object labels returned by the visual-language model guide an open-set detection and segmentation model for instance segmentation on the selected images. This constructs a well-organized indoor scene asset library associated with image indices, providing structured input for subsequent 3D generation and layout optimization. The viewpoint assessment results guided by the visual-language model are shown below. Figure 4 As shown.

[0052] The third stage involves object-level 3D generation and cross-modal association. Using the selected single-view or few-view samples as the driving force, an initial object-level mesh is generated and physically correlated with the global point cloud. Based on 3D object detection, point cloud clusters semantically consistent with each object are extracted from the global point cloud. Cross-modal matching is then achieved by projecting the point cloud onto the image plane using known camera poses. Initial alignment of scale and pose is achieved through coarse registration, followed by iterative nearest-point registration from point to plane to refine the pose, ensuring accurate positioning of the generated mesh at the real-world scale.

[0053] The fourth stage involves the deformation of differentiable meshes. This invention uses a local Jacobian field as the deformation representation, combining geometric and view constraints for optimization. High-precision geometric fitting and smooth deformation are achieved through the local Jacobian field representation, improving the consistency between the generated mesh and the measured data.

[0054] The fifth stage involves multi-view fusion and layout consistency optimization based on global geometric priors. After object-level alignment and deformation, using the global point cloud as a constraint prior, scene-level position, orientation, and scale are jointly optimized for each instance mesh, and object bounding box overlap is penalized to avoid interleaving. The effect of global layout optimization is as follows: Figure 5 As shown.

[0055] This embodiment considers multimodal collaboration in real-world scenarios as a feasible approach. Utilizing a vision-language model for semantic understanding and viewpoint quality assessment reduces generation distortion during the input phase. Leveraging the realistic geometric prior of the global point cloud, it strongly constrains scale and layout in subsequent stages. The deformation model uses local differential (Jacobi) fields, rather than direct vertex displacements, as basic variables, coupled with dual image and geometric supervision, maintaining structural stability and detail fidelity under significant deformation. Based on these considerations, the proposed 3D scene reconstruction method integrates end-to-end acquisition, filtering, generation, alignment, deformation, and layout. It uses realistic multimodal observations as constraints, semantically driven viewpoint filtering as the entry point, local Jacobian deformation as the core optimization variable, and introduces global geometric priors for multi-view fusion and layout consistency optimization. This deeply couples generational advantages with physical priors, resulting in editable indoor 3D results with consistent geometric and realistic scales.

[0056] This invention provides a 3D scene reconstruction method that utilizes a mobile LiDAR scanning system to collect multimodal data of indoor scenes, including synchronously acquired color images, point cloud data, and pose information. Based on a vision-language model, candidate images are batch-wise evaluated for quality and output in a structured manner. This automatically selects the optimal view from the multi-view image set—one with high object visibility, good integrity, and close to a frontal view—as high-quality input for 3D object generation, and accordingly constructs an editable scene asset library at the instance level. After rigidly registering the generated initial 3D mesh with the measured point cloud, a differentiable mesh deformation module is used to optimize the mesh vertex positions under the joint constraints of multi-view image supervision and point cloud geometric supervision, improving geometric accuracy and surface consistency. Based on global point cloud geometric priors, the scene layout is globally consistent, achieving scale alignment and collision-free placement of objects with the real environment. This invention achieves significantly better geometric accuracy and visual quality than existing methods on both synthetic and real datasets, and is suitable for applications such as virtual reality, robot navigation, and indoor digital twins.

[0057] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional scene reconstruction method described in any of the above embodiments.

[0058] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. The processor 610, memory 620, input / output interface 630, and communication interface 640 are interconnected internally via the bus 650.

[0059] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0060] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0061] The input / output interface 630 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0062] The communication interface 640 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0063] Bus 650 includes a pathway for transmitting information between various components of the device, such as processor 610, memory 620, input / output interface 630, and communication interface 640.

[0064] It should be noted that although the above-described device only shows the processor 610, memory 620, input / output interface 630, communication interface 640, and bus 650, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0065] The electronic devices described above are used to implement the corresponding three-dimensional scene reconstruction methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0066] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the three-dimensional scene reconstruction method as described in any of the above embodiments.

[0067] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0068] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the three-dimensional scene reconstruction method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0069] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the invention as described above, which are not provided in the details for the sake of brevity.

[0070] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the embodiments of the invention may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0071] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0072] It should be understood that the above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that the method... include: Acquire multi-view images, global point cloud data, and pose information of the 3D scene; Generate an object-level initial mesh for each target object in the image based on the multi-view image; Based on the pose information, the global point cloud data and the object-level initial mesh are initially registered; By using an optimization objective function that combines geometric constraints, view constraints, Laplace smoothing constraints, and Jacobi rigid constraints, the initially registered mesh is smoothly deformed to obtain an optimized mesh. Using the global point cloud data as a geometric prior and aiming to minimize the layout optimization loss, the scene-level position, orientation, and scale of each optimization grid are jointly optimized to output the optimized 3D scene model.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The acquisition of multi-view images, global point cloud data and pose information of the three-dimensional scene specifically includes: using a mobile scanning system to synchronously acquire multi-view images, global point cloud data and pose information of the three-dimensional scene during the movement. And / or, the three-dimensional scene is an indoor three-dimensional scene; And / or, the multi-view image is a multi-view color image.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of generating an object-level initial mesh for each target object in the image based on the multi-view image specifically includes: For each target object, the image with the highest visual quality score for that target object is selected from the multi-view images, and the selected image is used to generate the object-level initial mesh for that target object.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that, The visual quality rating includes: sharpness and size rating, integrity rating, and orthographic rating.

5. The three-dimensional scene reconstruction method according to claim 1, characterized in that, Based on the pose information, preliminary registration is performed between the global point cloud data and the object-level initial mesh, specifically including: Detect target objects in the global point cloud data, and extract point cloud clusters corresponding to each target object from the global point cloud data; Based on the pose information, the point cloud clusters are projected onto the image to perform preliminary registration of the global point cloud data with the object-level initial mesh.

6. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The optimization objective function is: ; in, The optimization objective function is... For the optimized mesh vertex coordinates, For a local Jacobian field, For rendering parameters, For multi-view image supervision loss, The Chamfer distance is used for geometric alignment loss. For regularization terms, For coloring consistency loss, For contour consistency loss, For Laplace regularization weights, For Jacobi regularization weights, For Laplace smoothing constraints, It is a Jacobi rigid constraint.

7. The three-dimensional scene reconstruction method according to claim 6, characterized in that, , , They are respectively: ； in, For use in rendering a set of views for consistency evaluation; In the first In each view, based on the current grid With parameters The image obtained by differentiable rendering; For the first A real image of a view, It is an L1 norm. For the set of views used for contour consistency assessment, For the current grid In the The outline image rendered from the view. For the first A true outline of each view. For grid The point set obtained by surface sampling For the target point cloud cluster corresponding to this object, x for Sampling points in p for The point in the middle, It is a Euclidean second norm.

8. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The layout optimization loss is: ; in, The layout optimization loss is... The number of objects participating in layout optimization, For objects Surface sampling point set, For object The corresponding point cloud clusters, The Chamfer distance; , Each is an object Object The surrounding box, The intersection-union ratio in three dimensions serves as a collision penalty.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional scene reconstruction method as described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the three-dimensional scene reconstruction method as described in any one of claims 1-8.

Images