Sparse view scene reconstruction method, electronic device and computer readable storage medium

Abstract

The present application discloses a sparse perspective scene reconstruction method, electronic device and computer-readable storage medium, the method comprising: obtaining perspective images collected at different perspectives in the sparse perspective scene, determining the geometric relationship between different perspectives based on the perspective images, and determining the depth information and three-dimensional point cloud matched by the pixels of the perspective images using the geometric relationship; dividing the sparse perspective scene into a facet set composed of multiple facets based on the three-dimensional point cloud; wherein the facets correspond to facet parameters; based on the perspective images and the depth information, iteratively optimizing the facet parameters and the number of facets in the facet set until the convergence condition is met, and obtaining a reconstruction model corresponding to the sparse perspective scene. In the above manner, the present application can improve the model quality of three-dimensional reconstruction in sparse perspective scenes.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a sparse view scene reconstruction method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Three-dimensional reconstruction is an important branch of computer vision technology. Three-dimensional reconstruction is to infer the three-dimensional structure of the scene based on the data collected by the sensor, and then generate a reconstruction model. In the prior art, when performing three-dimensional reconstruction based on images, it is usually necessary to arrange a large number of cameras in the target scene and collect images from many angles for three-dimensional reconstruction. When the target scene is a sparse perspective scene, there is a large parallax between any two cameras, and the model quality obtained by the conventional three-dimensional reconstruction method is poor. In view of this, how to improve the model quality of three-dimensional reconstruction in sparse perspective scenes has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a sparse perspective scene reconstruction method, an electronic device and a computer-readable storage medium, which can improve the model quality of three-dimensional reconstruction in a sparse perspective scene.

[0004] To solve the above technical problems, the first aspect of the present application provides a method for reconstructing a sparse perspective scene, comprising: obtaining perspective images captured at different perspectives within a sparse perspective scene, determining the geometric relationship between different perspectives based on the perspective images, and using the geometric relationship to determine the depth information and three-dimensional point cloud matched by the pixels of the perspective image; dividing the sparse perspective scene into a face element set composed of multiple face elements based on the three-dimensional point cloud; wherein the face elements correspond to face element parameters; based on the perspective image and the depth information, iteratively optimizing the face element parameters of the face elements in the face element set and the number of face elements until a convergence condition is met, thereby obtaining a reconstruction model corresponding to the sparse perspective scene.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] The above scheme obtains the view images collected by cameras at different view angles in the sparse view scene, determines the corresponding geometric relationship between different view angles based on the view images, thereby aligning the sparse view geometric relationship, and uses the geometric relationship to obtain the depth information of the pixels of the view image in the three-dimensional space, and guides the pixels of the view image to be converted into the three-dimensional space through the geometric relationship obtained by the geometric prior to obtain the three-dimensional point cloud. Based on the three-dimensional point cloud, the sparse view scene is converted into a facet set composed of multiple facets, and each facet has its own facet parameters. The view image and depth information are used as the supervision basis for iterative optimization of the facets in the facet set, and the facet parameters and the number of facets of the facets are continuously optimized until the convergence condition is met, so that the facet parameters and the number of facets in the facet set have a higher degree of adaptability with the sparse view scene, thereby obtaining a reconstruction model corresponding to the sparse view scene. Therefore, this application uses geometric prior guidance to directly perform pose estimation and point cloud computing in the process of building the reconstructed model, which has better adaptability to different lighting conditions and weak texture scenes. After obtaining the three-dimensional point cloud, it is divided into a set of face elements and iteratively optimizes the face element parameters and the number of face elements, thereby optimizing the rendering quality, obtaining better reconstruction effects, and improving the model quality of three-dimensional reconstruction in sparse perspective scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an implementation method of a sparse view scene reconstruction method of the present application;

[0010] Figure 2 It is a flowchart of another implementation method of the sparse view scene reconstruction method of the present application;

[0011] Figure 3 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0012] Figure 4 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0015] The sparse view scene reconstruction method provided in the present application is used to perform three-dimensional reconstruction of scenes with large parallax, and its corresponding execution subject is a processing unit capable of performing image processing.

[0016] See also Figure 1 , Figure 1 : is a flow chart of an implementation method of a sparse view scene reconstruction method of the present application, the method comprising:

[0017] S101: Obtaining perspective images collected at different perspectives in a sparse perspective scene, determining geometric relationships between different perspectives based on the perspective images, and determining depth information and a three-dimensional point cloud matched by pixels of the perspective images using the geometric relationships.

[0018] Specifically, perspective images captured by cameras at different perspectives in a sparse perspective scene are obtained, and the corresponding geometric relationships between different perspectives are determined based on the perspective images, so as to align the sparse perspective geometric relationships, and use the geometric relationships to obtain depth information of pixels of the perspective images in three-dimensional space. The geometric relationships obtained through geometric priors are used to guide the pixels of the perspective images to be converted into three-dimensional space to obtain a three-dimensional point cloud.

[0019] It can be understood that the view images are collected by cameras corresponding to different viewpoints in a sparse view scene. When the number of viewpoints is small (usually within 5 viewpoints), in order to obtain the best possible reconstruction quality, the camera position and angle distribution should be relatively uniform, the imaging should be clear, and the interference of moving foreground should be reduced.

[0020] In one application, a perspective image of any perspective is used as a reference image and a reference coordinate system of the reference image is determined, a conversion function is determined when converting the perspective image of each perspective to the reference coordinate system, and a geometric relationship between different perspectives is obtained based on the conversion function. The depth information matched by the pixels of the perspective image in the three-dimensional space corresponding to the sparse perspective scene is determined using the geometric relationship, and the pixels of the perspective image are converted into the three-dimensional space using the geometric relationship to obtain three-dimensional coordinates, thereby obtaining a three-dimensional point cloud composed of all pixels in the three-dimensional space.

[0021] In another application method, every two perspective images are combined to obtain an image pair, the perspective of any image in the image pair is used as a reference perspective, the conversion relationship between the perspective images in each image pair is determined, the geometric relationship between different perspectives is obtained based on the conversion relationship, the geometric relationship is used to determine the depth of field corresponding to the pixels in the perspective image in the three-dimensional space, the depth information matched by the pixels in the perspective image is obtained, the reference coordinate system corresponding to the sparse perspective scene is obtained, the pixels in the perspective image are converted to the reference coordinate system based on the geometric relationship, and the three-dimensional point cloud composed of all pixels in the three-dimensional space is obtained.

[0022] S102: Divide the sparse view scene into a facet set consisting of a plurality of facets based on the three-dimensional point cloud; wherein the facets correspond to facet parameters.

[0023] Specifically, facet division is performed based on the three-dimensional point cloud, and the sparse view scene is converted into a facet set composed of multiple facets, and each facet has its own facet parameters.

[0024] In one application mode, a preset initial number is obtained, and facet division is performed based on the three-dimensional point cloud, so as to convert the sparse view scene into a facet set composed of the initial number of facets.

[0025] In another application, facet segmentation is performed based on the three-dimensional point cloud, and the sparse view scene is converted into a facet set consisting of an adaptive number of facets, wherein the adaptive number is positively correlated with the number of the three-dimensional point cloud.

[0026] Optionally, the facet parameters include at least some parameters of the center point position, color function, transparency, and two orthogonal tangent vectors. The facet parameters can be customized, and this application does not impose any specific restrictions on this.

[0027] S103: Based on the view image and the depth information, iteratively optimize the facet parameters and the number of facets in the facet set until a convergence condition is met, thereby obtaining a reconstruction model corresponding to the sparse view scene.

[0028] Specifically, the view image and depth information are used as the supervision basis for iterative optimization of the facets in the facet set, and the facet parameters and the number of facets are continuously optimized until the convergence condition is met. This makes the facet parameters and the number of facets in the facet set more compatible with the sparse view scene, thereby obtaining a reconstruction model corresponding to the sparse view scene.

[0029] In one application method, based on the current number of face elements and face element parameters, an estimated model corresponding to this iteration is obtained, the spatial features of the estimated model are obtained, and the estimated loss of this iteration is determined based on the spatial features as well as the perspective image and depth information. Based on the estimated loss, the face element parameters and the number of face elements in the face element set are iteratively optimized until the convergence conditions are met, thereby obtaining a reconstructed model corresponding to the sparse perspective scene.

[0030] In another application method, a reference condition corresponding to the facet parameters is obtained, faces whose facet parameters are lower than the reference condition are cropped, the facet set is updated, and based on the facets in the facet set and their corresponding facet parameters, the estimated structure and estimated depth corresponding to the estimated model corresponding to this iteration are obtained. Based on the estimated result, the estimated depth, and the perspective image and depth information, the estimated loss of this iteration is determined, and based on the estimated loss, the facet parameters and the number of facets in the facet set are iteratively optimized until the convergence condition is met, thereby obtaining a reconstructed model corresponding to the sparse perspective scene.

[0031] Optionally, the convergence condition at least corresponds to the number of iterations, that is, the convergence condition at least includes that the number of iterations satisfies the number condition. In other application scenarios, the convergence condition may also correspond to the estimated loss, which is not specifically limited in this application.

[0032] It can be understood that the present application uses geometric prior guidance to directly perform pose estimation and point cloud computing in the process of building the reconstructed model, and has better adaptability to different lighting conditions and weak texture scenes. After obtaining the three-dimensional point cloud, it is divided into a set of facets and iteratively optimizes the facet parameters and the number of facets, thereby optimizing the rendering quality, obtaining better reconstruction effects, and improving the model quality of three-dimensional reconstruction in sparse perspective scenes.

[0033] In some implementation scenarios, the above-mentioned sparse view scene reconstruction method is implemented based on a three-dimensional reconstruction network model, which includes a positioning network and a rendering network. The positioning network is used to determine the depth information and three-dimensional point cloud matched by the pixels of the view image, and the rendering network is used to obtain a facet set and iteratively optimize the facet parameters and the number of facets in the facet set; wherein the three-dimensional reconstruction network model is trained using training images collected in sparse view scenes and their corresponding training labels, and the training labels include camera parameters matched by the camera corresponding to the training image and the depth information of the training image.

[0034] Specifically, the training images cover as many scene types as possible, such as indoors and outdoors. The camera arrangements corresponding to the training images are as diverse as possible, and include real data sets and synthetic data sets. The data sets correspond to the camera parameters matched by the camera that collected the training images and the depth information of the training images as training labels to achieve supervised training of the 3D reconstruction network model, where the camera pose can be determined based on the camera parameters to obtain the geometric relationship between different perspectives. Therefore, by inputting the perspective images collected in the sparse perspective scene into the 3D reconstruction network model through the end-to-end 3D reconstruction network model obtained through training, the sparse perspective scene 3D reconstruction can be achieved end-to-end, improving the reconstruction efficiency.

[0035] The above scheme obtains the view images collected by cameras at different view angles in the sparse view scene, determines the corresponding geometric relationship between different view angles based on the view images, thereby aligning the sparse view geometric relationship, and uses the geometric relationship to obtain the depth information of the pixels of the view image in the three-dimensional space, and guides the pixels of the view image to be converted into the three-dimensional space through the geometric relationship obtained by the geometric prior to obtain the three-dimensional point cloud. Based on the three-dimensional point cloud, the sparse view scene is converted into a facet set composed of multiple facets, and each facet has its own facet parameters. The view image and depth information are used as the supervision basis for iterative optimization of the facets in the facet set, and the facet parameters and the number of facets of the facets are continuously optimized until the convergence condition is met, so that the facet parameters and the number of facets in the facet set have a higher degree of adaptability with the sparse view scene, thereby obtaining a reconstruction model corresponding to the sparse view scene. Therefore, this application uses geometric prior guidance to directly perform pose estimation and point cloud computing in the process of building the reconstructed model, which has better adaptability to different lighting conditions and weak texture scenes. After obtaining the three-dimensional point cloud, it is divided into a set of face elements and iteratively optimizes the face element parameters and the number of face elements, thereby optimizing the rendering quality, obtaining better reconstruction effects, and improving the model quality of three-dimensional reconstruction in sparse perspective scenes.

[0036] See also Figure 2 , Figure 2 : is a flow chart of another embodiment of the sparse perspective scene reconstruction method of the present application, the method comprising:

[0037] S201: Obtaining perspective images collected at different perspectives in a sparse perspective scene, determining geometric relationships between different perspectives based on the perspective images, and determining depth information and three-dimensional point clouds matched by pixels of the perspective images using the geometric relationships.

[0038] Specifically, perspective images captured by cameras at different perspectives in a sparse perspective scene are obtained, the corresponding geometric relationships between different perspectives are determined based on the perspective images, the depth information of the pixels of the perspective images in the three-dimensional space is obtained using the geometric relationships, and the pixels of the perspective images are guided to be converted into the three-dimensional space to obtain a three-dimensional point cloud using the geometric relationships.

[0039] In one application method, perspective images collected at different perspectives in a sparse perspective scene are obtained to obtain an image pair consisting of perspective images at every two perspectives and a reference coordinate system matching the image pair; wherein the reference coordinate system corresponds to any perspective image in the image pair; image features of the perspective images in the image pair are obtained, and based on the image features and the reference coordinate system, the geometric relationship between different perspectives and the reference point cloud of the pixels in the reference coordinate system are determined; based on the image features and the geometric relationship, the depth information matched by the pixels of the perspective images is determined, and based on the geometric relationship, the reference point clouds in all reference coordinate systems are merged into the reference coordinate system to obtain a three-dimensional point cloud of the pixels in the reference coordinate system; wherein the reference coordinate system corresponds to the sparse perspective scene.

[0040] Specifically, the perspective images captured by the camera at different perspectives in the sparse perspective scene are obtained, the perspective images captured at every two perspectives are combined to obtain multiple image pairs and determine the reference coordinate system for image pair matching, wherein the reference coordinate system corresponds to any perspective image in the image pair, that is, any perspective image in the image pair is used as a reference perspective.

[0041] Furthermore, feature extraction is performed on the view image in the image pair to obtain image features. Based on the image features and the reference coordinate system, the geometric relationship corresponding to the conversion between different view angles is determined, so that geometric information can be fully mined between all combined view angles, so that full-range 3D reconstruction can be performed in sparse view scenes, and compared with the texture feature-based method, it has better adaptability to different lighting conditions and weak texture scenes. Based on the image features and the reference coordinate system, the pixels in the view image are converted to the reference coordinate system to obtain the reference point cloud of the pixels in the reference coordinate system.

[0042] It can be understood that based on image features and geometric relationships, the depth of field corresponding to the pixels of the perspective image in the three-dimensional space is analyzed to obtain the depth information matched by the pixels of the perspective image, and based on the geometric relationship, the reference point clouds in all reference coordinate systems are merged into the reference coordinate system corresponding to the sparse perspective scene, so that the pixels in all perspective images are merged into the same coordinate system, and accurate three-dimensional point clouds of all pixels in the reference coordinate system are obtained.

[0043] It should be noted that obtaining image features of the viewpoint image in the image pair and determining, based on the image features and a reference coordinate system, the geometric relationship between different viewpoints and the reference point cloud of pixels in the reference coordinate system include: obtaining image features of the viewpoint image in the image pair and determining, based on the image features and a reference coordinate system, the camera parameters corresponding to the camera that captures the viewpoint image; and determining, based on the image features and the camera parameters, the geometric relationship between different viewpoints and the reference point cloud of pixels in the reference coordinate system.

[0044] Specifically, feature extraction is performed on the view image in the image pair to obtain image features. Based on the image features and the reference coordinate system, the camera parameters corresponding to the camera that captures the view image are estimated, wherein the camera parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters are parameters that describe the internal properties of the camera, including at least focal length, principal point and distortion coefficient, and the camera extrinsic parameters are parameters that describe the position and posture of the camera in the reference coordinate system, usually including rotation matrix and translation vector. Therefore, estimating the camera parameters can determine the camera posture of the camera at the corresponding view when capturing the view image.

[0045] Furthermore, based on the image features and the camera pose fed back by the camera parameters, the geometric relationship corresponding to the conversion between different perspectives is determined, and the pixels in the perspective image are converted to the reference coordinate system to obtain the reference point cloud of the pixels in the reference coordinate system. Therefore, the camera pose and reference point cloud are estimated based on the geometric relationship. On the basis of fully mining the geometric information, the pose calculation under the constraints of the geometric relationship is performed to reduce the prior information required for 3D reconstruction.

[0046] S202: Divide the sparse view scene into a facet set consisting of a plurality of facets based on the three-dimensional point cloud; wherein the facets correspond to facet parameters.

[0047] Specifically, facet division is performed based on the three-dimensional point cloud, and the sparse view scene is converted into a facet set composed of multiple facets, and each facet has its own facet parameters.

[0048] S203: Based on the view image and the depth information, iteratively optimize the facet parameters and the number of facets in the facet set until a convergence condition is met, thereby obtaining a reconstruction model corresponding to the sparse view scene.

[0049] Specifically, the view image and depth information are used as the supervision basis for iterative optimization of the facets in the facet set, and the facet parameters and the number of facets are continuously optimized until the convergence condition is met to obtain the reconstructed model corresponding to the sparse view scene.

[0050] In one application method, based on the view image and the facet parameters, the rendering quality loss when rendering a facet set is determined, and based on the depth information and the facet parameters, the depth loss corresponding to the facets in the facet set is determined; at least based on the rendering quality loss and the depth loss, the facet parameters and the number of facets in the facet set are iteratively optimized.

[0051] Specifically, the estimated luminosity and estimated structure of the facet of the current facet parameter when rendering are determined, the image features corresponding to the perspective image are used as supervision basis, and the rendering quality loss is obtained based on the image features, the estimated luminosity and the estimated structure, and the estimated depth of the facet of the current facet parameter at different perspectives is determined, the obtained depth information is used as supervision basis, and the depth loss is obtained based on the depth information and the estimated depth.

[0052] Furthermore, based on at least the rendering quality loss and the depth loss, the facet parameters and the number of facets in the facet set are iteratively optimized to make the number of facets more reasonable and adjust the facet parameters with higher precision. Therefore, the view image and the obtained depth information are reused, and the iterative optimization process in the rendering stage is continuously supervised to improve the matching degree between the optimized facets and the sparse view scene.

[0053] Optionally, the facet parameters correspond to reference conditions, and the facets may be iteratively optimized based on the facet parameters.

[0054] In an application scenario, the facet parameters include the transparency of the facets, and the transparency corresponds to a transparency threshold. Wherein, at least based on the rendering quality loss and the depth loss, before iteratively optimizing the facet parameters and the number of facets in the facet set, it also includes: cutting the facets in the facet set whose transparency is lower than the transparency threshold, and updating the facet set.

[0055] Specifically, the transparency of each facet in the facet set is obtained, and the facets with transparency lower than the transparency threshold are screened out and cropped, thereby updating the facet set to reduce the probability of generating floating objects due to overfitting, thereby improving the viewing effect.

[0056] It should be noted that transparency lower than the transparency threshold means that other areas can be observed through the corresponding surface element. Therefore, after cutting out the surface elements with transparency lower than the transparency threshold, the generation of floating objects can be reduced.

[0057] It should be noted that, at least based on the rendering quality loss and the depth loss, the facet parameters and the number of faces in the facet set are iteratively optimized, including: obtaining the depth gradient matched by the depth information and the facet normal of the facet in the facet set, and obtaining the consistency loss based on the facet normal and the depth gradient; based on the rendering quality loss, the depth loss and the consistency loss, the facet parameters and the number of faces in the facet set are iteratively optimized.

[0058] Specifically, the depth gradient matched by the depth information and the facet normal corresponding to the facet in the facet set are obtained, and the consistency loss is determined based on the consistency between the facet normal and the depth gradient. Based on the rendering quality loss, depth loss and consistency loss, the facet parameters and number of the facets in the facet set are jointly iteratively optimized, thereby improving the smoothness of the model surface during the three-dimensional reconstruction process.

[0059] S204: In response to obtaining the specified viewing angle, obtaining associated facets associated with the specified viewing angle in the reconstructed model, performing surface depth reconstruction based on facet parameters of the associated facets, and obtaining a reference surface model.

[0060] Specifically, when a specified viewing angle is obtained, the face elements associated with the specified viewing angle in the reconstructed model are obtained as associated face elements, and surface depth reconstruction is performed based on the face element parameters of the associated face elements to obtain a reference surface model at the specified viewing angle, so that the content displayed at the specified viewing angle is more refined.

[0061] Optionally, surface depth reconstruction is performed by truncated signed distance fusion to obtain the surface geometry of the reference surface model.

[0062] S205: Perform semantic segmentation on the reference surface model to obtain a region to be corrected in the reference surface model, adjust the reference surface model based on the region to be corrected, and obtain a target surface model corresponding to the reconstructed model at a specified viewing angle; wherein the reflectivity of the region to be corrected is greater than a reflectivity threshold or the transparency is less than a transparency threshold.

[0063] Specifically, semantic segmentation is performed on the reference surface model, and the area to be corrected is obtained from the reference surface model, wherein the reflectivity of the area to be corrected is greater than a reflectivity threshold or the transparency is less than a transparency threshold, that is, the area to be corrected has the characteristics of high reflectivity or low transparency.

[0064] Furthermore, the reference surface model is adjusted based on the area to be corrected, and the corresponding area is extracted from the reference surface model for surface fitting to optimize potential broken surface problems, so as to obtain the target surface model corresponding to the reconstructed model at a specified viewing angle and improve the accuracy of the target surface model.

[0065] In some implementation scenarios, the above-mentioned sparse view scene reconstruction method is implemented based on a three-dimensional reconstruction network model, which includes a positioning network and a rendering network. The positioning network is used to determine the depth information and three-dimensional point cloud matched by the pixels of the view image, and the rendering network is used to obtain a facet set and iteratively optimize the facet parameters and the number of facets in the facet set; wherein the three-dimensional reconstruction network model is trained using training images collected in sparse view scenes and their corresponding training labels, and the training labels include camera parameters matched by the camera corresponding to the training image and the depth information of the training image.

[0066] Specifically, during the training process, the 3D reconstruction network model is iteratively optimized through the adam optimizer. The loss function is divided into the loss of the positioning network and the rendering network, that is, the overall loss function is:

[0067] (1)

[0068] in, represents the loss function of the positioning network, Represents the loss function of the rendering network.

[0069] Furthermore, the localization network is used to estimate the depth information and 3D point cloud matched by the pixels of the camera parameter-determined view image. The corresponding loss function includes the regression loss corresponding to the pixels in the image pair in the reference coordinate system and the alignment loss between different image pairs:

[0070] (2)

[0071] in, represents the regression loss, Represents the alignment loss, where the positioning network has a corresponding confidence when estimating the position of pixels in the view image, and the regression loss corresponds to the weighted sum of the losses of each pixel position, where the weight is the confidence of the network output.

[0072] It should be noted that the rendering network is used for high-quality model generation and rendering, and the corresponding loss functions include rendering quality loss, depth loss, and consistency loss:

[0073] (3)

[0074] in, represents the rendering quality loss, which is defined as the weighted average of the photometric error and structural similarity error between the rendered image and the true value; represents the depth loss, which is defined as the corresponding loss between the depth information and the depth prediction output at the current viewing angle; Represents the consistency loss between the surfel normal and the depth gradient.

[0075] It can be understood that the trained 3D reconstruction network model can realize the above-mentioned sparse view scene reconstruction method. When the view images collected at different viewpoints in the sparse view scene are input into the 3D reconstruction network model, the positioning network outputs the depth information and 3D point cloud matched by the pixels of the view image, and divides the sparse view scene into a facet set composed of multiple facets in the rendering network. Based on the view image and depth information, the facet parameters and the number of facets in the facet set are iteratively optimized to obtain the reconstruction model corresponding to the sparse view scene.

[0076] In this embodiment, in addition to ensuring the rendering effect, when a specified viewing angle is obtained, the face elements associated with the specified viewing angle in the reconstructed model are obtained as associated face elements, and the surface depth reconstruction is performed based on the face element parameters of the associated face elements to obtain a reference surface model under the specified viewing angle, so that the content displayed under the specified viewing angle is more refined. The reference surface model is semantically segmented, the area to be corrected is obtained from the reference surface model, the reference surface model is adjusted based on the area to be corrected, the corresponding area is extracted from the reference surface model for surface fitting, the potential broken surface problem is optimized, and the target surface model corresponding to the reconstructed model under the specified viewing angle is obtained, thereby improving the accuracy of the target surface model. Therefore, the present application can obtain relatively accurate geometric information, generate surface reconstruction results under different viewing angles, avoid geometric ambiguity, and can be used for subsequent processes such as material separation and collision detection.

[0077] See also Figure 3 , Figure 3 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 30 includes a memory 301 and a processor 302 coupled to each other, wherein the memory 301 stores program data (not shown), and the processor 302 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0078] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0079] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0080] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0082] The above description is only an implementation method of the present application, and does not limit the protection scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the protection scope of the present application.

Claims

1. A sparse view scene reconstruction method, characterized in that: The method comprises: Acquire perspective images collected at different perspectives in a sparse perspective scene, determine geometric relationships between different perspectives based on the perspective images, and use the geometric relationships to determine depth information and three-dimensional point clouds matched by pixels of the perspective images; Dividing the sparse view scene into a facet set consisting of a plurality of facets based on the three-dimensional point cloud; wherein the facets correspond to facet parameters; Iteratively optimizing the facet parameters of the facets and the number of facets in the facet set based on the view image and the depth information until a convergence condition is met, thereby obtaining a reconstruction model corresponding to the sparse view scene; The iteratively optimizing the facet parameters of the facets and the number of the facets in the facet set based on the view image and the depth information includes: Determine, based on the view image and the facet parameters, a rendering quality loss when rendering the facet set, and determine, based on the depth information and the facet parameters, a depth loss corresponding to the facet in the facet set; At least based on the rendering quality loss and the depth loss, the facet parameters of the facets in the facet set and the number of the facets are iteratively optimized; specifically comprising: obtaining the depth gradient matched by the depth information and the facet normal of the facet in the facet set, and obtaining the consistency loss based on the facet normal and the depth gradient; based on the rendering quality loss, the depth loss and the consistency loss, the facet parameters of the facets in the facet set and the number of the facets are iteratively optimized; wherein the rendering quality loss represents the loss corresponding to the luminosity and structure of the rendered image and the perspective image during rendering, the depth loss represents the loss corresponding to the depth prediction during rendering and the depth information at the current perspective, and the consistency loss represents the consistency loss between the facet normal and the depth gradient.

2. The sparse view scene reconstruction method according to claim 1, characterized in that: The method of acquiring perspective images collected at different perspectives in a sparse perspective scene, determining geometric relationships between different perspectives based on the perspective images, and determining depth information and a three-dimensional point cloud matched by pixels of the perspective images using the geometric relationships includes: Acquire viewing images captured at different viewing angles in a sparse viewing scene, and obtain an image pair consisting of the viewing images at every two viewing angles and a reference coordinate system matching the image pair; wherein the reference coordinate system corresponds to any viewing image in the image pair; Acquire image features of the viewing angle image in the image pair, and determine the geometric relationship between different viewing angles and a reference point cloud of the pixel in the reference coordinate system based on the image features and the reference coordinate system; Based on the image features and the geometric relationship, the depth information matched by the pixels of the perspective image is determined, and based on the geometric relationship, the reference point clouds in all the reference coordinate systems are merged into the reference coordinate system to obtain the three-dimensional point cloud of the pixels in the reference coordinate system; wherein the reference coordinate system corresponds to the sparse perspective scene.

3. The sparse view scene reconstruction method according to claim 2, characterized in that: The acquiring image features of the viewing angle image in the image pair, and determining the geometric relationship between different viewing angles and the reference point cloud of the pixel in the reference coordinate system based on the image features and the reference coordinate system, comprises: Acquire image features of the viewing angle image in the image pair, and determine camera parameters corresponding to a camera that acquires the viewing angle image based on the image features and the reference coordinate system; Based on the image features and the camera parameters, the geometric relationship between different viewing angles and the reference point cloud of the pixels in the reference coordinate system are determined.

4. The sparse view scene reconstruction method according to claim 1, characterized in that: The facet parameter includes the transparency of the facet, and the transparency corresponds to a transparency threshold; Before iteratively optimizing the bin parameters of the bins and the number of bins in the bin set based at least on the rendering quality loss and the depth loss, the method further includes: The facets in the facet set whose transparency is lower than the transparency threshold are cut off, and the facet set is updated.

5. The sparse view scene reconstruction method according to claim 1, characterized in that: After the facet parameters of the facets and the number of the facets in the facet set are iteratively optimized based on the view image and the depth information until a convergence condition is satisfied and a reconstruction model corresponding to the sparse view scene is obtained, the method further includes: In response to obtaining a specified viewing angle, obtaining an associated face element associated with the specified viewing angle in the reconstructed model, and performing surface depth reconstruction based on face element parameters of the associated face element to obtain a reference surface model; The reference surface model is semantically segmented to obtain a region to be corrected in the reference surface model, and the reference surface model is adjusted based on the region to be corrected to obtain a target surface model corresponding to the reconstructed model at the specified viewing angle; wherein the reflectivity of the region to be corrected is greater than a reflectivity threshold or the transparency is less than a transparency threshold.

6. The sparse view scene reconstruction method according to any one of claims 1 to 5, characterized in that: The method is implemented based on a 3D reconstruction network model, wherein the 3D reconstruction network model includes a positioning network and a rendering network, wherein the positioning network is used to determine the depth information and the 3D point cloud matched by the pixels of the view image, and the rendering network is used to obtain the face element set and iteratively optimize the face element parameters of the face elements in the face element set and the number of the face elements; The three-dimensional reconstruction network model is trained using training images collected in sparse viewing scenes and their corresponding training labels, wherein the training labels include camera parameters matched by a camera corresponding to the training image and depth information of the training image.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images