Three-dimensional reconstruction method and device, equipment and storage medium

Abstract

The invention provides a three-dimensional reconstruction method and device, equipment and a storage medium, and belongs to the field of image processing, and the method comprises the steps: obtaining point cloud data of a plurality of visual angles of a target scene of a to-be-constructed model; based on the point cloud data and a preset spherical Gaussian function, initializing parameters of the three-dimensional Gaussian primitive to obtain an initial three-dimensional model; selecting one target image from the plurality of target images as a supervision image; projecting a three-dimensional Gaussian primitive in the initial three-dimensional model to a two-dimensional view space according to a camera parameter matched with a visual angle corresponding to the supervision image to obtain a rendered image; and determining whether the three-dimensional model is accurate or not according to the rendered image and the supervised image, and if the three-dimensional model is not accurate, optimizing parameters of three-dimensional Gaussian primitives in the initial three-dimensional model until an accurate target three-dimensional model is obtained. The three-dimensional Gaussian primitive is constructed based on the preset spherical Gaussian function, so that illumination of the constructed three-dimensional model is closer to a real environment, and a real and accurate three-dimensional model is obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] AR technology superimposes virtual 3D objects, videos, texts, pictures and other computer-generated objects onto real scenes in real time, and realizes natural human-computer interaction through virtual-real fusion. It has broad application prospects in medical and health care, military simulation, industrial maintenance, education and entertainment, and other industries. The high-realistic rendering of virtual-real fusion images can effectively enhance the user's immersive experience. At present, most virtual-real fusion images are represented by 3D Gaussian primitives using the 3D Gaussian Splatting method to represent 3D scenes, and the image representation of the 3D scene is rendered using the spatial position, spherical harmonic function coefficients, opacity and other information stored in the 3D Gaussian primitives.

[0003] However, the current 3D Gaussian Splatting method uses spherical harmonics to simulate illumination, which has certain limitations. For example, the basis functions of the spherical harmonics are fixed, resulting in low degrees of freedom; the truncation of the spherical harmonics may introduce numerical errors when truncating the series; and the harmonics cannot effectively express the original rapidly changing signals during modeling, resulting in a ringing effect, which in turn leads to poor results in the reconstructed three-dimensional model.

[0004] Therefore, how to construct a more realistic and accurate three-dimensional model is an urgent problem to be solved. Summary of the invention

[0005] The main purpose of this application is to provide a three-dimensional reconstruction method, device, equipment and storage medium, aiming to improve the authenticity and accuracy of three-dimensional model construction.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method, the three-dimensional reconstruction method comprising the following steps:

[0007] Acquire multiple target images corresponding to multiple perspectives of a target scene to be modeled, and acquire point cloud data of the target scene according to the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images;

[0008] Initializing the parameters of the three-dimensional Gaussian primitives based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, wherein the parameters include lobe energy parameters of the three-dimensional Gaussian primitives, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian primitives;

[0009] Selecting one of the target images as a supervision image;

[0010] According to the camera parameters of the view angle matching corresponding to the supervision image, the three-dimensional Gaussian primitives in the initial three-dimensional model are projected into a two-dimensional view space to obtain a rendered image;

[0011] Determining a target loss value based on a difference between the rendered image and the supervised image;

[0012] When the target loss value is greater than a preset loss value, optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image;

[0013] When the target loss value is less than or equal to a preset loss value, the optimization of the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model is stopped, and the latest initial three-dimensional model is determined as the target three-dimensional model of the target scene.

[0014] In a second aspect, the present application further provides a three-dimensional reconstruction device, the three-dimensional reconstruction device comprising an acquisition module, a model initialization module, a generation module, a determination module and an update module, wherein:

[0015] The acquisition module is used to acquire a plurality of target images corresponding to a plurality of perspectives of a target scene to be modeled, and acquire point cloud data of the target scene according to the plurality of target images, wherein the plurality of perspectives are different, and one perspective corresponds to one or more target images;

[0016] The model initialization module is used to initialize the parameters of the three-dimensional Gaussian primitives based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, wherein the parameters include lobe energy parameters of the three-dimensional Gaussian primitives, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian primitives;

[0017] The acquisition module is further used to select one of the target images as a supervision image from the multiple target images;

[0018] The generating module is used to project the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters of the view angle matching corresponding to the supervision image, so as to obtain a rendered image;

[0019] The determination module is used to determine a target loss value according to a difference between the rendered image and the supervised image;

[0020] The updating module is used for optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value when the target loss value is greater than a preset loss value, so as to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image;

[0021] The determination module is also used to stop optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model when the target loss value is less than or equal to a preset loss value, and determine the latest initial three-dimensional model as the target three-dimensional model of the target scene.

[0022] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the three-dimensional reconstruction method as described above are implemented.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method as described above are implemented.

[0024] The present application provides a 3D reconstruction method, apparatus, device and storage medium. The present application obtains multiple target images corresponding to multiple viewpoints of a target scene to be modeled, and obtains point cloud data of the target scene according to the multiple target images, wherein the multiple viewpoints are different and one viewpoint corresponds to one or more target images; based on the point cloud data and a preset spherical Gaussian function, the parameters of a three-dimensional Gaussian basis element are initialized to obtain an initial three-dimensional model of the target scene, wherein the parameters include the lobe energy parameters of the three-dimensional Gaussian basis element, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian basis element; a target image is selected from the multiple target images as a supervision image; and according to the supervision image, a target image is selected as a supervision image. The camera parameters corresponding to the perspective match are used to project the three-dimensional Gaussian primitives in the initial three-dimensional model to the two-dimensional view space to obtain a rendered image; the target loss value is determined based on the difference between the rendered image and the supervised image; when the target loss value is greater than the preset loss value, the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model are optimized according to the target loss value to update the initial three-dimensional model, and the step of selecting a target image from multiple target images as the supervised image is returned to be executed; when the target loss value is less than or equal to the preset loss value, the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model are stopped from being optimized, and the latest initial three-dimensional model is determined as the target three-dimensional model of the target scene. In this application, the three-dimensional Gaussian primitives are constructed based on a preset spherical Gaussian function, so that the lighting of the constructed three-dimensional model is closer to the real environment, thereby making the three-dimensional model more realistic and accurate, greatly improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of a process flow of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0027] Figure 2 for Figure 1 A schematic flow chart of the sub-steps of the three-dimensional reconstruction method;

[0028] Figure 3 A schematic block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0029] Figure 4 for Figure 3 A schematic block diagram of submodules of a three-dimensional reconstruction device;

[0030] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0031] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0034] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0035] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] AR technology superimposes virtual 3D objects, videos, texts, pictures and other computer-generated objects onto real scenes in real time, and realizes natural human-computer interaction through virtual-real fusion. It has broad application prospects in medical and health care, military simulation, industrial maintenance, education and entertainment, and other industries. The high-realistic rendering of virtual-real fusion images can effectively enhance the user's immersive experience. At present, most virtual-real fusion images are represented by 3D Gaussian primitives using the 3D Gaussian Splatting method to represent 3D scenes, and the image representation of the 3D scene is rendered using the spatial position, spherical harmonic function coefficients, opacity and other information stored in the 3D Gaussian primitives.

[0037] However, the current 3D Gaussian Splatting method uses spherical harmonics to simulate illumination, which has certain limitations. For example, the basis functions of the spherical harmonics are fixed, resulting in low degrees of freedom; the truncation of the spherical harmonics may introduce numerical errors when truncating the series; and the harmonics cannot effectively express the original rapidly changing signals during modeling, resulting in a ringing effect, which in turn leads to poor results in the reconstructed three-dimensional model.

[0038] To solve the above problems, the embodiments of the present application provide a three-dimensional reconstruction method, device, equipment and storage medium. Among them, the three-dimensional reconstruction method includes: obtaining multiple target images corresponding to multiple perspectives of the target scene to be modeled, and obtaining point cloud data of the target scene based on the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images; based on the point cloud data and a preset spherical Gaussian function, initializing the parameters of the three-dimensional Gaussian basis element to obtain the initial three-dimensional model of the target scene, the parameters include the lobe energy parameters of the three-dimensional Gaussian basis element, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian basis element; selecting one target image from the multiple target images as a supervision image; and matching the camera parameters according to the perspective corresponding to the supervision image. , projecting the three-dimensional Gaussian primitives in the initial three-dimensional model to a two-dimensional view space to obtain a rendered image; determining a target loss value according to a difference between the rendered image and the supervisory image; when the target loss value is greater than a preset loss value, optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, and returning to execute the step of selecting one of the target images from the multiple target images as a supervisory image; when the target loss value is less than or equal to the preset loss value, stopping optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model, and determining the latest initial three-dimensional model as the target three-dimensional model of the target scene.

[0039] Among them, the three-dimensional reconstruction method can be applied to computer devices, which can be electronic devices such as mobile phones, tablet computers, laptops, desktop computers, personal digital assistants and wearable devices.

[0040] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0041] Please refer to Figure 1 , Figure 1 A schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application.

[0042] like Figure 1 As shown, the three-dimensional reconstruction method includes steps S101 to S107.

[0043] Step S101, obtaining multiple target images corresponding to multiple perspectives of a target scene to be modeled, and obtaining point cloud data of the target scene based on the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images.

[0044] The target scene includes but is not limited to a table, a computer, a room, a studio, a playground and all objects on the playground, etc. The multi-view includes any view of the target scene, for example, a face is selected as the front face, and the target view includes the front view, the back view, the side view, the top view and the top view.

[0045] It should be noted that the multiple target images include multiple different perspectives, and one perspective corresponds to one or more target images. The perspective and the number of target images at each perspective can be set according to the perspective situation. The embodiment of the present invention does not specifically limit this. For example, the perspective includes a front perspective, a side perspective and a top-down perspective. The front perspective includes 4 target images, the side perspective includes 10 target images and the top-down perspective includes 5 target images.

[0046] In some embodiments, the camera collects images of the target scene to be modeled at different viewing angles to obtain multiple target images. By collecting images of the target scene by the camera, multiple target images can be accurately obtained.

[0047] In some embodiments, at the same time point or near the time point, the camera collects images of the target scene to be modeled at different perspectives to obtain multiple target images. By collecting images of the target scene at different perspectives at the same time point or near the time point, the lighting information in each target image is the same or infinitely close to the same, thereby improving the lighting information in the 3D model of the 3D reconstruction to be more accurate and realistic.

[0048] It should be noted that the point cloud data constructed based on multiple target images can be selected according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the method of constructing the point cloud data can be the SfM (Structure-from-Motion)-COLMAP method.

[0049] In some embodiments, acquiring point cloud data of the target scene based on multiple target images is a step of performing feature extraction and matching, camera pose estimation, and sparse three-dimensional point cloud generation on the multiple target images to obtain point cloud data of the target scene.

[0050] Exemplarily, COLMAP uses algorithms such as SIFT to extract key points and their descriptors from each target image; by calculating the similarity between descriptors, feature points are matched to find image pairs with the required number of feature point matches; for each pair of matched feature points, the fundamental matrix (Fundamental Matrix) or essential matrix (Essential Matrix) is estimated by calculating the epipolar geometry, and the matching pairs are optimized and improved using the RANSAC algorithm to estimate the relative pose of the camera; Bundle Adjustment (BA) is performed on the two initially selected target images, and then new target images are added cyclically for new BA until no more target images can be added, thereby obtaining point cloud data.

[0051] Step S102: based on the point cloud data and a preset spherical Gaussian function, the parameters of the three-dimensional Gaussian basis element are initialized to obtain an initial three-dimensional model of the target scene, wherein the parameters include lobe energy parameters of the three-dimensional Gaussian basis element, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian basis element.

[0052] The three-dimensional Gaussian primitive includes a position parameter, a shape parameter, a color parameter and an opacity parameter, and the parameters include a lobe energy parameter of the three-dimensional Gaussian primitive, and the lobe energy parameter is used to describe the illumination information of the three-dimensional Gaussian primitive.

[0053] In some embodiments, step S102 includes sub-steps S1021 to S1023.

[0054] Sub-step S1021: performing mean processing on the point cloud data to obtain position parameters of the three-dimensional Gaussian basis element, and performing covariance processing on the point cloud data to obtain shape parameters of the three-dimensional Gaussian basis element.

[0055] In some embodiments, the point cloud data is processed by averaging to obtain the point cloud mean, and the point cloud mean is used as the position parameter of the three-dimensional Gaussian primitive. By calculating the mean of the point cloud data, the position parameter of the three-dimensional Gaussian primitive can be accurately obtained.

[0056] In some embodiments, a preset point cloud mean formula is obtained, and the preset point cloud mean formula is: Among them, μ is the mean of the point cloud, N i , Y i , Z i is the coordinate point of the point cloud, and N is the total number of point clouds. Based on the preset point cloud mean formula, the point cloud mean can be accurately obtained by calculating the point cloud data, and the point cloud mean is used as the position parameter of the three-dimensional Gaussian basis element.

[0057] In some embodiments, the position parameters of the three-dimensional Gaussian primitives are used as the center point of the Gaussian ellipsoid, and covariance calculation is performed based on the sparse point cloud to obtain a covariance matrix, and the covariance matrix is ​​used as the shape parameter of the three-dimensional Gaussian primitives. The covariance matrix of the Gaussian ellipsoid includes the rotation matrix R of the Gaussian ellipsoid and the scaling matrix S on each axis.

[0058] Sub-step S1022: Initialize the lobe energy parameters of the three-dimensional Gaussian basis element according to the preset spherical Gaussian function and the point cloud data to obtain color parameters, and determine the opacity parameters according to the point cloud data.

[0059] The preset spherical Gaussian function is G(v; μ, λ, a) = ae λ(μ·v-1) , where v is the color parameter, μ is the axis or direction of the lobe, λ is the sharpness of the lobe, and a is the amplitude or intensity of the lobe.

[0060] In some embodiments, the lobe energy parameters of the three-dimensional Gaussian primitives are initialized according to the preset spherical Gaussian function and the point cloud data to obtain the color parameters. The lobe energy parameters of the three-dimensional Gaussian primitives are initialized according to the preset spherical Gaussian function and the point cloud data to accurately obtain the color parameters.

[0061] In some embodiments, the opacity of the three-dimensional Gaussian primitives can be accurately obtained by processing the point cloud data based on Neural Radiance Fields (NeRF) technology.

[0062] Sub-step S1023: forming an initial three-dimensional model of the target scene according to the position parameters, shape parameters, color parameters and opacity parameters of the three-dimensional Gaussian primitives.

[0063] A three-dimensional model is constructed according to the position parameters, shape parameters, color parameters and opacity parameters of the three-dimensional Gaussian primitives to obtain an initial three-dimensional model of the target scene.

[0064] Step S103: Selecting a target image from the multiple target images as a supervision image.

[0065] The supervisory image is used to determine the error of the generated rendering image. The method of selecting the supervisory image can be set according to the actual situation, and the embodiment of the present invention does not specifically limit this, for example, randomly selecting.

[0066] By selecting a target image from multiple target images as a supervision image, the efficiency and accuracy of three-dimensional model construction can be effectively improved.

[0067] Step S104: project the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters of the view angle matching corresponding to the supervision image to obtain a rendered image.

[0068] The camera parameters include, but are not limited to, internal parameters such as the focal length and principal point coordinates of the camera, and external parameters such as the position and posture of the camera in the world coordinate system.

[0069] In some embodiments, camera parameters that match the viewing angle corresponding to the supervision image are obtained, and the three-dimensional Gaussian primitives in the initial three-dimensional model are projected into a two-dimensional view space according to the camera parameters to obtain two-dimensional Gaussian primitives; and a rendered image is obtained according to the two-dimensional Gaussian primitives.

[0070] In some embodiments, the method of projecting the three-dimensional Gaussian primitives in the initial three-dimensional model into the two-dimensional view space according to the camera parameters to obtain the two-dimensional Gaussian primitives may be: projecting the three-dimensional Gaussian primitives into the two-dimensional view space according to the camera parameters to obtain the two-dimensional Gaussian primitives. Projecting the three-dimensional Gaussian primitives according to the camera parameters can accurately obtain the two-dimensional Gaussian primitives.

[0071] In some embodiments, the two-dimensional Gaussian primitives are sorted, and the sorted two-dimensional Gaussian primitives are subjected to fast differentiable rasterization processing to obtain a two-dimensional rendered image. By performing fast differentiable rasterization processing on the two-dimensional Gaussian primitives to obtain a two-dimensional rendered image, the efficiency of three-dimensional reconstruction is greatly improved.

[0072] Step S105: Determine a target loss value according to the difference between the rendered image and the supervised image.

[0073] The target loss value is used to characterize the error of the generated three-dimensional Gaussian primitive. The larger the target loss value is, the larger the error of the reconstructed three-dimensional Gaussian primitive is.

[0074] In some embodiments, the absolute difference between the pixel value of each pixel in the rendered image and the pixel value of the corresponding pixel in the supervisory image is calculated to obtain multiple pixel absolute differences; the absolute differences of each pixel are summed to obtain a total pixel absolute difference value, and the total pixel absolute difference value is determined as the target loss value. By calculating the absolute difference between the pixel value of each pixel in the rendered image and the pixel value of each pixel in the supervisory image, the pixel difference between each pixel can be accurately obtained, and then the absolute differences of each pixel are summed to accurately obtain the loss value between the two images.

[0075] Step S106: When the target loss value is greater than a preset loss value, the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model are optimized according to the target loss value to update the initial three-dimensional model, and the step of selecting a target image from the multiple target images as a supervision image is returned to be executed.

[0076] The preset loss value can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset loss value can be set to 0.0002. The lower the preset loss value is set, the higher the accuracy of the three-dimensional reconstruction is when the target loss value is less than or equal to the preset loss value.

[0077] In some embodiments, when the target loss value is greater than the preset loss value, the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model are optimized according to the target loss value to update the initial three-dimensional model. The optimized parameters of the three-dimensional Gaussian primitives include but are not limited to the axial direction or direction of the lobe, the sharpness of the lobe and / or the amplitude or intensity of the lobe. By adjusting the axial direction or direction of the lobe, the sharpness of the lobe and / or the amplitude or intensity of the lobe of the three-dimensional Gaussian primitives, the

[0078] In some embodiments, when the target loss value is greater than a preset loss value, the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model are optimized according to the target loss value to update the initial three-dimensional model, and the following steps are continued: selecting a target image from multiple target images as a supervisory image; projecting the three-dimensional Gaussian primitives in the initial three-dimensional model to a two-dimensional view space according to the camera parameters matching the perspective corresponding to the supervisory image to obtain a rendered image; determining the target loss value according to the difference between the rendered image and the supervisory image; when the target loss value is greater than a preset loss value, optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, until the target loss value is less than or equal to the preset loss value to obtain a high-precision three-dimensional model.

[0079] Step S107: when the target loss value is less than or equal to a preset loss value, stop optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model, and determine the latest initial three-dimensional model as the target three-dimensional model of the target scene.

[0080] When the target loss value is less than or equal to the preset loss value, the 3D model is determined to be infinitely close to the target scene, the optimization of the parameters of the 3D Gaussian primitives in the initial 3D model is stopped, and the latest initial 3D model is determined as the target 3D model of the target scene. Infinitely close means that the 3D model is identical to the target scene within an acceptable error range.

[0081] In some embodiments, after generating the target 3D model of the target scene, a scene image of any viewing angle may be generated. The target 3D model can be used to output a scene image of any viewing angle.

[0082] In some embodiments, the parameters of the three-dimensional Gaussian primitives are obtained when the target loss value is less than or equal to the preset loss value, and the parameters of the three-dimensional Gaussian primitives are stored; when the same scene to be constructed is three-dimensionally reconstructed after the current moment, the scene to be constructed is three-dimensionally reconstructed according to the parameters of the three-dimensional Gaussian primitives to obtain a target three-dimensional model corresponding to the scene to be constructed. The parameters of the three-dimensional Gaussian primitives are stored and directly used for the three-dimensional reconstruction of the same scene to be constructed thereafter, which greatly improves the efficiency and accuracy of the three-dimensional reconstruction.

[0083] In some embodiments, the method for determining whether the scene to be constructed is the same as the target scene can be: determining the similarity between the scene to be constructed and the target scene, if the similarity between the scene to be constructed and the target scene is greater than or equal to the preset similarity, determining that the scene to be constructed is the same as the target scene; if the similarity between the scene to be constructed and the target scene is less than the preset similarity, determining that the scene to be constructed is not the same as the target scene. Among them, the preset similarity can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset similarity can be set to 0.999.

[0084] The three-dimensional reconstruction method provided in the above embodiment obtains multiple target images corresponding to multiple perspectives of the target scene to be modeled, and obtains point cloud data of the target scene according to the multiple target images, wherein the multiple perspectives are different and one perspective corresponds to one or more target images; based on the point cloud data and a preset spherical Gaussian function, the parameters of the three-dimensional Gaussian basis element are initialized to obtain an initial three-dimensional model of the target scene, the parameters including the lobe energy parameters of the three-dimensional Gaussian basis element, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian basis element; a target image is selected from the multiple target images as a supervision image; and the supervision image is matched according to the perspective corresponding to the supervision image. Camera parameters, project the three-dimensional Gaussian primitives in the initial three-dimensional model to the two-dimensional view space to obtain a rendered image; determine the target loss value based on the difference between the rendered image and the supervised image; when the target loss value is greater than the preset loss value, optimize the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, and return to execute the step of selecting a target image from multiple target images as the supervised image; when the target loss value is less than or equal to the preset loss value, stop optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model, and determine the latest initial three-dimensional model as the target three-dimensional model of the target scene. In this application, the three-dimensional Gaussian primitives are constructed based on a preset spherical Gaussian function, so that the lighting of the constructed three-dimensional model is closer to the real environment, thereby making the three-dimensional model more realistic and accurate, greatly improving the user's viewing experience.

[0085] See also Figure 3 , Figure 3 A schematic block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application.

[0086] like Figure 3 As shown, the three-dimensional reconstruction device 200 includes an acquisition module 210, a model initialization module 220, a generation module 230, a determination module 240 and an update module 250, wherein:

[0087] The acquisition module 210 is used to acquire multiple target images corresponding to multiple perspectives of the target scene to be modeled, and acquire point cloud data of the target scene according to the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images;

[0088] The model initialization module 220 is used to initialize the parameters of the three-dimensional Gaussian basis element based on the point cloud data and the preset spherical Gaussian function to obtain the initial three-dimensional model of the target scene, wherein the parameters include the lobe energy parameters of the three-dimensional Gaussian basis element, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian basis element;

[0089] The acquisition module 210 is further used to select one of the target images as a supervision image from the multiple target images;

[0090] The generating module 230 is used to project the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters of the view angle matching corresponding to the supervision image to obtain a rendered image;

[0091] The determination module 240 is used to determine a target loss value according to a difference between the rendered image and the supervision image;

[0092] The updating module 250 is used for optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value when the target loss value is greater than a preset loss value, so as to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image;

[0093] The determination module 240 is also used to stop optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model when the target loss value is less than or equal to a preset loss value, and determine the latest initial three-dimensional model as the target three-dimensional model of the target scene.

[0094] In some embodiments, Figure 4As shown, the model initialization module 220 includes a generation submodule 221, a determination submodule 222 and a formation module 223, wherein:

[0095] The generating submodule 221 is used for performing mean processing on the point cloud data to obtain position parameters of the three-dimensional Gaussian primitives, and performing covariance processing on the point cloud data to obtain shape parameters of the three-dimensional Gaussian primitives;

[0096] The generating submodule 221 is further used to initialize the lobe energy parameters of the three-dimensional Gaussian basis element according to the preset spherical Gaussian function and the point cloud data to obtain color parameters;

[0097] The determination submodule 222 is used to determine an opacity parameter according to the point cloud data;

[0098] The forming module 223 is used to form an initial three-dimensional model of the target scene according to the position parameters, shape parameters, color parameters and opacity parameters of the three-dimensional Gaussian primitives.

[0099] In some embodiments, the generating module 230 is further configured to:

[0100] Projecting the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters to obtain two-dimensional Gaussian primitives;

[0101] A rendered image is obtained according to the two-dimensional Gaussian primitives.

[0102] In some embodiments, the generating module 230 is further configured to:

[0103] The two-dimensional Gaussian primitives are sorted, and fast differentiable rasterization processing is performed on the sorted two-dimensional Gaussian primitives to obtain a two-dimensional rendering image.

[0104] In some embodiments, the determining module 240 is further configured to:

[0105] Calculating the absolute difference between the pixel value of each pixel in the rendered image and the pixel value of the corresponding pixel in the supervision image to obtain a plurality of pixel absolute differences;

[0106] The absolute difference values ​​of each pixel are summed up to obtain a total absolute difference value of the pixels, and the total absolute difference value of the pixels is determined as the target loss value.

[0107] In some embodiments, the 3D reconstruction device 200 is further used for:

[0108] Acquire the parameters of the three-dimensional Gaussian primitive when the target loss value is less than or equal to the preset loss value, and store the parameters of the three-dimensional Gaussian primitive;

[0109] When a scene to be constructed that is the same as the target scene is three-dimensionally reconstructed after the current moment, the scene to be constructed is three-dimensionally reconstructed according to the parameters of the three-dimensional Gaussian primitives to obtain a target three-dimensional model corresponding to the scene to be constructed.

[0110] In some embodiments, the 3D reconstruction device 200 is further used for:

[0111] Acquire a scene to be constructed, and determine the similarity between the scene to be constructed and the target scene;

[0112] When the similarity between the scene to be constructed and the target scene is greater than or equal to a preset similarity, it is determined that the scene to be constructed is the same as the target scene.

[0113] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned 3D reconstruction device can refer to the corresponding process in the aforementioned 3D reconstruction method embodiment, and will not be repeated here.

[0114] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0115] like Figure 5 As shown, the computer device 300 includes a processor 302 and a memory 303 connected via a system bus 301 , wherein the memory 303 may include a storage medium and an internal memory.

[0116] The storage medium may store a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor may execute any three-dimensional reconstruction method.

[0117] The processor 302 is used to provide computing and control capabilities and support the operation of the entire computer device.

[0118] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any three-dimensional reconstruction method.

[0119] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] It should be understood that the processor 302 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] In one embodiment, the processor 302 is used to run a computer program stored in the memory to implement the following steps:

[0122] Acquire multiple target images corresponding to multiple perspectives of a target scene to be modeled, and acquire point cloud data of the target scene according to the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images;

[0123] Initializing the parameters of the three-dimensional Gaussian primitives based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, wherein the parameters include lobe energy parameters of the three-dimensional Gaussian primitives, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian primitives;

[0124] Selecting one of the target images as a supervision image;

[0125] According to the camera parameters of the view angle matching corresponding to the supervision image, the three-dimensional Gaussian primitives in the initial three-dimensional model are projected into a two-dimensional view space to obtain a rendered image;

[0126] Determining a target loss value based on a difference between the rendered image and the supervised image;

[0127] When the target loss value is greater than a preset loss value, optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image;

[0128] When the target loss value is less than or equal to a preset loss value, the optimization of the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model is stopped, and the latest initial three-dimensional model is determined as the target three-dimensional model of the target scene.

[0129] In one embodiment, when the processor 302 initializes the parameters of the three-dimensional Gaussian basis element based on the point cloud data and the preset spherical Gaussian function to obtain the initial three-dimensional model of the target scene, it is used to implement:

[0130] Performing mean processing on the point cloud data to obtain position parameters of the three-dimensional Gaussian basis element, and performing covariance processing on the point cloud data to obtain shape parameters of the three-dimensional Gaussian basis element;

[0131] Initializing the lobe energy parameters of the three-dimensional Gaussian basis element according to the preset spherical Gaussian function and the point cloud data to obtain color parameters, and determining opacity parameters according to the point cloud data;

[0132] An initial three-dimensional model of the target scene is formed according to the position parameters, shape parameters, color parameters and opacity parameters of the three-dimensional Gaussian primitives.

[0133] In one embodiment, when the processor 302 implements the camera parameters matched according to the perspective corresponding to the supervision image, projects the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space to obtain a rendered image, it is used to implement:

[0134] Projecting the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters to obtain two-dimensional Gaussian primitives;

[0135] A rendered image is obtained according to the two-dimensional Gaussian primitives.

[0136] In one embodiment, when the processor 302 obtains the rendered image according to the two-dimensional Gaussian primitive, it is configured to implement:

[0137] The two-dimensional Gaussian primitives are sorted, and fast differentiable rasterization processing is performed on the sorted two-dimensional Gaussian primitives to obtain a two-dimensional rendering image.

[0138] In one embodiment, when the processor 302 determines the target loss value according to the difference between the rendered image and the supervised image, it is configured to implement:

[0139] Calculating the absolute difference between the pixel value of each pixel in the rendered image and the pixel value of the corresponding pixel in the supervision image to obtain a plurality of pixel absolute differences;

[0140] The absolute difference values ​​of each pixel are summed up to obtain a total absolute difference value of the pixels, and the total absolute difference value of the pixels is determined as the target loss value.

[0141] In one embodiment, the processor 302 is further configured to implement:

[0142] Acquire the parameters of the three-dimensional Gaussian primitive when the target loss value is less than or equal to the preset loss value, and store the parameters of the three-dimensional Gaussian primitive;

[0143] When a scene to be constructed that is the same as the target scene is three-dimensionally reconstructed after the current moment, the scene to be constructed is three-dimensionally reconstructed according to the parameters of the three-dimensional Gaussian primitives to obtain a target three-dimensional model corresponding to the scene to be constructed.

[0144] In one embodiment, the processor 302 is further configured to implement:

[0145] Acquire a scene to be constructed, and determine the similarity between the scene to be constructed and the target scene;

[0146] When the similarity between the scene to be constructed and the target scene is greater than or equal to a preset similarity, it is determined that the scene to be constructed is the same as the target scene.

[0147] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned three-dimensional reconstruction method embodiment, and will not be repeated here.

[0148] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the three-dimensional reconstruction method of the present application.

[0149] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0150] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0151] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0152] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0153] It should also be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0154] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire multiple target images corresponding to multiple perspectives of a target scene to be modeled, and acquire point cloud data of the target scene according to the multiple target images, wherein the multiple perspectives are different, and one perspective corresponds to one or more target images; Initializing the parameters of the three-dimensional Gaussian basis element based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, wherein the parameters include a lobe energy parameter of the three-dimensional Gaussian basis element, and the lobe energy parameter is used to describe the illumination information of the three-dimensional Gaussian basis element; Selecting one of the target images as a supervision image; According to the camera parameters of the view angle matching corresponding to the supervision image, the three-dimensional Gaussian primitives in the initial three-dimensional model are projected into a two-dimensional view space to obtain a rendered image; Determining a target loss value based on a difference between the rendered image and the supervised image; When the target loss value is greater than a preset loss value, optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image; When the target loss value is less than or equal to a preset loss value, the optimization of the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model is stopped, and the latest initial three-dimensional model is determined as the target three-dimensional model of the target scene.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional Gaussian primitives include position parameters, shape parameters, color parameters, and opacity parameters; the parameters of the three-dimensional Gaussian primitives are initialized based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, including: Performing mean processing on the point cloud data to obtain position parameters of the three-dimensional Gaussian basis element, and performing covariance processing on the point cloud data to obtain shape parameters of the three-dimensional Gaussian basis element; Initializing the lobe energy parameters of the three-dimensional Gaussian basis element according to the preset spherical Gaussian function and the point cloud data to obtain color parameters, and determining opacity parameters according to the point cloud data; An initial three-dimensional model of the target scene is formed according to the position parameters, shape parameters, color parameters and opacity parameters of the three-dimensional Gaussian primitives.

3. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of projecting the three-dimensional Gaussian primitives in the initial three-dimensional model to a two-dimensional view space according to the camera parameters of the view angle matched by the supervision image to obtain a rendered image includes: Projecting the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters to obtain two-dimensional Gaussian primitives; A rendered image is obtained according to the two-dimensional Gaussian primitives.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The step of obtaining a rendered image according to the two-dimensional Gaussian primitives comprises: The two-dimensional Gaussian primitives are sorted, and fast differentiable rasterization processing is performed on the sorted two-dimensional Gaussian primitives to obtain a two-dimensional rendering image.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining a target loss value according to a difference between the rendered image and the supervised image comprises: Calculating the absolute difference between the pixel value of each pixel in the rendered image and the pixel value of the corresponding pixel in the supervision image to obtain a plurality of pixel absolute differences; The absolute difference values ​​of each pixel are summed up to obtain a total absolute difference value of the pixels, and the total absolute difference value of the pixels is determined as the target loss value.

6. The three-dimensional reconstruction method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire the parameters of the three-dimensional Gaussian primitive when the target loss value is less than or equal to the preset loss value, and store the parameters of the three-dimensional Gaussian primitive; When a scene to be constructed that is the same as the target scene is three-dimensionally reconstructed after the current moment, the scene to be constructed is three-dimensionally reconstructed according to the parameters of the three-dimensional Gaussian primitives to obtain a target three-dimensional model corresponding to the scene to be constructed.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: The method further comprises: Acquire a scene to be constructed, and determine the similarity between the scene to be constructed and the target scene; When the similarity between the scene to be constructed and the target scene is greater than or equal to a preset similarity, it is determined that the scene to be constructed is the same as the target scene.

8. A three-dimensional reconstruction device, characterized in that: The three-dimensional reconstruction device includes an acquisition module, a model initialization module, a generation module, a determination module and an update module, wherein: The acquisition module is used to acquire a plurality of target images corresponding to a plurality of perspectives of a target scene to be modeled, and acquire point cloud data of the target scene according to the plurality of target images, wherein the plurality of perspectives are different, and one perspective corresponds to one or more target images; The model initialization module is used to initialize the parameters of the three-dimensional Gaussian primitives based on the point cloud data and a preset spherical Gaussian function to obtain an initial three-dimensional model of the target scene, wherein the parameters include lobe energy parameters of the three-dimensional Gaussian primitives, and the lobe energy parameters are used to describe the illumination information of the three-dimensional Gaussian primitives; The acquisition module is further used to select one of the target images as a supervision image from the multiple target images; The generating module is used to project the three-dimensional Gaussian primitives in the initial three-dimensional model into a two-dimensional view space according to the camera parameters of the view angle matching corresponding to the supervision image, so as to obtain a rendered image; The determination module is used to determine a target loss value according to a difference between the rendered image and the supervised image; The updating module is used for optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model according to the target loss value when the target loss value is greater than a preset loss value, so as to update the initial three-dimensional model, and returning to execute the step of selecting one target image from the multiple target images as a supervision image; The determination module is also used to stop optimizing the parameters of the three-dimensional Gaussian primitives in the initial three-dimensional model when the target loss value is less than or equal to a preset loss value, and determine the latest initial three-dimensional model as the target three-dimensional model of the target scene.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 7 are implemented.

Images