Image reconstruction method and device, equipment, medium and product

Abstract

The invention relates to an image reconstruction method and device, equipment, a medium and a product, and the method comprises the steps: generating a Gaussian ellipsoid set based on a point cloud corresponding to a to-be-reconstructed image, the Gaussian ellipsoid set comprises a plurality of Gaussian ellipsoids, and each Gaussian ellipsoid has a corresponding relation with one point in the point cloud; according to the geometric parameters of the Gaussian ellipsoid, determining a weight coefficient corresponding to the Gaussian ellipsoid; determining a diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters and the observation direction vector of the Gaussian ellipsoid; determining a reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters and the reflection direction vector of the Gaussian ellipsoid; and performing fusion processing on the weight coefficient, the diffuse reflection color component and the reflection highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set to generate a target reconstruction image. According to the invention, image reconstruction can be carried out based on fusion of diffuse reflection and highlight reflection colors in an object or a scene, and the quality and reality sense of a target reconstruction image are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image reconstruction method, device, equipment, medium and product. Background Art

[0002] High-quality 3D reconstruction based on multiple views and supporting real-time rendering is a hot issue in current 3D computer vision. Gaussian Splatting is a rendering technique widely used in computer graphics, which allows real-time rendering of realistic scenes learned from small image samples, and can restore the full-view image of the real scene or object through limited images.

[0003] However, it is difficult for the Gaussian splash algorithm to accurately model the highlight part of the object with reflective components, and the highlight part is very sensitive to changes in viewing angle. Therefore, the loss of highlight details of the reconstructed image under the new viewing angle leads to a significant decrease in the realism of the image. Summary of the invention

[0004] One of the purposes of the present application is to provide an image reconstruction method to solve the problem in the related art that when reconstructing a new view of a three-dimensional scene, there is low quality and high reconstruction difficulty when reconstructing an image with reflective properties; the second purpose is to provide an image reconstruction device; the third purpose is to provide an electronic device; the fourth purpose is to provide a computer-readable storage medium; and the fifth purpose is to provide a computer program product.

[0005] In order to achieve the above purpose, the present application provides an image reconstruction method, and the technical solution adopted is as follows:

[0006] Based on the point cloud corresponding to the image to be reconstructed, a Gaussian ellipsoid set is generated, wherein the Gaussian ellipsoid set includes a plurality of Gaussian ellipsoids, and each Gaussian ellipsoid has a corresponding relationship with a point in the point cloud;

[0007] According to the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined;

[0008] According to the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse reflection color component corresponding to the Gaussian ellipsoid is determined;

[0009] According to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined;

[0010] The weight coefficient, diffuse color component and reflective highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set are fused separately to generate a target reconstructed image.

[0011] According to the above technical means, firstly, a Gaussian ellipsoid set is generated based on the point cloud corresponding to the image to be reconstructed. In this way, multiple Gaussian ellipsoids are generated based on the 3D Gaussian splash algorithm, which not only improves the interpretability of the three-dimensional scene represented by it, but also can render a detailed and realistic scene with higher efficiency; secondly, based on the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined, based on the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse reflection color component corresponding to the Gaussian ellipsoid is determined, and based on the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined. In this way, the diffuse reflection and highlight reflection colors in the object or scene can be effectively decomposed, so that the image reconstruction method provided by the present application can be reconstructed with high quality even in objects or scenes with reflective properties, and the reconstruction performance of the Gaussian splash algorithm on image data sets with metallic luster, light reflection and shadow is enhanced, and real-time rendering is supported. Finally, the diffuse color component and the reflective highlight color component are fused based on the weight coefficient to generate a target reconstructed image, so that the generated target reconstructed image is fused based on the corresponding weight coefficient, and the color expression of the 3D Gaussian splash algorithm in the reflected view appearance is supplemented by the reflective highlight color component, and the ability to represent diffuse reflection and capture details is enhanced by the diffuse color component, thereby improving the quality and realism of the target reconstructed image.

[0012] Furthermore, according to the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined, including: concatenating the shortest axis direction vector and the eigenvector of the spherical harmonic function corresponding to the Gaussian ellipsoid to obtain the characteristic matrix corresponding to the Gaussian ellipsoid; inputting the characteristic matrix corresponding to the Gaussian ellipsoid into the target weight perception model to obtain the weight coefficient corresponding to the Gaussian ellipsoid.

[0013] According to the above technical means, the shortest axis direction vector corresponding to the Gaussian ellipsoid and the eigenvector of the spherical harmonic function are spliced ​​to obtain the characteristic matrix corresponding to the Gaussian ellipsoid, and the characteristic matrix is ​​further input into the target weight perception model to obtain the weight coefficient corresponding to the Gaussian ellipsoid. In this way, the weights corresponding to the diffuse color component and the reflective highlight color component that make up the Gaussian ellipsoid can be determined, thereby improving the color expression effect of the target reconstructed image.

[0014] Furthermore, the weight coefficients, diffuse color components and reflection highlight color components corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set are fused separately to generate a target reconstructed image, including: determining the diffuse weight coefficient corresponding to the diffuse color component and the reflection highlight weight coefficient corresponding to the reflection highlight color component based on the weight coefficient corresponding to the Gaussian ellipsoid; performing weighted calculation on the diffuse color component corresponding to the Gaussian ellipsoid and the reflection highlight color component corresponding to the Gaussian ellipsoid based on the diffuse weight coefficient and the reflection highlight weight coefficient to determine the mixed color component corresponding to the Gaussian ellipsoid; performing differentiable rasterization processing on the mixed color components corresponding to multiple Gaussian ellipsoids in the Gaussian ellipsoid set to generate a target reconstructed image.

[0015] According to the above technical means, based on the diffuse reflection weight coefficient and the reflection highlight weight coefficient corresponding to the Gaussian ellipsoid, the diffuse reflection color component corresponding to the Gaussian ellipsoid and the reflection highlight color component corresponding to the Gaussian ellipsoid are weightedly calculated to determine the mixed color component corresponding to the Gaussian ellipsoid, and further based on the mixed color components corresponding to all Gaussian ellipsoids in the Gaussian ellipsoid set, the target reconstructed image is generated. In this way, the three-dimensional modeling can be converted to a two-dimensional plane more efficiently, improving the efficiency of generating the target reconstructed image.

[0016] Furthermore, differentiable rasterization is performed on the mixed color components corresponding to each of the multiple Gaussian ellipsoids in the Gaussian ellipsoid set to generate a target reconstructed image, including: projecting the Gaussian ellipsoid set into an initial reconstructed image based on a transformation matrix; rendering the initial reconstructed image based on the mixed color components corresponding to each of the multiple Gaussian ellipsoids to generate a target reconstructed image.

[0017] According to the above technical means, the Gaussian ellipsoid set is projected into an initial reconstructed image through a transformation matrix, and the initial reconstructed image is further rendered based on the mixed color components corresponding to each of the multiple Gaussian ellipsoids to generate a target reconstructed image. In this way, based on multiple Gaussian ellipsoids with mixed color components, the target reconstructed image is projected and generated, which can achieve high-quality reconstruction in scenes with reflective properties, thereby improving the effect of the target reconstructed image.

[0018] Furthermore, based on the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse reflection color component corresponding to the Gaussian ellipsoid is determined, including: concatenating the feature matrix and the observation direction vector to obtain the observation direction matrix; inputting the observation direction matrix into the target observation direction processing model to obtain the diffuse reflection color component corresponding to the Gaussian ellipsoid.

[0019] According to the above technical means, the feature matrix and the viewing direction vector are spliced ​​to generate the viewing direction matrix, which is further input into the target viewing direction processing model to output the diffuse color component corresponding to the Gaussian ellipsoid. In this way, there is a strong correlation between the diffuse color component and the viewing direction, which can represent the diffuse and blurred reflection in the scene, and improve the method's ability to represent diffuse reflection and capture details.

[0020] Furthermore, according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined, including: concatenating the characteristic matrix and the reflection direction vector to obtain the reflection direction matrix; inputting the reflection direction matrix into the target reflection direction processing model to obtain the reflection highlight color component corresponding to the Gaussian ellipsoid.

[0021] According to the above technical means, the reflection direction matrix is ​​generated by splicing based on the feature matrix and the reflection direction vector, and the reflection direction matrix is ​​further input into the target reflection direction processing model to output the reflection highlight color component corresponding to the Gaussian ellipsoid. In this way, there is a strong correlation between the reflection highlight color component and the reflection direction, which can supplement the color expression of the 3D Gaussian in the reflection view appearance, improve the reconstruction realism of the image to be reconstructed with the reflection component, and improve the reconstruction quality of the image.

[0022] Furthermore, the method also includes: determining the observation direction of the Gaussian ellipsoid based on the camera parameters of the image to be reconstructed and the coordinates of the center of the Gaussian ellipsoid; encoding the observation direction to determine the observation direction vector of the Gaussian ellipsoid; determining the reflection direction vector corresponding to the Gaussian ellipsoid based on the shortest axis direction vector corresponding to the Gaussian ellipsoid and the observation direction vector.

[0023] According to the above technical means, the observation direction of the Gaussian ellipsoid is determined by the camera parameters of the image to be reconstructed and the coordinates of the center of any Gaussian ellipsoid, and the observation direction is further encoded to determine the observation direction vector of the Gaussian ellipsoid, and then the reflection direction vector corresponding to the Gaussian ellipsoid is calculated based on the shortest axis direction vector and the observation direction vector corresponding to the Gaussian ellipsoid. In this way, the observation direction can be accurately determined, thereby improving the quality of the generated target reconstructed image, and the calculation rate of the reflection direction vector can be increased, thereby improving the performance of image reconstruction.

[0024] Furthermore, the method also includes: constructing an initial weight perception model, an initial observation direction processing model and an initial reflection direction processing model; constructing a training sample set, the training sample set including multiple training samples, each training sample including a synthetic Gaussian ellipsoid set and a corresponding real three-dimensional image; based on the target loss function, using the training sample set to jointly train the initial weight perception model, the initial observation direction processing model and the initial reflection direction processing model to obtain the target weight perception model, the target observation direction processing model and the target reflection direction processing model.

[0025] According to the above technical means, based on the target loss function, the constructed initial weight perception model, initial observation direction processing model and initial reflection direction processing model are jointly trained using the training sample set to obtain the target weight perception model, target observation direction processing model and target reflection direction processing model. In this way, multiple models can share information and optimize together, which improves the generalization ability of the model and the efficiency of model training.

[0026] Furthermore, the method also includes: constructing an initial loss function based on a Gaussian loss function, a mixed photometric loss function, a smoothing loss function and preset coefficients; adjusting the preset coefficients based on preset evaluation parameters to determine a target loss function, wherein the preset evaluation parameters are used to evaluate the reconstruction effect of the target reconstructed image.

[0027] According to the above technical means, the preset coefficients in the initial loss function are adjusted based on the preset evaluation parameters to determine the target loss function. In this way, multiple models generated by training based on the target loss function can have good performance in the reconstruction of different images, thereby improving the quality of the target reconstructed image.

[0028] An image reconstruction device, the image reconstruction device comprising:

[0029] A first generating module is used to generate a Gaussian ellipsoid set based on a point cloud corresponding to the image to be reconstructed, wherein the Gaussian ellipsoid set includes a plurality of Gaussian ellipsoids, and each Gaussian ellipsoid has a corresponding relationship with a point in the point cloud;

[0030] A first determination module is used to determine a weight coefficient corresponding to the Gaussian ellipsoid according to geometric parameters of the Gaussian ellipsoid;

[0031] The second determination module is used to determine the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector;

[0032] A third determination module is used to determine the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector;

[0033] The second generation module is used to fuse the weight coefficient, diffuse color component and reflection highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set to generate a target reconstructed image.

[0034] An electronic device comprises a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, any of the above methods is implemented.

[0035] A computer-readable storage medium stores a computer program, which implements any of the above methods when executed by a processor.

[0036] A computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a processor, any of the above methods is implemented.

[0037] Beneficial effects of this application:

[0038] (1) Based on the point cloud corresponding to the image to be reconstructed, a set of Gaussian ellipsoids is generated. In this way, multiple Gaussian ellipsoids are generated based on the 3D Gaussian splash algorithm, which not only improves the interpretability of the three-dimensional scene represented by it, but also can render a detailed and realistic scene with higher efficiency.

[0039] (2) Based on the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined, based on the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse color component corresponding to the Gaussian ellipsoid is determined, and based on the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined. In this way, the method can effectively decompose the diffuse reflection and highlight reflection colors in the object or scene, so that the image reconstruction method provided by the present application can reconstruct with high quality even in objects or scenes with reflective properties, enhance the reconstruction performance of the 3D Gaussian splash algorithm on image data sets with metallic luster, light reflection and shadow, and support real-time rendering.

[0040] (3) The diffuse color component and the reflective highlight color component are fused based on the weight coefficient to generate a target reconstructed image, so that the generated target reconstructed image is fused based on the corresponding weight coefficient. The reflective highlight color component supplements the color expression of the 3D Gaussian splash algorithm in the reflected view appearance, and the diffuse color component enhances the ability to represent diffuse reflection and capture details, thereby improving the quality and realism of the target reconstructed image.

[0041] (4) Based on the target loss function, the constructed initial weight perception model, initial observation direction processing model and initial reflection direction processing model are jointly trained using the training sample set to obtain the target weight perception model, target observation direction processing model and target reflection direction processing model. In this way, multiple models can share information and optimize together, which improves the generalization ability of the model and the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of an implementation process of an image reconstruction method provided in an embodiment of the present application Figure 1 ;

[0043] Figure 2 A schematic diagram of a target reconstructed image and different color components provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a framework of an image reconstruction method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of image reconstruction effects using different algorithms provided in the embodiments of the present application Figure 1 ;

[0046] Figure 5 A schematic diagram of image reconstruction effects using different algorithms provided in the embodiments of the present application Figure 2 ;

[0047] Figure 6 A schematic diagram of image reconstruction effects using different algorithms provided in the embodiments of the present application Figure 3 ;

[0048] Figure 7 A schematic diagram of image reconstruction effects using different algorithms provided in the embodiments of the present application Figure 4 ;

[0049] Figure 8 A schematic diagram of image reconstruction effects using different algorithms provided in the embodiments of the present application Figure 5 ;

[0050] Fig. 9 A schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present application;

[0051] Fig.10 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0054] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] High-quality 3D reconstruction based on multiple views and supporting real-time rendering is a hot topic in current 3D computer vision. Neural Radiance Fields (NeRF) combines the new perspective synthesis task with neural networks, but the training and reasoning process of the NeRF model is relatively long and cannot achieve real-time rendering. There are existing studies to reduce the training cost of the NeRF model and improve the reconstruction quality, but it is still difficult to ensure fast training and rendering while improving the reconstruction accuracy.

[0058] Recently, the newly proposed 3D Gaussian splash algorithm (also called 3D Gaussian splash algorithm, or 3D Gaussian splash algorithm, or 3D Gaussian) has surpassed the level of NeRF model in terms of training speed and rendering quality through explicit expression, and has become an alternative to the NeRF model. The 3D Gaussian splash algorithm estimates point cloud information from a set of images through the Structure from Motion (SFM) algorithm, uses the point cloud information to initialize the Gaussian sphere parameters, and uses the Gaussian sphere to express objects or scenes. It retains the idea of ​​volume rendering, projects the three-dimensional Gaussian sphere onto the two-dimensional image plane, and uses rasterization technology for rendering. However, the 3D Gaussian splash algorithm has difficulty modeling highlights and reflective components in objects or scenes, such as metallic luster and projections on car windows.

[0059] In the related technology, a highlight recovery model is constructed based on the Gaussian splash algorithm, and a set of highlight spherical harmonic coefficients are generated. Then, a highlight image is rendered based on the trained highlight spherical harmonic coefficients, and this highlight image is mixed with the original image rendered by the 3D Gaussian splash algorithm to obtain the final rendered image. The loss of the rendered image and the real image is calculated to optimize the highlight recovery model. However, this method decouples the highlight and the base color at the image level, trains a set of highlight spherical harmonic coefficients separately, and uses the coefficients to render the highlight image. The decomposed highlight image may not be the accurate highlight part. Even if a higher quality is obtained under the reconstructed perspective, it cannot ensure that the highlight of the object can be correctly synthesized under the new perspective.

[0060] Therefore, the existing reconstruction methods based on Gaussian splashing ignore the physical modeling of the reflected view, and there is still room for improvement when it comes to the reconstruction of objects and scenes with reflective properties. Therefore, how to ensure that the training cost and time of 3D reconstruction are controllable while supporting real-time rendering and improving the reconstruction quality of objects and scenes with reflective properties is a technical problem that needs to be solved urgently.

[0061] Based on this, the embodiment of the present application provides an image reconstruction method, device, equipment, medium and product. First, based on the point cloud corresponding to the image to be reconstructed, a Gaussian ellipsoid set is generated. In this way, multiple Gaussian ellipsoids are generated based on the 3D Gaussian splash algorithm, which not only improves the interpretability of the three-dimensional scene represented by it, but also can render a detailed and realistic scene with higher efficiency; secondly, based on the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined, based on the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse reflection color component corresponding to the Gaussian ellipsoid is determined, and based on the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined. In this way, the method can effectively decompose the diffuse reflection and highlight reflection colors in the object or scene, so that the image reconstruction method provided by the present application can reconstruct with high quality even in objects or scenes with reflective properties, enhance the reconstruction performance of the 3D Gaussian splash algorithm on image data sets with metallic luster, light reflection and shadow, and support real-time rendering. Finally, the diffuse color component and the reflective highlight color component are fused based on the weight coefficient to generate a target reconstructed image, so that the generated target reconstructed image is fused based on the corresponding weight coefficient, and the color expression of the 3D Gaussian splash algorithm in the appearance of the reflected view is supplemented by the reflective highlight color component, and the ability to represent diffuse reflection and capture details is enhanced by the diffuse color component, thereby improving the quality and realism of the target reconstructed image. The method provided in the embodiment of the present application can be performed by an electronic device, and the electronic device can be various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a vehicle terminal, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or can be implemented as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0062] Below, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application.

[0063] In one embodiment of the present application, Figure 1 A schematic diagram of an implementation process of an image reconstruction method provided in an embodiment of the present application Figure 1 .

[0064] like Figure 1 As shown, the image reconstruction method may include:

[0065] S101, generating a Gaussian ellipsoid set based on a point cloud corresponding to an image to be reconstructed.

[0066] The Gaussian ellipsoid set includes multiple Gaussian ellipsoids, and each Gaussian ellipsoid has a corresponding relationship with a point in the point cloud.

[0067] In an embodiment of the present application, the image to be reconstructed may include two-dimensional images of an object or scene at multiple different shooting distances or shooting angles, the object or scene includes more than two objects, and there are parts or elements with reflective properties, and these parts or elements can form highlight areas on the image to be reconstructed.

[0068] In an embodiment of the present application, a point cloud corresponding to the image to be reconstructed can be generated by a motion structure recovery algorithm, and the point cloud can be a collection of data points on the surface of an object in the image to be reconstructed in three-dimensional space. Using the motion structure recovery algorithm, key points or feature points are first detected from the image sequence to be reconstructed; feature descriptors, such as the Scale Invariant Feature Transform (SIFT) algorithm and the Speeded Up Robust Features (SURF) algorithm, are used to match the extracted feature points to find the corresponding feature point pairs in different images to be reconstructed; the matched feature point pairs are then used to describe the epipolar geometric relationship between the two camera views through the basic matrix, and the result of the basic matrix estimation is used to further calculate the internal and external parameters of the camera, which include the focal length, optical center and other internal properties of the camera, and the external parameters describe the position and posture of the camera in the world coordinate system; finally, the internal and external parameters of the camera and the matched feature point pairs are used to calculate the coordinates of the point cloud corresponding to the image to be reconstructed.

[0069] In an embodiment of the present application, after obtaining the point cloud corresponding to the image to be reconstructed, each point in the point cloud can be further initialized as a Gaussian ellipsoid. Specifically, each point in the point cloud can be initialized as a Gaussian ellipsoid using the spherical harmonic order and Gaussian sphere initialization parameters obtained by pre-training the 3D Gaussian splash algorithm. Exemplarily, a third-order spherical harmonic function can be used to describe anisotropic Gaussian sphere expression scenes, and the initialization parameters of each Gaussian ellipsoid may include the center point position (Gaussian mean), covariance matrix, scale, opacity, spherical harmonic function features (used to represent color), etc.

[0070] S102, determining a weight coefficient corresponding to the Gaussian ellipsoid according to geometric parameters of the Gaussian ellipsoid.

[0071] It should be noted that the geometric parameters of the Gaussian ellipsoid can be determined based on the spherical harmonics of the Gaussian ellipsoid, which can be used to describe the geometric characteristics of the Gaussian ellipsoid, such as the position information, normal vector information, shape information, reflection information and color information of the Gaussian ellipsoid.

[0072] In the embodiment of the present application, the weight coefficient corresponding to the Gaussian ellipsoid may be used to determine the diffuse reflection weight coefficient corresponding to the diffuse reflection color component, or the reflective highlight weight coefficient corresponding to the reflective highlight color component. Since the weight coefficient is 1, the weight coefficient corresponding to the other color component may be determined by 1-weight coefficient. Exemplarily, a pre-trained sigmoid single-layer perceptron may be used to determine the weight coefficients of the diffuse reflection color component and the reflective highlight color component in the Gaussian ellipsoid according to the geometric parameters of an input Gaussian ellipsoid.

[0073] In the embodiment of the present application, the geometric parameters corresponding to different Gaussian ellipsoids in the Gaussian ellipsoid set may also be different. Therefore, for a certain Gaussian ellipsoid in the Gaussian ellipsoid set, the weight coefficient corresponding to the Gaussian ellipsoid can be determined based on the geometric parameters of the Gaussian ellipsoid using a pre-trained sigmoid single-layer perceptron, and this step is repeated until the weight coefficient corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set is obtained.

[0074] S103, determining the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector.

[0075] It should be noted that the color of an object observed by the human eye is actually the light transmitted from the light source to the surface of the object and then reflected on the surface and then transmitted to the human eye. The reflection of light on the surface of the object is divided into diffuse reflection and reflection. Among them, the intensity of light reflected in all directions on the surface of the object is the same for diffuse reflection, which is called diffuse reflection color; reflection is also called specular reflection, and the intensity of reflected light in different directions on the surface of the object is different, which is called specular reflection color. In related technologies, the Gaussian splash algorithm directly performs differentiable rasterization on the Gaussian ellipsoid, without considering the reflection of light by components with reflective characteristics at different viewing angles, resulting in low realism and poor quality of the reconstructed image.

[0076] In the embodiment of the present application, the observation direction refers to the direction from the camera to a certain Gaussian ellipsoid. Since the image to be reconstructed may contain multiple two-dimensional images of different viewing angles, there may be multiple observation directions. The observation encoding matrix obtained after feature encoding (Encoding) of multiple observation directions is the observation direction vector corresponding to the Gaussian ellipsoid. Among them, the encoding method is preferably but not limited to sine-cosine periodic function encoding.

[0077] Furthermore, for any Gaussian ellipsoid in the Gaussian ellipsoid set, based on the geometric parameters corresponding to the Gaussian ellipsoid and the observation direction vector corresponding to the Gaussian ellipsoid, illustratively, a multi-layer perceptron network (Multi-LayerPerceptron, MLP) can be used to output the diffuse reflection color component corresponding to the Gaussian ellipsoid, and this step is repeated until the diffuse reflection color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set is obtained.

[0078] S104, determining the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector.

[0079] In the embodiment of the present application, the reflection direction may be the direction of light reflected after the light irradiates the component with reflective characteristics in the image to be reconstructed. The reflection direction vector is the reflection encoding matrix obtained after the reflection direction is feature encoded, and the encoding method is preferably but not limited to sine-cosine periodic function encoding.

[0080] Furthermore, for any Gaussian ellipsoid in the Gaussian ellipsoid set, based on the geometric parameters corresponding to the Gaussian ellipsoid and the observation direction vector corresponding to the Gaussian ellipsoid, illustratively, a multi-layer perceptron network can be used to output the reflection highlight color component corresponding to the Gaussian ellipsoid, and this step is repeated until the reflection highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set is obtained.

[0081] S105, performing fusion processing on the weight coefficient, diffuse reflection color component and reflection highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set respectively to generate a target reconstructed image.

[0082] In the embodiment of the present application, for any Gaussian ellipsoid in the Gaussian ellipsoid set, according to the diffuse color component, the reflection highlight color component and the weight coefficient of the Gaussian ellipsoid, the diffuse color component and the reflection highlight color component are fused according to their respective corresponding weights to obtain the mixed color corresponding to the Gaussian ellipsoid, so that the diffuse color component and the reflection highlight color component can achieve a better color decomposition effect, and the color expression of the 3D Gaussian splash algorithm in the reflection view appearance is supplemented by the reflection highlight color component. However, it is difficult to represent the diffuse reflection and blurred reflection in the object or scene only by the reflection direction color component, and the diffuse color component is used to enhance the ability to represent diffuse reflection and capture details, so that the Gaussian ellipsoid can present a better modeling effect. This step is repeated until each Gaussian ellipsoid in the Gaussian ellipsoid set is fused.

[0083] In the embodiment of the present application, the projections of the multiple Gaussian ellipsoids obtained by the above fusion in three dimensions are transformed into a two-dimensional plane to generate a target reconstructed image, wherein the target reconstructed image is an image reconstructed at a required viewing angle after three-dimensional modeling by the Gaussian ellipsoid set.

[0084] In the embodiments of the present application, Figure 2 A schematic diagram of a target reconstructed image and different color components provided in an embodiment of the present application. Figure 2 As shown, for three different images, a1, a2 and a3 are target reconstructed images, b1, b2 and b3 are images corresponding to diffuse reflection color components, and c1, c2 and c3 are images corresponding to reflective highlight color components.

[0085] It should also be noted that the above steps S102, S103 and S104 can be executed in parallel or in a preset order.

[0086] The embodiment of the present application provides an image reconstruction method. First, based on the point cloud corresponding to the image to be reconstructed, a Gaussian ellipsoid set is generated. In this way, multiple Gaussian ellipsoids are generated based on the 3D Gaussian splash algorithm, which not only improves the interpretability of the three-dimensional scene represented by it, but also can render a detailed and realistic scene with high efficiency; secondly, based on the geometric parameters of the Gaussian ellipsoid, the weight coefficient corresponding to the Gaussian ellipsoid is determined, based on the geometric parameters of the Gaussian ellipsoid and the observation direction vector, the diffuse color component corresponding to the Gaussian ellipsoid is determined, and based on the geometric parameters of the Gaussian ellipsoid and the reflection direction vector, the reflection highlight color component corresponding to the Gaussian ellipsoid is determined. In this way, the method can effectively decompose the diffuse reflection and highlight reflection colors in the object or scene, so that the image reconstruction method provided by the present application can reconstruct with high quality even in objects or scenes with reflective properties, enhance the reconstruction performance of the 3D Gaussian splash algorithm on image data sets with metallic luster, light reflection and shadow, and support real-time rendering. Finally, the diffuse color component and the reflective highlight color component are fused based on the weight coefficient to generate a target reconstructed image, so that the generated target reconstructed image is fused based on the corresponding weight coefficient, and the color expression of the 3D Gaussian splash algorithm in the reflected view appearance is supplemented by the reflective highlight color component, and the ability to represent diffuse reflection and capture details is enhanced by the diffuse color component, thereby improving the quality and realism of the target reconstructed image.

[0087] In another embodiment of the present application, for the aforementioned step S102, determining the weight coefficient corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid may include:

[0088] S201, concatenating the shortest axis direction vector corresponding to the Gaussian ellipsoid and the eigenvector of the spherical harmonic function to obtain a characteristic matrix corresponding to the Gaussian ellipsoid.

[0089] In the embodiment of the present application, the geometric parameters of the Gaussian ellipsoid may include the relevant features of the spherical harmonics of the Gaussian ellipsoid, such as the shortest axis direction vector and the spherical harmonics feature vector corresponding to the Gaussian ellipsoid. Among them, the shortest axis direction vector corresponding to any Gaussian ellipsoid in the Gaussian ellipsoid set refers to the normal vector corresponding to the short axis length of the Gaussian ellipsoid; the spherical harmonics feature corresponding to any Gaussian ellipsoid in the Gaussian ellipsoid set refers to the vector composed of the coefficients of the basis function of the spherical harmonics of the Gaussian ellipsoid.

[0090] In the embodiment of the present application, a matrix obtained by concatenating the shortest axis direction vector corresponding to a Gaussian ellipsoid and the spherical harmonic function eigenvector corresponding to the Gaussian ellipsoid is used as the characteristic matrix corresponding to the Gaussian ellipsoid.

[0091] S202, inputting the feature matrix corresponding to the Gaussian ellipsoid into the target weight perception model to obtain the weight coefficient corresponding to the Gaussian ellipsoid.

[0092] In the embodiment of the present application, the target weight perception model may also be referred to as a weight perception model, which may be obtained by pre-training the constructed initial weight perception module. The weight perception model is preferably, but not limited to, a sigmoid single-layer perceptron, which learns a weight coefficient W between 0 and 1, that is, the weight coefficient W of each Gaussian ellipsoid is sigmoid (f w (sh,n)), where n represents the shortest axis direction vector of each Gaussian ellipsoid, which is used as the normal vector of the estimated value; sh represents the spherical harmonic function feature encoding color information in 3D Gaussian; the function expression of sigmoid is y represents the input feature of the sigmoid function, and the weight coefficient W can be the weight coefficient corresponding to the diffuse color component or the weight coefficient corresponding to the reflective highlight color component, which is actually determined according to the training situation.

[0093] It should be noted that, for any Gaussian ellipsoid in the Gaussian ellipsoid set, the weight coefficient corresponding to the any Gaussian ellipsoid can be determined based on the above steps S201 to S202.

[0094] The embodiment of the present application provides an image reconstruction method, which concatenates the shortest axis direction vector corresponding to the Gaussian ellipsoid and the eigenvector of the spherical harmonic function to obtain a characteristic matrix corresponding to the Gaussian ellipsoid, and further inputs the characteristic matrix into the target weight perception model to obtain the weight coefficient corresponding to the Gaussian ellipsoid. In this way, the weights corresponding to the diffuse color component and the reflective highlight color component that constitute the Gaussian ellipsoid can be determined, thereby improving the color expression effect of the target reconstructed image.

[0095] In some embodiments, for the aforementioned step S105, the weight coefficient, diffuse color component and reflective highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set are fused respectively to generate a target reconstructed image, which may include:

[0096] S301, based on the weight coefficient corresponding to the Gaussian ellipsoid, determine the diffuse reflection weight coefficient corresponding to the diffuse reflection color component and the reflective highlight weight coefficient corresponding to the reflective highlight color component.

[0097] Based on the above embodiment, the feature matrix corresponding to a Gaussian ellipsoid is input into the trained target weight perception model, and the weight coefficient W corresponding to the Gaussian ellipsoid is output. It should be understood that the weight coefficient W may refer to the diffuse reflection weight coefficient corresponding to the diffuse reflection color component of the Gaussian ellipsoid, and the reflection highlight weight coefficient corresponding to the reflection highlight color component of the Gaussian ellipsoid is (1-W); or, W is the reflection highlight weight coefficient corresponding to the reflection highlight color component of the Gaussian ellipsoid, and the diffuse reflection weight coefficient corresponding to the diffuse reflection color component of the Gaussian ellipsoid is (1-W).

[0098] S302, based on the diffuse reflection weight coefficient and the reflection highlight weight coefficient, weighted calculation is performed on the diffuse reflection color component corresponding to the Gaussian ellipsoid and the reflection highlight color component corresponding to the Gaussian ellipsoid to determine the mixed color component corresponding to the Gaussian ellipsoid.

[0099] In the embodiment of the present application, the diffuse reflection weight coefficient and the reflection highlight weight coefficient corresponding to each Gaussian ellipsoid are used to fuse the diffuse reflection color component and the reflection highlight color component corresponding to the Gaussian ellipsoid to obtain the mixed color component corresponding to the Gaussian ellipsoid. The fusion method can be weighted fusion through the weight coefficient. Preferably, the mixed color component of any Gaussian ellipsoid can be calculated and determined using the following formula (1):

[0100] C final =W·C diffuse +(1-W)·C specular (1)

[0101] Wherein, for example, W represents the diffuse reflection weight coefficient corresponding to any Gaussian ellipsoid, which is obtained by learning the weighted perceptron, and (1-W) represents the reflection highlight weight coefficient. diffuse Represents the diffuse color component of the Gaussian ellipsoid, C specular Indicates the reflection highlight color component corresponding to the Gaussian ellipsoid.

[0102] It should be noted that, for any Gaussian ellipsoid in the Gaussian ellipsoid set, its corresponding mixed color component can be determined based on the above formula (1), and step S301 is repeated until the mixed color components corresponding to all Gaussian ellipsoids in the Gaussian ellipsoid set are obtained.

[0103] S303, performing differentiable rasterization processing on the mixed color components corresponding to the multiple Gaussian ellipsoids in the Gaussian ellipsoid set to generate a target reconstructed image.

[0104] In the embodiment of the present application, all Gaussian ellipsoids are processed by differentiable rasterization based on their respective mixed colors to obtain a reconstructed rendering image of the object or scene, that is, a target reconstructed image. Specifically, the reconstructed rendering image of the object or scene can be obtained based on the differentiable rasterization method in the pre-trained 3D Gaussian splash algorithm in the embodiment of the present application.

[0105] It should be noted that Differentiable Rasterization is the process of converting a 3D scene into a 2D image. The three-dimensional Gaussian ellipsoid set is projected onto a two-dimensional plane using a two-dimensional covariance matrix to generate a target reconstructed image.

[0106] The embodiment of the present application provides a one-time image reconstruction method, which performs weighted calculation on the diffuse color component corresponding to the Gaussian ellipsoid and the reflected highlight color component corresponding to the Gaussian ellipsoid based on the weight coefficient corresponding to the Gaussian ellipsoid, determines the mixed color component corresponding to the Gaussian ellipsoid, and further generates a target reconstructed image based on the mixed color components corresponding to all Gaussian ellipsoids in the Gaussian ellipsoid set. In this way, the three-dimensional modeling can be converted to a two-dimensional plane more efficiently, thereby improving the efficiency of generating the target reconstructed image.

[0107] In some embodiments, for the above step S302, performing differentiable rasterization processing on the mixed color components corresponding to the multiple Gaussian ellipsoids in the Gaussian ellipsoid set to generate the target reconstructed image may include:

[0108] S401, projecting the Gaussian ellipsoid set into an initial reconstructed image based on the transformation matrix.

[0109] In the embodiment of the present application, each Gaussian ellipsoid can be expressed as the following formula (2):

[0110]

[0111] Where x is the position of any point in the object or scene, that is, the coordinates of the center of the Gaussian ellipsoid corresponding to the scene. ∑ is the three-dimensional covariance matrix, or 3D covariance matrix, and μ is the Gaussian mean. The three-dimensional covariance matrix can be further decomposed into a rotation matrix R and a scaling matrix S.

[0112] Wherein, ∑ can be expressed based on the following formula (3):

[0113] ∑=RSS T R T (3)

[0114] In the embodiment of the present application, all Gaussian ellipsoids in the three-dimensional Gaussian ellipsoid set are projected onto a two-dimensional plane through a two-dimensional covariance matrix, i.e., a transformation matrix. That is, the covariance matrix representing the Gaussian ellipsoid is multiplied by the transformation matrix from the three-dimensional projection to the two-dimensional plane. The transformation matrix can be determined based on the following formula (4):

[0115] ∑ ′ =JW∑W T J T (4)

[0116] Among them, J is the preset Jacobian matrix of the projection transformation, W is the given observation matrix, ∑ is the three-dimensional covariance matrix, ∑ ′ is the two-dimensional covariance matrix, i.e., the transformation matrix.

[0117] In the embodiment of the present application, the above transformation matrix ∑ ′ After that, the covariance matrix representing the Gaussian ellipsoid in the Gaussian ellipsoid set can be multiplied by the transformation matrix, and the advanced ellipsoid set can be projected onto a two-dimensional plane to generate an initial reconstructed image.

[0118] S402, rendering the initial reconstructed image based on the mixed color components corresponding to the multiple Gaussian ellipsoids to generate a target reconstructed image.

[0119] In the embodiment of the present application, the initial reconstructed image can be further divided into regular blocks of a preset size, for example, 16×16. During the pixel rendering process, all Gaussian balls overlapping at the position are calculated by view change and sorted by depth. Finally, the arranged Gaussian balls are subjected to alpha blending, and the final mixed color value C(x ′ ), the calculation formula is shown in the following formula (5):

[0120]

[0121] Among them, x ′ is the position of the pixel coordinate in the initial reconstructed image, c i Represents the mixed color component corresponding to the i-th Gaussian ellipsoid, α i Represents the opacity of the i-th Gaussian ellipsoid. σ i is a value between 0 and 1, which is used to indicate the contribution of the i-th Gaussian ellipsoid to the final pixel color. N is the ordered Gaussian ellipsoid projected onto the two-dimensional plane. G i ′ (x ′ ) represents the i-th Gaussian ellipsoid at position x ′ The two-dimensional Gaussian representation of the pixel after the projective transformation. Using differentiable rasterization, the properties of the Gaussian ellipsoid can be directly optimized.

[0122] The embodiment of the present application provides an image reconstruction method, which projects a Gaussian ellipsoid set into an initial reconstructed image through a transformation matrix, and further renders the initial reconstructed image based on the mixed color components corresponding to each of the multiple Gaussian ellipsoids to generate a target reconstructed image. In this way, based on multiple Gaussian ellipsoids with mixed color components, the target reconstructed image is projected and generated, which can achieve high-quality reconstruction in a scene with reflective properties, thereby improving the effect of the target reconstructed image.

[0123] In another embodiment of the present application, for the aforementioned step S103, determining the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector may include:

[0124] S501, concatenating the feature matrix and the observation direction vector to obtain an observation direction matrix.

[0125] In the embodiment of the present application, the shortest axis direction vector corresponding to any Gaussian ellipsoid and the characteristic matrix composed of the corresponding spherical harmonic function characteristics can be spliced ​​with the observation direction vector to generate an observation direction matrix.

[0126] Alternatively, in some embodiments, the spherical harmonics encoding same as 3D Gaussian splash can be concatenated with the feature matrix to obtain the observation direction matrix. The spherical harmonics encoding processing formula is the following formula (6):

[0127]

[0128] Among them, d is the observation direction, k is the spherical harmonic coefficient, Y is the spherical harmonic basis function, l and m represent the order and corresponding basis of the spherical harmonic of the Gaussian ellipsoid, respectively.

[0129] S502, inputting the viewing direction matrix into the target viewing direction processing model to obtain the diffuse reflection color component corresponding to the Gaussian ellipsoid.

[0130] In an embodiment of the present application, the target observation direction processing model, which may also be referred to as the observation direction color processing model, may be obtained by pre-training the initial observation direction processing model, and may be a multi-layer perceptron network, such as a two-layer, three-layer, or four-layer MLP. Preferably, the target observation direction processing model adopts a three-layer perceptron structure. The ReLU function is used as the activation function between the MLP layers of the target observation direction processing model, and the expression of the ReLU function is: ReLU(x0)=max(0,x0), where x0 is the independent variable of the ReLU function. Specifically, the process of obtaining the diffuse reflection color component of the Gaussian ellipsoid using the target observation direction processing model can be determined based on the following formula (7):

[0131] C diffuse =f view(sh,n,ω0) (7)

[0132] Among them, f view (,,) represents the target observation direction processing model, and ω0 represents the observation direction vector from the camera position to the Gaussian ellipsoid position.

[0133] The embodiment of the present application provides an image reconstruction method, which generates an observation direction matrix based on the splicing of the feature matrix and the observation direction vector, further inputs the observation direction matrix into the target observation direction processing model, and outputs the diffuse color component corresponding to the Gaussian ellipsoid. In this way, there is a strong correlation between the diffuse color component and the observation direction, which can represent the diffuse reflection and blurred reflection in the scene, and improves the method's ability to represent diffuse reflection and capture details.

[0134] In some embodiments, for the aforementioned step S104, determining the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector may include:

[0135] S601, concatenating the characteristic matrix and the reflection direction vector to obtain a reflection direction matrix.

[0136] In the embodiment of the present application, the shortest axis direction vector corresponding to any Gaussian ellipsoid, the corresponding spherical harmonic function characteristics and the corresponding reflection direction vector can be spliced ​​to generate a reflection direction matrix.

[0137] Alternatively, in some embodiments, the shortest axis direction vector corresponding to any Gaussian ellipsoid, the corresponding spherical harmonic function feature, and the spherical harmonic function code identical to the 3D Gaussian splash may be concatenated to generate a reflection direction matrix.

[0138] S602, inputting the reflection direction matrix into the target reflection direction processing model to obtain the reflection highlight color component corresponding to the Gaussian ellipsoid.

[0139] In an embodiment of the present application, the target reflection direction processing model may also be referred to as a reflection direction color processing model, which may be obtained based on pre-training of the initial reflection direction processing model. The target reflection direction processing model may be a multi-layer perceptron network, such as a two-layer, three-layer, or four-layer MLP. Preferably, the reflection direction color processing model adopts a three-layer perceptron structure. The ReLU function is used as an activation function between the MLP layers of the target reflection direction processing model. Specifically, the process of obtaining the reflection highlight color component of the Gaussian ellipsoid using the target reflection direction processing model may be expressed based on the following formula (8):

[0140] C specular =f ref (sh,n,ω v ) (8)

[0141] Among them, f ref (,,) represents the target reflection direction processing model, ω v Represents the reflection direction vector.

[0142] The embodiment of the present application provides an image reconstruction method, which generates a reflection direction matrix based on the splicing of the feature matrix and the reflection direction vector, further inputs the reflection direction matrix into the target reflection direction processing model, and outputs the reflection highlight color component corresponding to the Gaussian ellipsoid. In this way, there is a strong correlation between the reflection highlight color component and the reflection direction, which can supplement the color expression of the 3D Gaussian in the appearance of the reflection view, improve the reconstruction realism of the image to be reconstructed with the reflection component, and improve the reconstruction quality of the image.

[0143] In some embodiments, the image reconstruction method may further include:

[0144] S701, determining the observation direction of the Gaussian ellipsoid according to the camera parameters of the image to be reconstructed and the coordinates of the center of the Gaussian ellipsoid.

[0145] In the embodiment of the present application, the camera parameters of the image to be reconstructed may include, for example, the camera position coordinates of the image to be reconstructed. The observation direction of the Gaussian ellipsoid may be determined by the camera position coordinates of the image to be reconstructed in three-dimensional space toward the center of any Gaussian ellipsoid. It should be understood that when the image to be reconstructed includes multiple two-dimensional images, the camera position coordinates of each two-dimensional image may be different, and the observation direction of the Gaussian ellipsoid may be determined by combining the multiple camera position coordinates and the coordinates of the center of any Gaussian ellipsoid.

[0146] S702, encoding the observation direction to determine the observation direction vector of the Gaussian ellipsoid.

[0147] The observation direction vector can be generated by encoding the observation direction through a sine-cosine periodic function. The encoding processing formula of the sine-cosine periodic function can be the following formula (9):

[0148] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) (9)

[0149] Wherein, p represents the position of the character in the viewing direction vector.

[0150] S703, determining a reflection direction vector corresponding to the Gaussian ellipsoid according to the shortest axis direction vector corresponding to the Gaussian ellipsoid and the observation direction vector.

[0151] In the embodiment of the present application, in order to more accurately and quickly obtain the reflection direction vector of the Gaussian ellipsoid, according to a simple incident reflection illumination physical model, the reflection direction vector ω of each Gaussian ellipsoid is v It can be calculated based on the shortest axis direction vector n of the Gaussian ellipsoid and the observation direction vector ω0, as shown in the following formula (10):

[0152] ω v =2(ω0·n)n-ω0 (10)

[0153] The embodiment of the present application provides an image reconstruction method, which determines the observation direction of the Gaussian ellipsoid through the camera parameters of the image to be reconstructed and the coordinates of the center of any Gaussian ellipsoid, further encodes the observation direction, determines the observation direction vector of the Gaussian ellipsoid, and then calculates and determines the reflection direction vector corresponding to the Gaussian ellipsoid based on the shortest axis direction vector and the observation direction vector corresponding to the Gaussian ellipsoid. In this way, the observation direction can be accurately determined, thereby improving the quality of the generated target reconstructed image, and the calculation rate of the reflection direction vector can be increased, thereby improving the performance of image reconstruction.

[0154] In yet another embodiment of the present application, the image reconstruction method may further include:

[0155] S801, constructing an initial weight perception model, an initial observation direction processing model and an initial reflection direction processing model.

[0156] In the embodiment of the present application, the network structure of the initial weight perception model, the initial observation direction processing model and the initial reflection direction processing model can be constructed based on the preset network parameters, and the network parameters of the network structure can be initialized. Specifically, as mentioned above, the initial weight perception model is a single-layer perceptron structure, and the initial observation direction processing model and the initial reflection direction processing model both adopt a three-layer perceptron network structure.

[0157] S802: construct a training sample set, where the training sample set includes a plurality of training samples.

[0158] Each training sample includes a synthetic Gaussian ellipsoid set and a corresponding real three-dimensional image.

[0159] In an embodiment of the present application, each training sample set includes multiple training samples, wherein each training sample includes a set of synthetic Gaussian ellipsoid sets corresponding to an object or scene in an image to be reconstructed and a real three-dimensional image corresponding to the image to be reconstructed. The synthetic Gaussian ellipsoid set can be a set of images of an object or scene in an image to be reconstructed, which is transformed into a point cloud by a motion structure recovery algorithm, and further based on a pre-trained 3D Gaussian splash algorithm, the point cloud is initialized as a Gaussian ellipsoid set generated by a Gaussian ellipsoid; the real three-dimensional image is a real three-dimensional image corresponding to the same image to be reconstructed, and specifically can be a three-dimensional image collected using a camera or a collector.

[0160] S803, based on the target loss function, the initial weight perception model, the initial observation direction processing model and the initial reflection direction processing model are jointly trained using the training sample set to obtain the target weight perception model, the target observation direction processing model and the target reflection direction processing model.

[0161] In an embodiment of the present application, after obtaining the training sample set, the target loss function is used as the target, and the synthetic Gaussian ellipsoid set in the training sample set is used to jointly train the initial weight perception model, the initial observation direction processing model, and the initial reflection direction processing model. Among them, the initial weight perception model is used to obtain the weight coefficient corresponding to each Gaussian ellipsoid in the synthetic Gaussian ellipsoid set, the initial observation direction processing model is used to obtain the diffuse color component corresponding to each Gaussian ellipsoid in the synthetic Gaussian ellipsoid set, and the initial reflection direction processing model is used to obtain the reflection highlight color component corresponding to each Gaussian ellipsoid in the synthetic Gaussian ellipsoid set. The diffuse color component and the reflection highlight color component are weighted and fused using the weight coefficient to obtain the mixed color component of the Gaussian ellipsoid, and then all the Gaussian ellipsoids of the sample are subjected to differentiable rasterization processing to obtain the synthetic reconstructed image corresponding to the sample.

[0162] Furthermore, according to the synthetically reconstructed image and the real three-dimensional image, the total loss is calculated based on the target loss function, and the network parameters are updated using the total loss gradient descent method. When the total loss reaches the minimum or reaches the preset number of training times, the joint training is stopped, and based on the pre-constructed test sample set, the preset evaluation parameters in the following embodiment are used as evaluation indicators to evaluate the three trained models. After the evaluation passes, the three trained models are used as the target weight perception model, the target observation direction processing model, and the target reflection direction processing model. It should be understood that the test sample set contains multiple test samples, each of which includes a synthetic Gaussian ellipsoid set and a corresponding real three-dimensional image.

[0163] The embodiment of the present application provides an image reconstruction method, which uses a training sample set to jointly train the constructed initial weight perception model, initial observation direction processing model and initial reflection direction processing model based on the target loss function, and obtains a target weight perception model, a target observation direction processing model and a target reflection direction processing model. In this way, multiple models can share information and optimize together, which improves the generalization ability of the model and the efficiency of model training.

[0164] In some embodiments, the image reconstruction method may further include:

[0165] S901, construct an initial loss function based on the Gaussian loss function, the mixed photometric loss function, the smoothing loss function and the preset coefficients.

[0166] In the embodiment of the present application, before the above step S101, a 3D Gaussian splash algorithm can be used to perform multiple generations of training based on the 3D Gaussian splash algorithm photometric loss function to complete the initialization of the Gaussian ellipsoid and improve the spherical harmonic order of the Gaussian ellipsoid. Among them, the 3D Gaussian splash algorithm photometric loss function, which can also be called the Gaussian loss function, can be expressed as the following formula (11):

[0167]

[0168] Among them, λ1 represents the 3D Gaussian balance coefficient, Represents the true color C of the sample and the color obtained by the 3D Gaussian splash algorithm The absolute error between them, the absolute error is preferably but not limited to the average of the absolute color differences of all pixel pairs between the real image of the sample and the image reconstructed by the 3D Gaussian splashing algorithm; Indicates the structural similarity between the real image of the sample and the image reconstructed by the 3D Gaussian splash algorithm. When the photometric loss function of the 3D Gaussian splash algorithm reaches the minimum, or when the preset number of iterative training is reached, the training is stopped, and the 3D Gaussian splash algorithm parameters obtained after training are saved.

[0169] In the embodiment of the present application, in order to optimize the Gaussian sphere properties and improve the reconstruction quality of reflective object or scene in joint training, the initial loss function The weighted sum of the Gaussian loss calculated by the Gaussian loss function, the mixed photometric loss calculated by the mixed photometric loss function, and the smoothing loss calculated by the smoothing loss function can be expressed by the following formula (12):

[0170]

[0171] Among them, λ1 represents the 3D Gaussian loss weight, that is, the weight of the Gaussian loss, λ2 is the weight of the mixed photometric loss, and λ smooth To smooth the loss weight, the preset coefficients include λ1, λ2 and λ smooth .

[0172] In the embodiment of the present application, the mixed luminosity loss The calculation formula is as follows:

[0173]

[0174] Among them, λ represents the balance coefficient, Represents the true color C of the sample and the mixed color The absolute error between them is specifically the mean absolute color difference of all pixel pairs between the real image of the sample and the reconstructed rendered image; The structural similarity between the real image and the synthetically reconstructed image representing the sample can be calculated according to the above SSIM structural similarity calculation formula.

[0175] It should be noted that using only mixed photometric loss supervision, the color decomposition effect of training is poor and not interpretable. Therefore, a smoothing loss for camera color is introduced to effectively decouple the colors between components. The calculation formula is as follows:

[0176]

[0177] Among them, I i,j Represents the pixel value at position (i, j) in the synthetic reconstructed image I.

[0178] S902, adjusting the preset coefficients based on the preset evaluation parameters to determine the target loss function.

[0179] The preset evaluation parameters are used to evaluate the reconstruction effect of the target reconstructed image.

[0180] In an embodiment of the present application, the preset evaluation parameters may include parameters such as Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning image block similarity. Based on these preset evaluation parameters, the target weight perception model, target observation direction processing model, and target reflection direction processing model obtained by training based on the above steps S801 to S803 using the target loss function, as well as the target reconstructed image generated by differentiable rasterization processing, are evaluated. After multiple tests, a set of preset coefficients with better preset evaluation parameter values, that is, a better reconstruction effect of the target reconstructed image, is used as the coefficients of the target loss function.

[0181] In the embodiment of the present application, the peak signal-to-noise ratio can be calculated and determined based on the following formula (15):

[0182]

[0183] Wherein, MAX1 represents the maximum value of the color of the synthetically reconstructed image point, and MSE represents the mean square error of two m×n monochrome images I and K. MSE can be calculated and determined based on the following formula (16):

[0184]

[0185] In the embodiment of the present application, the structural similarity SSIM of two images can be calculated and determined based on the following formula (17):

[0186]

[0187] Among them, μ x and μ y represents two images, namely the real image and the synthetic reconstructed image (reconstructed rendered image) in the sample image, or the pixel mean of the real image of the sample and the reconstructed image x, y obtained based on the above model after training, σ x and σ y Represents the pixel variance of two images x and y, σ xy Represents the pixel covariance between two images x and y; c1 = (k1L1) 2 、c2=(k2L2) 2 Represent two constants, L1 and L2 represent the pixel value ranges of the two images x and y respectively, and k1 and k2 represent the first coefficient and the second coefficient respectively.

[0188] In an embodiment of the present application, learned perceptual image patch similarity (LPIPS), also known as perceptual loss, is used to measure the difference between two images. Its core principle is to use a deep learning model to simulate the human visual system's perception of image content. It is an image similarity evaluation indicator based on deep learning.

[0189] The embodiment of the present application provides an image reconstruction method, which adjusts the preset coefficients in the initial loss function based on the preset evaluation parameters to determine the target loss function. In this way, multiple models generated by training based on the target loss function can have good performance in the reconstruction of different images, thereby improving the quality of the target reconstructed image.

[0190] In another embodiment of the present application, Figure 3 A schematic diagram of a framework of an image reconstruction method provided in an embodiment of the present application. Figure 3 As shown, the image reconstruction method may include the following steps:

[0191] S1001 , obtaining an original image group of an object or scene, performing motion structure recovery processing on the original image group to obtain a point cloud 122 , and initializing each point in the point cloud 122 as a Gaussian ellipsoid 123 .

[0192] The original image group of the object or scene, i.e., the image to be reconstructed 121 in the aforementioned embodiment, may include two-dimensional images of the object or scene at multiple different shooting distances or shooting angles, and there are parts or elements with reflective properties on the object or in the scene, forming a highlight area on the original image. Moreover, the scene includes more than two objects.

[0193] In the step of initializing each point in the point cloud 122 as a Gaussian ellipsoid 123, each point is initialized as a Gaussian ellipsoid 123 according to the spherical harmonic order and Gaussian sphere initialization parameters obtained by pre-training the 3D Gaussian splash algorithm, referring to Figure 2 In one example, the present application uses a third-order spherical harmonic function to describe anisotropic Gaussian spheres to express scenes, and each Gaussian ellipsoid 123 includes attributes such as center point position (Gaussian mean), covariance matrix, scale, opacity, spherical harmonic function features (used to represent color), etc.

[0194] S1002, using a weight sensor (target weight perception model) to process the geometric parameters of the Gaussian ellipsoid 123 of each point in the point cloud 122, such as the shortest axis direction vector 1242 and the spherical harmonic function eigenvector 1241, to obtain the weight coefficient of each Gaussian ellipsoid. Specifically, the matrix of the shortest axis direction vector 1242 and the spherical harmonic function eigenvector 1241 of the Gaussian ellipsoid of each point is spliced ​​together as a feature matrix, and the feature matrix is ​​input into the weight sensor, that is, the target weight perception model 1271.

[0195] The weighted perceptron is preferably, but not limited to, a sigmoid single-layer perceptron.

[0196] S1003, using the observation direction color processing model (target observation direction processing model 1273), the shortest axis direction vector of the Gaussian ellipsoid of each point in the point cloud 122, the spherical harmonic function characteristics and the observation direction vector 1261 from the camera position to the Gaussian ellipsoid position are processed to obtain the diffuse reflection color component of each Gaussian ellipsoid.

[0197] The observation direction color processing model may be a multi-layer perceptron network.

[0198] Specifically, the observation direction vector is the observation coding matrix obtained after the observation direction 1251 is feature encoded. The encoding method is preferably but not limited to sine-cosine periodic function encoding, or spherical harmonic function encoding the same as 3D Gaussian splash. The observation coding matrix and the feature matrix are concatenated to obtain a new matrix, and the new matrix is ​​input into the observation direction color processing model for processing.

[0199] S1004, using the reflection direction color processing model (target reflection direction processing model 1272), the shortest axis direction vector 1242, the spherical harmonic function eigenvector 1241 and the reflection direction vector 1262 of the Gaussian ellipsoid of each point in the point cloud 122 to obtain the reflection highlight color component of each Gaussian ellipsoid.

[0200] Among them, the reflection direction color processing model can be a multi-layer perceptron network.

[0201] Specifically, the reflection direction vector 1262 is a reflection coding matrix obtained after the reflection direction 1252 is feature encoded. The encoding method is preferably but not limited to sine-cosine periodic function encoding, or spherical harmonic function encoding the same as 3D Gaussian splash. The reflection coding matrix and the feature matrix are concatenated to obtain a new matrix, and the new matrix is ​​input into the reflection direction color processing model for processing.

[0202] In the embodiment of the present application, a better color decomposition effect is achieved through the observation direction color processing model and the reflection direction color processing model. The reflection direction color processing model supplements the color expression of the 3D Gaussian in the reflection view appearance, but it is difficult to represent the diffuse reflection and blurred reflection in the object or scene by the reflection direction color processing model alone. Therefore, setting the observation direction color processing model enhances the correlation between the Gaussian ball attribute characteristics and the observation direction, and improves the model's ability to represent diffuse reflection and capture details.

[0203] S1005, using the weight coefficient to fuse the diffuse color component and the reflective highlight color component of each Gaussian ellipsoid to obtain a mixed color of the Gaussian ellipsoid. The fusion method may be weighted fusion using the weight coefficient.

[0204] Step S6: All Gaussian ellipsoids are subjected to differentiable rasterization processing based on their respective mixed colors to obtain a reconstructed rendering image of the object or scene. Specifically, the reconstructed rendering image of the object or scene, namely the target reconstructed image 129, can be obtained based on the differentiable rasterization 128 in the existing 3D Gaussian splash algorithm.

[0205] In the embodiment of the present application, step S1002, step S1003 and step S1004 may be executed in parallel or in a set order.

[0206] In the embodiment of the present application, each Gaussian ellipsoid can be expressed based on the aforementioned formula (2). Wherein, the 3D covariance matrix of the Gaussian ellipsoid can be expressed based on the aforementioned formula (3).

[0207] Differentiable rasterization is the process of converting a 3D scene into a 2D image. A three-dimensional Gaussian ellipsoid can be projected onto a two-dimensional plane through a two-dimensional covariance matrix. The covariance matrix representing the Gaussian ellipsoid is multiplied by a transformation matrix from the three-dimensional projection to the two-dimensional plane. The transformation matrix can be determined based on the above formula (4).

[0208] The image projected onto the two-dimensional plane is then divided into regular blocks of 16×16. During the pixel rendering process, all overlapping Gaussian spheres at that position are calculated through view changes and sorted by depth. Finally, the arranged Gaussian spheres are blended for transparency and the final mixed color value of the pixel is returned.

[0209] In the embodiment of the present application, in order to more accurately and quickly obtain the reflection direction vector of the Gaussian ellipsoid, according to a simple incident reflection illumination physical model, the reflection direction vector ω of each Gaussian ellipsoid is v It can be calculated based on the shortest axis direction vector n of the Gaussian ellipsoid and the observation direction vector ω0, as shown in the above formula (10).

[0210] In an optional implementation, in order to enhance the 3D reconstruction quality of objects and scenes with reflective properties based on a 3D Gaussian splash algorithm, make the 3D reconstruction training overhead and time controllable, and support real-time rendering, a weighted perceptron, an observation direction color processing model, and a reflection direction color processing model are obtained through joint training, and the joint training process includes:

[0211] S1101, using a 3D Gaussian splash algorithm to perform multiple generations of training based on the 3D Gaussian splash algorithm photometric loss function to complete Gaussian ellipsoid initialization and improve the spherical harmonic order of the Gaussian ellipsoid. The 3D Gaussian splash algorithm photometric loss function is shown in the above formula (11).

[0212] S1102, establish the network structure of the weighted perceptron, the observation direction color processing model and the reflection direction color processing model and initialize the network parameters of the network structure. Specifically, the weighted perceptron is a single-layer perceptron structure, and the observation direction color processing model and the reflection direction color processing model both use a three-layer perceptron network structure.

[0213] S1103, obtaining a training sample set, taking the total loss function (target loss function) as the target, and using the training sample set to jointly train the weighted perceptron, the observation direction color processing model, and the reflection direction color processing model, wherein each training sample includes a set of point clouds and real images of the object or scene, and in the joint training, the points in the point cloud of each training sample are initialized to a Gaussian ellipsoid according to the spherical harmonic order obtained by training in step 1. The point cloud of the training sample is a set of images of the corresponding object or scene obtained by transformation through the motion structure restoration algorithm (SFM), and the real image is a real three-dimensional image of the corresponding object or scene, which can be specifically an image collected by a camera or a collector.

[0214] The weighted perceptron obtains the weight coefficient of each Gaussian ellipsoid in the sample point cloud, the observation direction color processing model obtains the diffuse color component of the Gaussian ellipsoid corresponding to each point in the sample point cloud, and the reflection direction color processing model obtains the reflection highlight color component of the Gaussian ellipsoid corresponding to each point in the sample point cloud. The diffuse color component and the reflection highlight color component are weighted and fused using the weight coefficient to obtain the mixed color of the Gaussian ellipsoid. Then, all Gaussian ellipsoids of the sample are subjected to differentiable rasterization to obtain the reconstructed rendering image corresponding to the sample. Then, the total loss function is calculated based on the reconstructed rendering image and the real image, and the network parameters are updated based on the gradient descent method according to the total loss. When the total loss reaches the minimum or the preset number of training times is reached, the training is stopped. After that, the trained weighted perceptron, observation direction color processing model, and reflection direction color processing model can be tested using the pre-constructed test sample set.

[0215] Specifically, parameters such as peak signal-to-noise ratio PSNR, structural similarity SSIM, and learning image block similarity, that is, three indicators of preset evaluation parameters, can be used to evaluate the reconstruction effect.

[0216] In this embodiment, preferably, in order to optimize the Gaussian sphere properties and improve the reconstruction quality of reflective object or scene in joint training, the total loss function It is the weighted sum of the 3D Gaussian splash algorithm photometric loss, mixed photometric loss and smoothness loss of the reconstructed rendered image, as shown in the above formula (12).

[0217] Furthermore, the image reconstruction method for objects and scenes with reflective properties provided in the embodiments of the present application can be experimentally verified:

[0218] First, the data set is set. The image reconstruction method provided by the embodiment of the present application is comprehensively evaluated on 17 scenes in three different public data sets. The NeRF Synthetic data set contains objects with complex geometric shapes and realistic non-Lambertian materials, which is used to demonstrate the excellent ability of the image reconstruction method provided by the embodiment of the present application to reconstruct objects under weak reflection conditions. The Shinny Blender data set contains six different objects containing light reflection projection information, which is used to demonstrate the excellent reconstruction ability of the image reconstruction method provided by the embodiment of the present application when facing objects with reflective properties. At the same time, in order to prove the effectiveness of the image reconstruction method provided by the embodiment of the present application in modeling reflections in the real world, the Ref Real data set is also used. In addition, in order to further verify the performance of the image reconstruction method provided by the embodiment of the present application in general scenes, the garden scene of MIPNERF360 and the truck scene of the Tanks and Temples data set are selected for verification. In addition to the partially visualized qualitative analysis, the commonly used indicators (PSNR, SSIM, LPIPS, etc.) of the image reconstruction method provided by the embodiment of the present application in different scenes will also be reported, and quantitative comparisons will be made with the main comparison method 3D Gaussian splash algorithm and the current excellent method based on the 3D Gaussian splash algorithm.

[0219] Secondly, the experimental verification method. In order to accurately reconstruct scenes where there may be reflective attribute components, a hybrid model (composed of a weighted perceptron, an observation direction color processing model, and a reflection direction color processing model) is proposed, which can seamlessly combine the observation direction color processing model and the reflection direction color processing model with the original 3D Gaussian scheme. The observation direction color processing model is responsible for modeling the complex information of non-reflective color and diffuse color, and the reflection direction color processing model is responsible for modeling the reflective highlight component. The final result of the rendering is calculated by the color of the two components and a learnable weight coefficient. During the training process, the original 3D Gaussian will supervise the result and be responsible for initializing parameters and controlling geometric information. In the image reconstruction method provided in the embodiment of the present application, the object is first converted into a point through motion structure recovery and initialized as a Gaussian ellipsoid. The spherical harmonic eigenvector, normal (estimated using the shortest axis), encoded view and reflection direction of each Gaussian ellipsoid are input into the hybrid model. The hybrid model uses a multi-layer perceptron to learn view-dependent and reflection-dependent colors, and calculates a learnable reflection weight W to synthesize the final color. The image is rendered using a differentiable rasterization technique, and the output is supervised by the original 3D Gaussian rendered image.

[0220] Again, experimental settings. Pytorch is used to implement the image reconstruction method provided in the embodiment of the present application, and the number of training generations is set to 30000. In the first 3000 generations of training, since the order of the spherical harmonic function is still changing, only the original loss of 3D Gauss is used. Initialize and set λ1 to 1. After 3000 generations, enable all losses and set λ2 to 0.02. Set λ1=λ2=1 for the synthetic dataset and set λ1=λ2=0.5 for the real scene dataset. This can effectively control the number of generated Gaussian ellipsoids. In the image reconstruction method provided in the embodiment of the present application, the network for learning color is a three-layer MLP, and the weight network is a one-layer small MLP. The observation direction and reflection direction in the input are spherical harmonic encoded by tiny-cuda-nn. The training set and test set division strategy are the same as those for 3D Gaussian. If no clear dataset division is provided in the dataset, one image is extracted from every eight images and added to the test set.

[0221] Finally, experimental results and analysis. Quantitatively, the image reconstruction method provided by the embodiment of the present application achieves the best overall performance among the compared models. As can be seen from Tables 1 and 2 below, the image reconstruction method provided by the embodiment of the present application achieves the highest scores in PSNR (peak signal-to-noise ratio), SSIM (structural similarity) and LPIPS (perceptual loss) on the NeRF synthetic dataset and the ShinyBlender dataset.

[0222] Table 1

[0223]

[0224] Figures 4 to 8 The visualization of the rendering results of the image reconstruction method provided by the embodiment of the present application in five different scenes in the synthetic dataset is shown. The improvement in reconstruction quality is particularly obvious in metallic highlights, such as the cymbals and the surface of the toaster in the drum scene. Reflection projections can be seen on the car hood, car windows, and the surface of the liquid in the coffee cup. In addition, detailed diffuse lighting is also well captured, as shown in the Lego scene.

[0225] Table 1 is a comparison table of the new perspective synthesis effects of the NeRF Synthetic dataset, as shown in Table 1 and Figure 4 As shown, compared with the reconstructed image 131 generated by the 3D-GS algorithm, the reconstructed image 132 generated by the Scaffold-GS algorithm, the image 133 generated by the VDGS algorithm, and the real image 135, the target reconstructed image 134 generated by the image reconstruction method provided in the embodiment of the present application has a particularly obvious improvement in the reconstruction quality in terms of the metal highlights at the framed position, that is, the surface of the cymbal in the drum scene. In addition, as shown in Table 1 and Figure 5As shown, compared with the reconstructed image 141 generated by the 3D-GS algorithm, the reconstructed image 142 generated by the Scaffold-GS algorithm, the image 143 generated by the VDGS, and the real image 145, the target reconstructed image 144 generated by the image reconstruction method provided in the embodiment of the present application is well captured in terms of the diffuse highlights of the plastic material at the framed position, that is, the diffuse highlights in the Lego scene, and the improvement in reconstruction quality is also very obvious.

[0226] Table 2

[0227]

[0228] Table 2 is a comparison table of the new perspective synthesis effect of the Shiny Blender dataset, as shown in Table 2 and Figure 6 As shown, compared with the image 151 reconstructed by the 3D-GS algorithm and the real image 153, the target reconstructed image 152 generated by the image reconstruction method provided by the embodiment of the present application has a particularly obvious improvement in the reconstruction quality in terms of the metal highlights at the framed position, that is, the metal reflective position of the toaster. As shown in Table 2 and Figure 7 As shown, compared with the image 161 reconstructed by the 3D-GS algorithm and the real image 163, the target reconstructed image 162 generated by the image reconstruction method provided in the embodiment of the present application can be seen with a reflection projection on the surface of the car hood at the selected position, and the reconstruction quality is better. Figure 8 As shown, compared with the image 171 reconstructed by the 3D-GS algorithm and the real image 173, the target reconstructed image 172 generated by the image reconstruction method provided in the embodiment of the present application has a reflected projection on the liquid surface of the coffee cup at the framed position, and the quality of the reconstructed target reconstructed image is better.

[0229] Table 3

[0230]

[0231] Table 3 is a comparison table of new perspective synthesis effects of real data sets. As shown in Table 3, compared with the 3D-GS algorithm, the target reconstructed image generated by this application has better performance in evaluation parameters such as peak signal-to-noise ratio, structural similarity and perceptual loss. It should be noted that the 3D-GS algorithm directly performs corresponding rasterization processing on the Gaussian ellipsoid to generate the corresponding reconstructed image.

[0232] When reconstructing real-world datasets, Table 3 shows that the image reconstruction method provided by the embodiment of the present application outperforms the main baseline, namely the original 3D Gaussian, and successfully models and decomposes the reflected highlights in the real-world scenes.

[0233] It can be seen that the hybrid model proposed in the embodiment of the present application decomposes the scene color expression. The reflection component in the model expresses the highlight reflection color by simply modeling the light reflection physical model. The camera direction component expresses the diffuse reflection color and is combined with the original 3D Gaussian. The color of the diffuse reflection component is controlled by a smooth loss function, and the rendering effects of the two components are visualized, which improves the interpretability of the scene color decomposition. It can better represent 3D objects with metallic luster, light reflection, and shadows. The experimental results on the public data set show that the image reconstruction method provided by the embodiment of the present application has superior performance.

[0234] In an embodiment of the present application, the color of the object or scene is decomposed, and a reflection direction color processing model is built to supplement the color expression of the 3D Gaussian in the reflection view appearance, achieving a certain effect. It was observed in the experiment that it is difficult to represent the diffuse reflection and blurred reflection in the scene with only the reflection direction color processing model. Therefore, the observation direction color processing model is added to enhance the correlation between the Gaussian ball attribute features and the observation direction, and improve the model's ability to represent diffuse reflection and capture details. The color components learned by the two models are combined using a learnable weight coefficient and organically combined with the original 3D Gaussian solution to achieve high reconstruction quality in scenes with reflection properties. Experiments have found that when facing general data sets and complex scenes, the model provided by this application also maintains a high level, and the method is generalizable.

[0235] Based on the above embodiments, the present application also provides an image reconstruction device. Fig. 9 A schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the image reconstruction device 18 includes:

[0236] A first generating module 181 is used to generate a Gaussian ellipsoid set based on a point cloud corresponding to the image to be reconstructed, wherein the Gaussian ellipsoid set includes a plurality of Gaussian ellipsoids, and each Gaussian ellipsoid has a corresponding relationship with a point in the point cloud;

[0237] A first determination module 182, configured to determine a weight coefficient corresponding to the Gaussian ellipsoid according to geometric parameters of the Gaussian ellipsoid;

[0238] A second determination module 183 is used to determine the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector;

[0239] A third determination module 184 is used to determine the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector;

[0240] The second generation module 185 is used to fuse the weight coefficient, diffuse color component and reflective highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set to generate a target reconstructed image.

[0241] In some embodiments, the first determination module 182 is also used to concatenate the shortest axis direction vector corresponding to the Gaussian ellipsoid and the eigenvector of the spherical harmonic function to obtain a characteristic matrix corresponding to the Gaussian ellipsoid; and input the characteristic matrix corresponding to the Gaussian ellipsoid into the target weight perception model to obtain a weight coefficient corresponding to the Gaussian ellipsoid.

[0242] In some embodiments, the second generation module 185 is also used to determine the diffuse reflection weight coefficient corresponding to the diffuse reflection color component and the reflection highlight weight coefficient corresponding to the reflection highlight color component based on the weight coefficient corresponding to the Gaussian ellipsoid; based on the diffuse reflection weight coefficient and the reflection highlight weight coefficient, weighted calculation is performed on the diffuse reflection color component corresponding to the Gaussian ellipsoid and the reflection highlight color component corresponding to the Gaussian ellipsoid to determine the mixed color component corresponding to the Gaussian ellipsoid; differentiable rasterization is performed on the mixed color components corresponding to multiple Gaussian ellipsoids in the Gaussian ellipsoid set to generate a target reconstructed image.

[0243] In some embodiments, the second generation module 185 is further used to project the Gaussian ellipsoid set into an initial reconstructed image based on the transformation matrix; render the initial reconstructed image based on the mixed color components corresponding to each of the multiple Gaussian ellipsoids to generate a target reconstructed image.

[0244] In some embodiments, the second determination module 183 is further used to concatenate the feature matrix and the observation direction vector to obtain an observation direction matrix; and input the observation direction matrix into the target observation direction processing model to obtain the diffuse reflection color component corresponding to the Gaussian ellipsoid.

[0245] In some embodiments, the third determination module 184 is further used to concatenate the feature matrix and the reflection direction vector to obtain a reflection direction matrix; input the reflection direction matrix into the target reflection direction processing model to obtain the reflection highlight color component corresponding to the Gaussian ellipsoid.

[0246] In some embodiments, the second determination module 183 is also used to determine the observation direction of the Gaussian ellipsoid based on the camera parameters of the image to be reconstructed and the coordinates of the center of the Gaussian ellipsoid; encode the observation direction to determine the observation direction vector of the Gaussian ellipsoid; and determine the reflection direction vector corresponding to the Gaussian ellipsoid based on the shortest axis direction vector and the observation direction vector corresponding to the Gaussian ellipsoid.

[0247] In some embodiments, the second generation module 185 is also used to construct an initial weight perception model, an initial observation direction processing model, and an initial reflection direction processing model; construct a training sample set, the training sample set includes multiple training samples, each training sample includes a synthetic Gaussian ellipsoid set and a corresponding real three-dimensional image; based on the target loss function, the initial weight perception model, the initial observation direction processing model, and the initial reflection direction processing model are jointly trained using the training sample set to obtain a target weight perception model, a target observation direction processing model, and a target reflection direction processing model.

[0248] In some embodiments, the second generation module 185 is also used to construct an initial loss function based on a Gaussian loss function, a mixed photometric loss function, a smoothing loss function and preset coefficients; and adjust the preset coefficients based on preset evaluation parameters to determine the target loss function, wherein the preset evaluation parameters are used to evaluate the reconstruction effect of the target reconstructed image.

[0249] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0250] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0251] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the computer program.

[0252] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. The computer-readable storage medium can be transient or non-transient.

[0253] The present application also provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0254] It should be noted that Fig.10 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the hardware entity of the electronic device 19 includes: a processor 1901, a communication interface 1902 and a memory 1903, wherein:

[0255] The processor 1901 generally controls the overall operation of the electronic device 19 .

[0256] The communication interface 1902 may enable the electronic device to communicate with other terminals or servers via a network.

[0257] The memory 1903 is configured to store instructions and applications executable by the processor 1901, and can also cache data to be processed or processed by the processor 1901 and each module in the electronic device 19 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data can be transmitted between the processor 1901, the communication interface 1902, and the memory 1903 through the bus 1904.

[0258] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0259] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0260] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0261] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in each embodiment of the present application, each functional unit may be fully integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0262] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. An image reconstruction method, characterized in that: The method comprises: Generate a Gaussian ellipsoid set based on the point cloud corresponding to the image to be reconstructed, wherein the Gaussian ellipsoid set includes a plurality of Gaussian ellipsoids, and each of the Gaussian ellipsoids has a corresponding relationship with a point in the point cloud; Determine a weight coefficient corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid; Determine the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector; Determine the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector; The weight coefficient, diffuse color component and reflective highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set are fused respectively to generate a target reconstructed image.

2. The image reconstruction method according to claim 1, characterized in that: The step of determining a weight coefficient corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid comprises: The shortest axis direction vector corresponding to the Gaussian ellipsoid and the eigenvector of the spherical harmonic function are concatenated to obtain a characteristic matrix corresponding to the Gaussian ellipsoid; The feature matrix corresponding to the Gaussian ellipsoid is input into the target weight perception model to obtain the weight coefficient corresponding to the Gaussian ellipsoid.

3. The image reconstruction method according to claim 2, characterized in that: The step of fusing the weight coefficient, diffuse color component and reflective highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set to generate a target reconstructed image comprises: Based on the weight coefficient corresponding to the Gaussian ellipsoid, determine the diffuse reflection weight coefficient corresponding to the diffuse reflection color component and the reflection highlight weight coefficient corresponding to the reflection highlight color component; Based on the diffuse reflection weight coefficient and the reflection highlight weight coefficient, weighted calculation is performed on the diffuse reflection color component corresponding to the Gaussian ellipsoid and the reflection highlight color component corresponding to the Gaussian ellipsoid to determine the mixed color component corresponding to the Gaussian ellipsoid; The mixed color components corresponding to each of the plurality of Gaussian ellipsoids in the Gaussian ellipsoid set are subjected to differentiable rasterization processing to generate the target reconstructed image.

4. The image reconstruction method according to claim 3, characterized in that: The performing differentiable rasterization processing on the mixed color components corresponding to the plurality of Gaussian ellipsoids in the Gaussian ellipsoid set to generate the target reconstructed image comprises: Based on the transformation matrix, projecting the Gaussian ellipsoid set into an initial reconstructed image; The initial reconstructed image is rendered based on the mixed color components corresponding to each of the multiple Gaussian ellipsoids to generate the target reconstructed image.

5. The image reconstruction method according to claim 2, characterized in that: The step of determining the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector comprises: Concatenate the feature matrix and the observation direction vector to obtain an observation direction matrix; The viewing direction matrix is ​​input into the target viewing direction processing model to obtain the diffuse reflection color component corresponding to the Gaussian ellipsoid.

6. The image reconstruction method according to claim 2, characterized in that: The step of determining the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector comprises: Concatenate the characteristic matrix and the reflection direction vector to obtain a reflection direction matrix; The reflection direction matrix is ​​input into the target reflection direction processing model to obtain the reflection highlight color component corresponding to the Gaussian ellipsoid.

7. The image reconstruction method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determining the observation direction of the Gaussian ellipsoid according to the camera parameters of the image to be reconstructed and the coordinates of the center of the Gaussian ellipsoid; Encoding the observation direction to determine the observation direction vector of the Gaussian ellipsoid; The reflection direction vector corresponding to the Gaussian ellipsoid is determined according to the shortest axis direction vector corresponding to the Gaussian ellipsoid and the observation direction vector.

8. The image reconstruction method according to any one of claims 1 to 6, characterized in that: The method further comprises: Constructing an initial weight perception model, an initial observation direction processing model, and an initial reflection direction processing model; Constructing a training sample set, wherein the training sample set includes a plurality of training samples, each of which includes a synthetic Gaussian ellipsoid set and a corresponding real three-dimensional image; Based on the target loss function, the initial weight perception model, the initial observation direction processing model and the initial reflection direction processing model are jointly trained using the training sample set to obtain a target weight perception model, a target observation direction processing model and a target reflection direction processing model.

9. The image reconstruction method according to claim 8, characterized in that: The method further comprises: Construct an initial loss function based on Gaussian loss function, mixed photometric loss function, smoothing loss function and preset coefficients; The preset coefficients are adjusted based on preset evaluation parameters to determine a target loss function, wherein the preset evaluation parameters are used to evaluate a reconstruction effect of the target reconstructed image.

10. An image reconstruction device, characterized in that: The image reconstruction device comprises: A first generating module is used to generate a Gaussian ellipsoid set based on a point cloud corresponding to the image to be reconstructed, wherein the Gaussian ellipsoid set includes a plurality of Gaussian ellipsoids, and each of the Gaussian ellipsoids has a corresponding relationship with a point in the point cloud; A first determination module, used for determining a weight coefficient corresponding to the Gaussian ellipsoid according to geometric parameters of the Gaussian ellipsoid; A second determination module is used to determine the diffuse reflection color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the observation direction vector; A third determination module is used to determine the reflection highlight color component corresponding to the Gaussian ellipsoid according to the geometric parameters of the Gaussian ellipsoid and the reflection direction vector; The second generation module is used to fuse the weight coefficient, diffuse color component and reflection highlight color component corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid set to generate a target reconstructed image.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method described in any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images