Object Gaussian inverse rendering method based on global-local surface cooperative perception

Abstract

The embodiments of the present disclosure provide a method for Gaussian inverse rendering of objects based on global-local surface collaborative perception, which relates to the field of graphics inverse rendering. The method includes performing ray casting and sampling to obtain sampling points; using a geometric network and a color network to calculate the neural radiation distance value and color value of the sampling points to obtain a global geometric representation; initializing the Gaussian point cloud position according to the neural radiation distance value; selecting a local control point from the initialized Gaussian point cloud, determining the local sampling point with the local control point as the center and constructing a local distance function; using the local distance function and the neural radiation distance function to calculate the local distance value and the global neural radiation distance value of the local sampling point according to the local sampling point; based on the global geometric representation, using these two distance values ​​as constraints to reconstruct geometric details; adding attribute values ​​to the Gaussian point cloud to perform material and lighting modeling, and obtaining material attributes and lighting conditions. In this way, the roughness of the object surface is alleviated and the ability to capture details is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of graphics inverse rendering, and in particular to a Gaussian inverse rendering method for an object based on global-local surface collaborative perception. Background Art

[0002] Object inverse rendering is an important research direction based on computer vision and graphics, which aims to recover the underlying physical properties of the scene from given image or video data, such as light conditions, object shape, and surface material. Inverse rendering can be regarded as the reverse process of the current graphics rendering pipeline. This technology is widely used in games, filmmaking, virtual reality (VR), and augmented reality (AR). Existing object inverse rendering methods mainly include three categories. The first category is the object inverse rendering method based on display geometry representation. This type of method uses triangles as geometric proxies of objects and gradually optimizes the decoupled material attribute maps by gradient return. Although this type of method is compatible with the existing traditional rendering pipeline, it is limited by the number of triangles and cannot handle the material texture decoupling task at the object details. The second category is the object inverse rendering method based on implicit geometry representation. This method models the scene by simulating how ambient light interacts with the neural radiation field with material attributes, and estimates the ambient light illumination and material attribute parameters in the differentiable rendering process. However, this type of method is limited by the computational complexity of the reconstruction of the neural radiation field. The third category is the object inverse rendering method based on 3D Gaussian point cloud. As an explicit expression, 3D Gaussian point cloud has both high rendering quality and rendering efficiency. However, the existing 3D Gaussian point cloud method has the problem of perspective overfitting. The position and shape of the Gaussian ellipsoid in the reconstructed Gaussian scene will overfit to certain specific shooting perspectives, resulting in a rough surface of the object, which brings great challenges to the inverse rendering task. Summary of the invention

[0003] The present disclosure provides a method, device, equipment and storage medium for Gaussian inverse rendering of an object based on global-local surface collaborative perception.

[0004] According to a first aspect of the present disclosure, a method for inverse Gaussian rendering of an object based on global-local surface cooperative perception is provided. The method comprises:

[0005] Perform ray casting on each pixel in the multi-view image, perform sampling on the ray, and obtain each sampling point;

[0006] The geometric network and color network are used to calculate the neural radiation distance function value and color value of each sampling point respectively, and obtain the global geometric representation;

[0007] Initialize the Gaussian point cloud position according to the neural radiation distance function value of each sampling point, and obtain the initialized Gaussian point cloud;

[0008] Select local control Gaussian points from the initialized Gaussian point cloud, and determine local sampling points with the local control Gaussian points as the center;

[0009] Constructing a local distance function according to local control Gaussian points and local sampling points;

[0010] According to the local sampling points, the local distance function and the neural radiation distance function are used to respectively calculate the local distance function value and the global neural radiation distance function value of the local sampling points with respect to the local control Gaussian point;

[0011] According to the global geometric representation, the geometric details are reconstructed using the local distance function value and the global neural radiation distance function value as constraints;

[0012] After reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and the visibility of each Gaussian point in the Gaussian point cloud is calculated.

[0013] In some implementations of the first aspect, using a geometric network and a color network to respectively calculate a neural radiation distance function value and a color value of each sampling point, and obtaining a global geometric representation, includes:

[0014] Encoding each sampling point to obtain a spatial position representation of each sampling point;

[0015] The spatial position representation of each sampling point is input into the geometric network and the color network respectively, the neural radiation distance function value of each sampling point is calculated by the geometric network, and the color value of each sampling point is calculated by the color network;

[0016] The global geometric representation is obtained based on the neural radiation distance function value and color value of each sampling point.

[0017] In some implementations of the first aspect, initializing the Gaussian point cloud position according to the neural radiation distance function value of each sampling point includes:

[0018] The farthest point sampling method is used to downsample the point set corresponding to the neural radiation distance function value of each sampling point in the preset interval, and the sampling points obtained by downsampling are used as Gaussian points after the Gaussian point cloud position is initialized.

[0019] In some implementable manners of the first aspect, selecting a local control Gaussian point from an initialized Gaussian point cloud includes:

[0020] From the initialized Gaussian point cloud, a Gaussian point whose direct interpolation between Gaussian rendering depth and camera space depth is within a preset threshold is selected as a local control Gaussian point.

[0021] In some implementations of the first aspect, determining a local sampling point with the local control Gaussian point as the center includes:

[0022] With the local control Gaussian point as the center, random Gaussian noise is added along the shortest axis direction of the Gaussian point, and the local control Gaussian point and the added random Gaussian noise are used as local sampling points.

[0023] In some implementations of the first aspect, constructing a local distance function according to local control Gaussian points and local sampling points includes:

[0024] Construct a local density function according to local control Gaussian points and local sampling points;

[0025] According to the local density function and local sampling points, a local distance function is constructed.

[0026] In some implementations of the first aspect, the local density function is:

[0027] ;

[0028] in, represents the local density function, represents the local sampling point, , , They represent the local control Gaussian points alpha blending coefficients, spatial positions and covariances, Represents a transpose operation;

[0029] The local distance function is:

[0030] ;

[0031] in, represents the local distance function, is the minimum scaling factor for local control of Gaussian points.

[0032] In some implementations of the first aspect, the method further includes:

[0033] Input material properties, lighting conditions and visibility of each Gaussian point into the microsurface model to obtain the rendering color of each Gaussian point;

[0034] Integrate the rendered colors of each Gaussian point to obtain a rendered image;

[0035] The real image corresponding to the rendered image is used as a constraint to optimize the position of the Gaussian point cloud and the attribute values ​​of the Gaussian point cloud.

[0036] In some implementations of the first aspect, the method further includes:

[0037] According to the neural radiation distance function value and color value of each sampling point, a volume rendering image is generated using a volume rendering method;

[0038] The geometry network and color network are optimized using the real image corresponding to the volume rendered image as a constraint.

[0039] According to a second aspect of the present disclosure, a Gaussian inverse rendering device for an object based on global-local surface collaborative perception is provided. The device comprises:

[0040] The sampling point acquisition module is used to project rays to each pixel in the multi-view image, perform sampling on the rays, and obtain each sampling point;

[0041] A global geometric representation acquisition module is used to use a geometric network and a color network to respectively calculate the neural radiation distance function value and the color value of each sampling point and obtain a global geometric representation;

[0042] The Gaussian point cloud position initialization module is used to initialize the Gaussian point cloud position according to the neural radiation distance function value of each sampling point and obtain the initialized Gaussian point cloud;

[0043] A local sampling point determination module is used to select a local control Gaussian point from the initialized Gaussian point cloud and determine the local sampling point with the local control Gaussian point as the center;

[0044] A local distance function construction module is used to construct a local distance function according to local control Gaussian points and local sampling points;

[0045] A function value calculation module, used to calculate the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point according to the local sampling point by using the local distance function and the neural radiation distance function;

[0046] A geometric detail reconstruction module is used to reconstruct geometric details according to the global geometric representation, using the local distance function value and the global neural radiation distance function value as constraints;

[0047] The material and lighting reconstruction module is used to add attribute values ​​to the Gaussian point cloud for material and lighting modeling after reconstruction, obtain material attributes and lighting conditions, and calculate the visibility of each Gaussian point in the Gaussian point cloud.

[0048] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0049] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.

[0050] In the present disclosure, a ray is projected on each pixel in a multi-view image, and sampling is performed on the ray to obtain each sampling point; a neural radiation distance function value of each sampling point is calculated using a geometric network, and a color value of each sampling point is calculated using a color network; a global geometric representation is obtained based on the neural radiation distance function value and the color value of each sampling point; a Gaussian point cloud position is initialized based on the neural radiation distance function value of each sampling point to obtain an initialized Gaussian point cloud; a local control Gaussian point is selected from the initialized Gaussian point cloud, and a local sampling point is determined with the local control Gaussian point as the center; a local distance function is constructed based on the local control Gaussian point and the local sampling point; based on the local sampling point, the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point are calculated using the local distance function and the neural radiation distance function respectively; based on the global geometric representation, the geometric details of the Gaussian point cloud are reconstructed using the local distance function value and the global neural radiation distance function value as constraints; after the reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and the visibility of each Gaussian point in the Gaussian point cloud is calculated. In this way, the problem of rough surface of reconstructed objects caused by overfitting of viewing angle can be alleviated, the ability to capture complex surface details can be improved, accurate geometric reconstruction can be achieved, the decoupling speed can be accelerated, strong support can be provided for finer material decomposition, and processing efficiency and inverse rendering quality can be improved.

[0051] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0053] Figure 1 A flowchart of a method for Gaussian inverse rendering of an object based on global-local surface collaborative perception provided by an embodiment of the present disclosure is shown;

[0054] Figure 2 A structural diagram of a Gaussian inverse rendering device for an object based on global-local surface collaborative perception provided by an embodiment of the present disclosure is shown;

[0055] Figure 3A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0057] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0058] In view of the problems arising in the background technology, an embodiment of the present disclosure provides a Gaussian inverse rendering method for an object based on global-local surface collaborative perception. Specifically, a ray is projected on each pixel in the multi-view image, and sampling is performed on the ray to obtain each sampling point; the neural radiation distance function value of each sampling point is calculated using a geometric network, and the color value of each sampling point is calculated using a color network; a global geometric representation is obtained based on the neural radiation distance function value and the color value of each sampling point; the position of the Gaussian point cloud is initialized based on the neural radiation distance function value of each sampling point to obtain an initialized Gaussian point cloud; local control Gaussian points are selected from the initialized Gaussian point cloud, and local sampling points are determined with the local control Gaussian points as the center; a local distance function is constructed based on the local control Gaussian points and the local sampling points; based on the local sampling points, the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point are calculated using the local distance function and the neural radiation distance function respectively; based on the global geometric representation, the geometric details of the Gaussian point cloud are reconstructed using the local distance function value and the global neural radiation distance function value as constraints; after the reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and the visibility of each Gaussian point in the Gaussian point cloud is calculated. In this way, the problem of rough surface of reconstructed objects caused by overfitting of viewing angle can be alleviated, the ability to capture complex surface details can be improved, accurate geometric reconstruction can be achieved, the decoupling speed can be accelerated, strong support can be provided for finer material decomposition, and processing efficiency and inverse rendering quality can be improved.

[0059] The object Gaussian inverse rendering method based on global-local surface collaborative perception provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings through specific embodiments.

[0060] Figure 1 A flowchart of a method for Gaussian inverse rendering of an object based on global-local surface collaborative perception provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps:

[0061] S110, performing ray casting on each pixel in the multi-view image, sampling on the ray, and obtaining each sampling point.

[0062] In some embodiments, each pixel in the multi-view image projects light from the camera center point to the imaging plane, performs equidistant point sampling on the light, and obtains sampling points.

[0063] S120, using the geometric network and the color network to respectively calculate the neural radiation distance function value and the color value of each sampling point, and obtain the global geometric representation.

[0064] In some embodiments, the neural radiation distance function value and the color value of each sampling point are calculated using the geometric network and the color network, and a global geometric representation is obtained, including:

[0065] Encoding each sampling point to obtain a spatial position representation of each sampling point;

[0066] The spatial position representation of each sampling point is input into the geometric network and the color network respectively, the neural radiation distance function value of each sampling point is calculated by the geometric network, and the color value of each sampling point is calculated by the color network;

[0067] The global geometric representation is obtained based on the neural radiation distance function value and color value of each sampling point.

[0068] In some embodiments, encoding each sampling point to obtain a spatial position representation of each sampling point includes:

[0069] A multi-resolution hash encoder is used to encode each sampling point to obtain the spatial position representation of each sampling point.

[0070] In some embodiments, both the geometry network and the color network are composed of shallow MLP networks.

[0071] In some embodiments, the method 100 further includes:

[0072] According to the neural radiation distance function value and color value of each sampling point, a volume rendering image is generated using a volume rendering method;

[0073] Using the real image corresponding to the volume rendered image as a constraint, the geometry network and color network are optimized;

[0074] Furthermore, the color value corresponding to each pixel of the volume rendering image The calculation method is as follows:

[0075] ;

[0076] in, represents the camera center, Represents the reverse direction of the light projected from the camera center point, Represents the pixel points of the volume rendered image, represent transmittance, volume density and x The color value of the sampling point.

[0077] S130, initializing the Gaussian point cloud position according to the neural radiation distance function value of each sampling point, and obtaining the initialized Gaussian point cloud.

[0078] In some embodiments, the Gaussian point cloud position is initialized according to the neural radiation distance function value of each sampling point, including:

[0079] Use the farthest point sampling method to downsample the point set corresponding to the neural radiation distance function value of each sampling point in the preset interval, and use the downsampled sampling points as the Gaussian points after the Gaussian point cloud position is initialized;

[0080] Furthermore, the value of the nerve radiation distance function within the preset interval is close to 0.

[0081] S140, selecting a local control Gaussian point from the initialized Gaussian point cloud, and determining a local sampling point with the local control Gaussian point as the center.

[0082] In some embodiments, selecting a local control Gaussian point from an initialized Gaussian point cloud includes:

[0083] From the initialized Gaussian point cloud, select Gaussian points whose direct interpolation of Gaussian rendering depth and camera space depth is within a preset threshold as local control Gaussian points;

[0084] Furthermore, the direct interpolation within the preset threshold is smaller.

[0085] In some embodiments, determining a local sampling point with the local control Gaussian point as the center includes:

[0086] Taking the local control Gaussian point as the center, random Gaussian noise is added along the shortest axis direction of the Gaussian point, and the local control Gaussian point and the added random Gaussian noise are used as local sampling points;

[0087] For example, if the local control Gaussian point The spatial position of , the random Gaussian noise added is , then the local sampling point Expressed as .

[0088] S150, constructing a local distance function according to the local control Gaussian points and the local sampling points.

[0089] In some embodiments, in order to effectively control the spatial position of the Gaussian point cloud during the optimization process, so that the Gaussian point cloud can be closer to the surface of the object and reconstruct the surface of the fine object, a local density function and a local distance function are constructed.

[0090] In some embodiments, a local distance function is constructed according to the local control Gaussian points and the local sampling points, including:

[0091] Construct a local density function according to local control Gaussian points and local sampling points;

[0092] According to the local density function and local sampling points, a local distance function is constructed.

[0093] In some embodiments, the local density function is:

[0094] ;

[0095] in, represents the local density function, represents the local sampling point, , , They represent the local control Gaussian points alpha blending coefficients, spatial positions and covariances, Represents a transpose operation;

[0096] The local distance function is:

[0097] ;

[0098] in, represents the local distance function, is the minimum scaling factor for local control of Gaussian points.

[0099] S160, according to the local sampling points, using the local distance function and the neural radiation distance function, respectively calculate the local distance function value and the global neural radiation distance function value of the local sampling points with respect to the local control Gaussian point.

[0100] S170, reconstructing geometric details according to the global geometric representation, using the local distance function value and the global neural radiation distance function value as constraints.

[0101] In some embodiments, the reconstructed geometric details It is expressed as follows:

[0102] ;

[0103] in, Represents local sampling points The global neural radiation distance function value at is the set of all local sampling points, for The elements in .

[0104] S180, after the reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and visibility of each Gaussian point in the Gaussian point cloud is calculated.

[0105] In some embodiments, the property values ​​include intrinsic color, roughness, surface normal, metalness, and indirect lighting.

[0106] In some embodiments, a ray tracing method is used to calculate the visibility of each Gaussian point in the Gaussian point cloud.

[0107] In some embodiments, illumination modeling includes direct illumination and indirect illumination modeling, and direct illumination is modeled using a spherical Gaussian function, which can compress optimizable parameters and facilitate convergence;

[0108] Furthermore, direct light It can be expressed as:

[0109] ;

[0110] in, is a mixed spherical Gaussian function, For the lobe The optimizable parameters are is the total number of lobes, is the incident direction of the light.

[0111] In some embodiments, the method 100 further includes:

[0112] Input material properties, lighting conditions and visibility of each Gaussian point into the microsurface model to obtain the rendering color of each Gaussian point;

[0113] Integrate the rendered colors of each Gaussian point to obtain a rendered image;

[0114] Using the real image corresponding to the rendered image as a constraint, optimize the position of the Gaussian point cloud and the attribute values ​​of the Gaussian point cloud;

[0115] Furthermore, the following physical rendering formula is used to obtain the rendered image:

[0116] ;

[0117] in, Indicates the color corresponding to the Gaussian point obtained using the physical rendering formula, represents a Gaussian point, is the normal direction, For Gaussian points The incident radiance received in the direction, is the emission direction, The surface material.

[0118] In some embodiments, the real image corresponding to the volume rendering image (i.e., the real image corresponding to the rendered image) is used as a constraint to determine the main constraint function, and the secondary constraint functions such as material smoothness loss constraint and geometric constraint are combined to optimize the Gaussian point cloud position and the attribute value of the Gaussian point cloud; wherein the real image corresponding to the volume rendering image and the real image corresponding to the rendered image are both the multi-view images in step S110;

[0119] Furthermore, the main constraint function It is expressed as follows:

[0120] ;

[0121] in, is the color corresponding to the sampling point in the volume rendering method, is the true value color corresponding to the multi-view image.

[0122] According to an embodiment of the present disclosure, a ray is projected on each pixel in a multi-view image, and sampling is performed on the ray to obtain each sampling point; a neural radiation distance function value of each sampling point is calculated using a geometric network, and a color value of each sampling point is calculated using a color network; a global geometric representation is obtained based on the neural radiation distance function value and the color value of each sampling point; a Gaussian point cloud position is initialized based on the neural radiation distance function value of each sampling point to obtain an initialized Gaussian point cloud; a local control Gaussian point is selected from the initialized Gaussian point cloud, and a local sampling point is determined with the local control Gaussian point as the center; a local distance function is constructed based on the local control Gaussian point and the local sampling point; based on the local sampling point, the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point are calculated respectively using the local distance function and the neural radiation distance function; based on the global geometric representation, the geometric details of the Gaussian point cloud are reconstructed using the local distance function value and the global neural radiation distance function value as constraints; after the reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and the visibility of each Gaussian point in the Gaussian point cloud is calculated. In this way, the problem of rough surface of reconstructed objects caused by overfitting of viewing angle can be alleviated, the ability to capture complex surface details can be improved, accurate geometric reconstruction can be achieved, the decoupling speed can be accelerated, strong support can be provided for finer material decomposition, and processing efficiency and inverse rendering quality can be improved.

[0123] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0124] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0125] Figure 2 The structure diagram of a Gaussian inverse rendering device for an object based on global-local surface collaborative perception provided by an embodiment of the present disclosure is shown. The device 200 includes:

[0126] The sampling point acquisition module 210 is used to perform ray casting on each pixel in the multi-view image, perform sampling on the ray, and obtain each sampling point.

[0127] The global geometric representation acquisition module 220 is used to calculate the neural radiation distance function value and the color value of each sampling point by using the geometric network and the color network, and obtain the global geometric representation.

[0128] In some embodiments, the global geometric representation acquisition module 220 is specifically used to:

[0129] The geometric network and color network are used to calculate the neural radiation distance function value and color value of each sampling point respectively, and obtain the global geometric representation, including:

[0130] Encoding each sampling point to obtain a spatial position representation of each sampling point;

[0131] The spatial position representation of each sampling point is input into the geometric network and the color network respectively, the neural radiation distance function value of each sampling point is calculated by the geometric network, and the color value of each sampling point is calculated by the color network;

[0132] The global geometric representation is obtained based on the neural radiation distance function value and color value of each sampling point.

[0133] In some embodiments, the apparatus 200 is further configured to:

[0134] According to the neural radiation distance function value and color value of each sampling point, a volume rendering image is generated using a volume rendering method;

[0135] The geometry network and color network are optimized using the real image corresponding to the volume rendered image as a constraint.

[0136] The Gaussian point cloud position initialization module 230 is used to initialize the Gaussian point cloud position according to the neural radiation distance function value of each sampling point, and obtain the initialized Gaussian point cloud.

[0137] In some embodiments, the Gaussian point cloud position initialization module 230 is specifically used to:

[0138] According to the neural radiation distance function value of each sampling point, the Gaussian point cloud position is initialized, including:

[0139] The farthest point sampling method is used to downsample the point set corresponding to the neural radiation distance function value of each sampling point in the preset interval, and the sampling points obtained by downsampling are used as Gaussian points after the Gaussian point cloud position is initialized.

[0140] The local sampling point determination module 240 is used to select local control Gaussian points from the initialized Gaussian point cloud and determine local sampling points with the local control Gaussian points as the center.

[0141] In some embodiments, the local sampling point determination module 240 is specifically used to:

[0142] Select local control Gaussian points from the initialized Gaussian point cloud, including:

[0143] From the initialized Gaussian point cloud, a Gaussian point whose direct interpolation between Gaussian rendering depth and camera space depth is within a preset threshold is selected as a local control Gaussian point.

[0144] In some embodiments, the local sampling point determination module 240 is further configured to:

[0145] Taking the local control Gaussian point as the center, determine the local sampling points, including:

[0146] With the local control Gaussian point as the center, random Gaussian noise is added along the shortest axis direction of the Gaussian point, and the local control Gaussian point and the added random Gaussian noise are used as local sampling points.

[0147] The local distance function construction module 250 is used to construct a local distance function according to the local control Gaussian points and the local sampling points.

[0148] In some embodiments, the local distance function construction module 250 is specifically used to:

[0149] According to the local control Gaussian points and local sampling points, a local distance function is constructed, including:

[0150] Construct a local density function according to local control Gaussian points and local sampling points;

[0151] According to the local density function and local sampling points, a local distance function is constructed.

[0152] In some embodiments, the local distance function construction module 250 is further configured to:

[0153] The local density function is:

[0154] ;

[0155] in, represents the local density function, represents the local sampling point, , , They represent the local control Gaussian points alpha blending coefficients, spatial positions and covariances, Represents a transpose operation;

[0156] The local distance function is:

[0157] ;

[0158] in, represents the local distance function, is the minimum scaling factor for local control of Gaussian points.

[0159] The function value calculation module 260 is used to calculate the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point according to the local sampling point using the local distance function and the neural radiation distance function.

[0160] The geometric detail reconstruction module 270 is used to reconstruct geometric details according to the global geometric representation, using the local distance function value and the global neural radiation distance function value as constraints.

[0161] The material and lighting reconstruction module 280 is used to add attribute values ​​to the Gaussian point cloud to perform material and lighting modeling, obtain material attributes and lighting conditions, and calculate the visibility of each Gaussian point in the Gaussian point cloud after the reconstruction is completed.

[0162] In some embodiments, the apparatus 200 is further configured to:

[0163] Input material properties, lighting conditions and visibility of each Gaussian point into the microsurface model to obtain the rendering color of each Gaussian point;

[0164] Integrate the rendered colors of each Gaussian point to obtain a rendered image;

[0165] The real image corresponding to the rendered image is used as a constraint to optimize the position of the Gaussian point cloud and the attribute values ​​of the Gaussian point cloud.

[0166] Understandably, Figure 2Each module / unit in the illustrated device 200 has the function of implementing each step in the method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect. For the sake of brevity, they will not be described in detail here.

[0167] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0168] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0169] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with the systems and techniques described herein).

[0177] The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A Gaussian inverse rendering method for an object based on global-local surface collaborative perception, characterized in that: include: Perform ray casting on each pixel in the multi-view image, perform sampling on the ray, and obtain each sampling point; The geometric network and color network are used to calculate the neural radiation distance function value and color value of each sampling point respectively, and obtain the global geometric representation; Initialize the Gaussian point cloud position according to the neural radiation distance function value of each sampling point, and obtain the initialized Gaussian point cloud; Select local control Gaussian points from the initialized Gaussian point cloud, and determine local sampling points with the local control Gaussian points as the center; Constructing a local distance function according to local control Gaussian points and local sampling points; According to the local sampling points, the local distance function and the neural radiation distance function are used to respectively calculate the local distance function value and the global neural radiation distance function value of the local sampling points with respect to the local control Gaussian point; Reconstructing geometric details according to the global geometric representation, using the local distance function value and the global neural radiation distance function value as constraints; After reconstruction is completed, attribute values ​​are added to the Gaussian point cloud for material and lighting modeling, material attributes and lighting conditions are obtained, and the visibility of each Gaussian point in the Gaussian point cloud is calculated; The method of using the geometric network and the color network to respectively calculate the neural radiation distance function value and the color value of each sampling point and obtain the global geometric representation includes: Encoding each sampling point to obtain a spatial position representation of each sampling point; The spatial position representation of each sampling point is input into the geometric network and the color network respectively, the neural radiation distance function value of each sampling point is calculated by the geometric network, and the color value of each sampling point is calculated by the color network; Obtain global geometric representation based on the neural radiation distance function value and color value of each sampling point; The method of constructing a local distance function according to the local control Gaussian points and the local sampling points includes: Construct a local density function according to local control Gaussian points and local sampling points; According to the local density function and the local sampling points, a local distance function is constructed; The local density function is: ； in, represents the local density function, represents the local sampling point, , , They represent the local control Gaussian points alpha blending coefficients, spatial positions and covariances, Represents a transpose operation; The local distance function is: ； in, represents the local distance function, is the minimum scaling factor for local control of Gaussian points.

2. The method according to claim 1, characterized in that Initializing the Gaussian point cloud position according to the neural radiation distance function value of each sampling point includes: The farthest point sampling method is used to downsample the point set corresponding to the neural radiation distance function value of each sampling point in the preset interval, and the sampling points obtained by downsampling are used as Gaussian points after the Gaussian point cloud position is initialized.

3. The method according to claim 1, characterized in that The step of selecting local control Gaussian points from the initialized Gaussian point cloud comprises: From the initialized Gaussian point cloud, a Gaussian point whose direct interpolation between Gaussian rendering depth and camera space depth is within a preset threshold is selected as a local control Gaussian point.

4. The method according to claim 1, characterized in that The method of determining the local sampling point with the local control Gaussian point as the center includes: With the local control Gaussian point as the center, random Gaussian noise is added along the shortest axis direction of the Gaussian point, and the local control Gaussian point and the added random Gaussian noise are used as local sampling points.

5. The method according to claim 1, characterized in that The method further comprises: Inputting the material properties, lighting conditions and visibility of each Gaussian point into a microsurface model to obtain a rendering color of each Gaussian point; Integrate the rendered colors of each Gaussian point to obtain a rendered image; The real image corresponding to the rendered image is used as a constraint to optimize the position of the Gaussian point cloud and the attribute values ​​of the Gaussian point cloud.

6. The method according to claim 1, characterized in that The method further comprises: According to the neural radiation distance function value and color value of each sampling point, a volume rendering image is generated using a volume rendering method; The geometry network and color network are optimized using the real image corresponding to the volume rendered image as a constraint.

7. A Gaussian inverse rendering device for an object based on global-local surface collaborative perception corresponding to claim 1, characterized in that: include: The sampling point acquisition module is used to project rays to each pixel in the multi-view image, perform sampling on the rays, and obtain each sampling point; A global geometric representation acquisition module is used to use a geometric network and a color network to respectively calculate the neural radiation distance function value and the color value of each sampling point and obtain a global geometric representation; The Gaussian point cloud position initialization module is used to initialize the Gaussian point cloud position according to the neural radiation distance function value of each sampling point and obtain the initialized Gaussian point cloud; A local sampling point determination module is used to select local control Gaussian points from the initialized Gaussian point cloud and determine local sampling points with the local control Gaussian points as the center; A local distance function construction module is used to construct a local distance function according to local control Gaussian points and local sampling points; A function value calculation module, used to calculate the local distance function value and the global neural radiation distance function value of the local sampling point with respect to the local control Gaussian point according to the local sampling point by using the local distance function and the neural radiation distance function; A geometric detail reconstruction module, used to reconstruct geometric details according to the global geometric representation, using the local distance function value and the global neural radiation distance function value as constraints; The material and lighting reconstruction module is used to add attribute values ​​to the Gaussian point cloud for material and lighting modeling after reconstruction, obtain material attributes and lighting conditions, and calculate the visibility of each Gaussian point in the Gaussian point cloud.

Images