Three-dimensional low-light scene brightening method, computer device and program product based on 3D Gaussian sputtering

Abstract

The present application relates to a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, a computer device and a program product. The three-dimensional low-illumination scene brightening method includes: setting a plurality of fulcrums in a three-dimensional space, wherein the attributes of the fulcrums include fulcrum position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the fulcrums, and appearance attributes and geometric attributes for determining the features of Gaussian points associated with the fulcrums; extracting the reflection component and illumination component involved in rendering from the appearance attributes; optimizing the fulcrum position attributes, Gaussian point positioning attributes, appearance attributes and geometric attributes based on the difference between the 2D image and the 3D Gaussian rendered image under the corresponding viewing angle; using a multi-layer perceptron to extract the gamma value from the appearance attributes to obtain an enhanced illumination component; multiplying the reflection component and the enhanced illumination component to obtain the Gaussian point color, obtaining brightness enhanced images under different viewing angles through volume rendering, and obtaining a brightness enhancement loss optimization multi-layer perceptron based on the brightness enhanced image.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a three-dimensional low-light scene brightening method based on 3D Gaussian sputtering, a computer device, and a program product. Background Art

[0002] Brightening three-dimensional low-light scenes is an important task in the fields of vision and graphics. It can brighten three-dimensional scenes with low brightness values ​​and adaptively obtain 2D images with normal brightness values ​​at any viewing angle. Given a series of low-light images from different viewing angles, the entire three-dimensional scene can be modeled and the brightness values ​​of the scene can be adaptively adjusted to obtain high-quality images. Due to the lack of a priori image with normal illumination, scenes that are adaptively brightened often produce poor rendering results. To improve the brightness of three-dimensional low-light scenes, the current mainstream method is to brighten the low-light images and then retrain the three-dimensional scene. However, these methods suffer from problems such as poor multi-view consistency.

[0003] Neural Radiance Fields (NeRF) proposes the use of neural networks to model 3D scenes. Due to its excellent performance in reconstructing geometry and appearance, it has become an excellent scene representation. Due to its reliability, various methods currently use it to reconstruct and brighten 3D low-light scenes.

[0004] RawNeRF (Nerf in the dark: High dynamic range view synthesis from noisy raw images) constructs a NeRF representation of the scene using noisy images directly acquired from the camera sensor, performing both denoising and brightening. Aleth-NeRF (Aleth-nerf: Illumination adaptive nerf with concealing field assumption) decomposes low-light scenes into normal and dark fields by introducing the concept of dark fields, brightening the scene by eliminating the dark fields. However, despite these valuable efforts, due to the limitations of the NeRF representation, these works still have significant room for improvement in terms of training and rendering times.

[0005] The recent introduction of 3D Gaussian Splatting (3DGS) has provided a new approach to 3D scene reconstruction. Compared to NeRF, 3DGS not only improves 3D scene reconstruction quality but also significantly accelerates training and rendering speeds. Therefore, brightening low-light scenes constructed using 3DGS has become a hot topic of research. Summary of the Invention

[0006] Based on this, it is necessary to provide a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering to address the above technical problems.

[0007] This application provides a three-dimensional low-light scene brightening method based on 3D Gaussian sputtering, including:

[0008] Input 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing the 3D Gaussian.

[0009] Setting a plurality of pivot points in a three-dimensional space, wherein attributes of the pivot points include pivot point position attributes, Gaussian point positioning attributes for determining positions of Gaussian points associated with the pivot points, geometric attributes for determining geometric features of Gaussian points associated with the pivot points, and appearance attributes for determining color features of Gaussian points associated with the pivot points, wherein an initial position of the pivot point is determined based on the point cloud;

[0010] Extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute and opacity attribute involved in rendering from the geometric attributes;

[0011] Optimizing the pivot point position attribute, the Gaussian point positioning attribute, the appearance attribute, and the geometric attribute based on a difference between the 2D image and the 3D Gaussian rendered image under corresponding viewing angles;

[0012] Extracting a gamma value from the appearance attribute using a multi-layer perceptron, and enhancing the illumination component according to a power of the gamma value to obtain an enhanced illumination component;

[0013] The reflection component is multiplied by the enhanced illumination component to obtain a Gaussian point color, brightness enhanced images under different viewing angles are obtained through volume rendering, brightness enhancement loss is obtained according to the brightness enhanced image, and the multilayer perceptron is optimized using the brightness enhancement loss.

[0014] Optionally, the attributes of the fulcrum further include fulcrum characteristic attributes, which are used to characterize the attributes of Gaussian points associated with the fulcrum, and the shapes of the appearance attributes and geometric attributes are the same as the fulcrum characteristic attributes.

[0015] Optionally, the optimization process of optimizing the pivot point position attribute, the Gauss point positioning attribute, the appearance attribute, and the geometric attribute is implemented using a total loss:

[0016] L total =λ1L1+λ2L d

[0017] L total , represents the total loss;

[0018] λ1 and λ2 are both constants;

[0019] L1 represents the difference, which is the average of the absolute difference between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle;

[0020] L d , represents the depth loss.

[0021] Optionally, the depth loss is obtained using the following formula:

[0022]

[0023] L d , represents the depth loss;

[0024] i, j, represent any two Gaussian points;

[0025] w i , represents the volume rendering weight;

[0026] z j , represents the distance from the Gaussian point to the imaging plane.

[0027] Optionally, the illuminance component is enhanced according to the power of the gamma value to obtain an enhanced illuminance component, which is implemented using the following formula:

[0028]

[0029] Where,

[0030] i′ i , represents the enhanced illumination component;

[0031] i i , represents the illumination component before enhancement;

[0032] γ i , represents the gamma value.

[0033] Optionally, a brightness enhancement loss is obtained according to the brightness enhanced image, and is implemented using the following formula:

[0034] L enhance =E[(C e -m) 2 ]

[0035] Where,

[0036] L enhance , represents the brightness enhancement loss;

[0037] E, means taking the average value;

[0038] C e , represents the brightness enhanced image;

[0039] m is a constant.

[0040] Optionally, the three-dimensional low-illumination scene brightening method further includes:

[0041] A reference image is obtained by rendering according to the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image.

[0042] Optionally, the three-dimensional low-illumination scene brightening method further includes:

[0043] A reference image is obtained by rendering the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image, which is specifically implemented using the following formula:

[0044]

[0045] Where,

[0046] L enhance , represents the brightness enhancement loss;

[0047] E, means taking the average value;

[0048] C e , represents the brightness enhanced image;

[0049] m, a constant;

[0050] λ is a constant;

[0051] R represents the reference image;

[0052] var, represents variance;

[0053] α is a constant.

[0054] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional low-light scene brightening method based on 3D Gaussian sputtering described in the present application.

[0055] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering described in the present application.

[0056] The three-dimensional low-light scene brightening method based on 3D Gaussian sputtering in this application has at least the following effects:

[0057] This application intends to adopt the latest three-dimensional scene reconstruction technology as the basis, and use geometric appearance decoupling and illumination component gamma transform to achieve brightness enhancement, so as to effectively perform adaptive brightness enhancement on low-light scenes while maintaining the three-dimensional scene geometric structure, including the use of 3D Gaussian sputtering and gamma transform to achieve adaptive brightening of three-dimensional low-light scenes.

[0058] This application separately provides and uses appearance attributes and geometric attributes, extracting the scene's illumination and reflection components from the appearance attributes and enhancing the illumination component. This improves the brightness of the three-dimensional scene while maintaining the scene's geometric structure. This application also extracts gamma values ​​from the appearance attributes and adaptively enhances the illumination component using the extracted gamma values, effectively maintaining three-dimensional consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of a process for brightening a three-dimensional low-illumination scene based on 3D Gaussian sputtering in one embodiment of the present application;

[0060] Figure 2 Schematic diagram of a process for brightening a three-dimensional low-illumination scene based on 3D Gaussian sputtering in one embodiment of the present application;

[0061] Figure 3 for Figure 1 Model architecture diagram of step S100;

[0062] Figure 4 for Figure 1 Model architecture diagram of step S200;

[0063] Figure 5 for Figure 1 Model architecture diagram of step S300;

[0064] Figure 6 This is a diagram showing the overall architecture of a model of a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering in one embodiment of the present application;

[0065] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] At present, three-dimensional low-light scene brightening based on neural radiation field has achieved good results, but there are still problems such as long training time, slow rendering speed and poor three-dimensional consistency.

[0067] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0068] This application intends to adopt the latest three-dimensional scene reconstruction technology as the basis, and use geometric appearance decoupling and illumination component gamma transform to achieve brightness enhancement, so as to effectively perform adaptive brightness enhancement on low-light scenes while maintaining the three-dimensional scene geometric structure, including the use of 3D Gaussian sputtering and gamma transform to achieve adaptive brightening of three-dimensional low-light scenes.

[0069] See also Figure 1 An embodiment of the present application provides a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, including step S100, three-dimensional scene appearance geometric decoupling; step S200, three-dimensional scene illumination component and reflection component decomposition; step S300, brightness adaptive enhancement based on gamma transform.

[0070] See also Figure 1 and Figure 2 Step S100, decoupling the appearance geometry of the three-dimensional scene, including steps S110 to S120. Step S110, inputting 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing the 3D Gaussian; Step S120, setting a number of pivot points in the three-dimensional space, wherein the attributes of the pivot points include pivot point position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the pivot points, geometric attributes for determining the geometric features of Gaussian points associated with the pivot points, and appearance attributes for determining the color features of Gaussian points associated with the pivot points. The initialization position of the pivot point is determined based on the point cloud.

[0071] Step S200 decomposes the illumination and reflection components of the 3D scene, including steps S210 to S220. Step S210 extracts the reflection and illumination components involved in rendering from the appearance attributes, and extracts the scaling, rotation, and opacity attributes involved in rendering from the geometry attributes. Step S220 optimizes the pivot point position attributes, Gaussian point positioning attributes, appearance attributes, and geometry attributes based on the differences between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle.

[0072] Step S300, adaptive brightness enhancement based on gamma transform, includes steps S310 to S320. In step S310, a multi-layer perceptron is used to extract the gamma value from the appearance attributes and enhance the illuminance component by a power of the gamma value to obtain an enhanced illuminance component. In step S320, the enhanced illuminance component is multiplied by the reflection component to obtain the Gaussian point color. Brightness-enhanced images at different viewing angles are obtained through volume rendering. A brightness enhancement loss is obtained based on the brightness-enhanced images and the brightness enhancement loss is used to optimize the multi-layer perceptron.

[0073] In step S120 of this embodiment, the appearance attributes and geometric attributes are separately provided and used to extract the illumination component and the reflection component of the scene from the appearance attributes, and the illumination component is enhanced in step S310 to improve the brightness of the three-dimensional scene while maintaining the geometric structure of the scene.

[0074] In this embodiment, step S220 executes steps S310 and S320 after completing the reconstruction of the three-dimensional scene. Steps S310 and S320 do not affect the reconstructed three-dimensional scene, but only change the rendering process of the three-dimensional scene. By extracting the gamma value of the appearance attribute, the extracted gamma value is used to adaptively enhance the illumination component, which can better maintain three-dimensional consistency. At the same time, on the one hand, the brightening process only changes the rendering process and does not affect the 3D Gaussian itself, reducing computing power loss; on the other hand, it can also conveniently meet the user's needs for restoring low-light scenes.

[0075] In some embodiments, steps S100 to S300 and their sub-steps are explained in detail, and examples of optional embodiments are given.

[0076] See also Figure 2 and Figure 3 Conventional 3D Gaussian sputtering directly records properties such as position, scale, rotation, color, and opacity through Gaussian points. Step S100 includes steps S110 to S120. Step S110 can be referred to in the prior art.

[0077] Step S120 records the Gaussian point attributes through the fulcrum, wherein the fulcrum position attribute is used to record the fulcrum position, and the Gaussian point positioning attribute is used to determine the position of the Gaussian associated with the fulcrum. It can be understood that the Gaussian point positioning attribute includes a vector pointing from a fulcrum to multiple Gaussian points, and this pointing relationship is that the fulcrum is associated with the Gaussian point. Compared with 3DGS, the Gaussian point attributes of this embodiment are all recorded by the relevant attributes of the fulcrum, and during the optimization process, only the relevant attributes of the fulcrum need to be optimized. Specifically, the Gaussian point attributes associated with the fulcrum can be obtained through a multi-layer perceptron (MLP), which can achieve higher quality results while reducing computing power.

[0078] Both appearance attributes and geometric attributes are multidimensional feature vectors, such as 32-dimensional feature vectors. Before optimization begins, this embodiment decouples scene appearance and geometry by providing appearance attributes and geometric attributes separately. During the subsequent optimization process, these attributes are optimized separately. This embodiment treats geometric attributes and appearance attributes (color attributes) as different attributes of the fulcrum, avoiding the possibility that changes to the scene geometry during the brightening process may cause changes to the appearance attributes.

[0079] Furthermore, in step S120, the attributes of the fulcrum also include fulcrum feature attributes, which are used to characterize the attributes of the Gaussian points associated with the fulcrum. The appearance attributes and geometric attributes are obtained by replacing the fulcrum feature attributes, and the shapes of the appearance attributes and geometric attributes are the same as the fulcrum feature attributes. From another perspective, this means that a feature vector with the same shape as the fulcrum feature attribute is provided as the geometric attribute and appearance attribute, and these two features are used to replace the original fulcrum feature attributes in each fulcrum in the scene.

[0080] See also Figure 2 and Figure 4 Step S200 includes steps S210 to S220. Step S220: extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute, and opacity attribute involved in rendering from the geometric attributes.

[0081] After the appearance geometry decoupling stage, the scene appearance has been effectively extracted. Geometric features are used to extract the scale attribute s, rotation attribute r, and opacity attribute o of the associated Gaussian points. Based on the basic theory of computer graphics, the appearance attributes of the object can be used to obtain the illumination attributes of the associated Gaussian points.

[0082] (i.e., illuminance component r) and reflection attribute (i.e., reflection component i). The illuminance component reflects the brightness distribution of the environment, while the reflection component reflects the essential characteristics of the object.

[0083] The extraction of the reflection component and the illumination component is performed, for example, by a multi-layer perceptron. Specifically, the appearance attributes are input into two different multi-layer perceptrons (MLPs) to optimize the illumination attribute and the reflection attribute of the Gaussian point. This is shown in the following formula:

[0084] i i =F θ1 (f j ) (1)

[0085] r i =F θ2 (f j ) (2)

[0086] f j , representing appearance attributes;

[0087] i i , represents the illumination component of the Gaussian point associated with the support;

[0088] F θ1 , represents the multi-layer perceptron used to extract the illumination component;

[0089] r i , represents the reflection component of the Gaussian point associated with the support;

[0090] F θ2 , represents a multi-layer perceptron used to extract the reflection component.

[0091] A simple multiplication of the obtained illumination and reflection components yields Gaussian color properties, ultimately producing images from different perspectives through volume rendering. It can be understood that when brightening a 3D low-light scene, maintaining the reflection component and adjusting only the illumination component can prevent degradation in the resulting scene quality.

[0092] In step S220, based on the difference between the 2D image and the 3D Gaussian rendering image under the corresponding viewing angle, the optimization process of optimizing the pivot point position attributes, Gaussian point positioning attributes, appearance attributes and geometric attributes is implemented using the total loss:

[0093] L total =λ1L1+λ2L d (3)

[0094] L total , represents the total loss;

[0095] λ1 and λ2 are both constants, for example, λ1 is 1 and λ2 is 0.05;

[0096] L1, represents the difference between the 2D image and the 3D Gaussian rendering image at the corresponding viewing angle, which is the average of the absolute difference between the 2D image and the 3D Gaussian rendering image at the corresponding viewing angle;

[0097] L d , represents the depth loss.

[0098] This step uses equation (3) to optimize the scene and effectively reconstruct the 3D low-light scene. Note that this step only involves 3D scene reconstruction and does not improve the brightness of the scene.

[0099] Depth loss makes the Gaussian points as close to the surface of the object as possible, effectively alleviating the lack of constraints on the scene geometry structure in low-light images. See equation (4) for details.

[0100] L d =∑ i，j w i w j |z i -z j | (4)

[0101] L d , represents the depth loss;

[0102] i, j, represent any two Gaussian points;

[0103] w i , represents the volume rendering weight;

[0104] z j , represents the distance from the Gaussian point to the imaging plane.

[0105] This embodiment introduces a depth constraint in this step, reducing the number of Gaussian points in incorrect locations. After the first two stages, we have achieved reconstruction of the 3D low-light scene. Next, we fix the scene's geometric structure and operate only on the scene's appearance.

[0106] See also Figure 2 and Figure 5 Step S300 includes steps S310 to S320. Step S310: extracting a gamma value from the appearance attribute using a multi-layer perceptron, and enhancing the illuminance component according to the power of the gamma value to obtain an enhanced illuminance component.

[0107] The gamma value is extracted from the appearance attributes using a multilayer perceptron, which is implemented using Equation (5). Equation (5) inputs the appearance features into a multilayer perceptron to obtain the gamma value corresponding to the Gaussian point:

[0108] γ i =F θ3 (f j ) (5)

[0109] γ i , represents the gamma value of the Gaussian point associated with the pivot point;

[0110] f j , indicating appearance attributes;

[0111] F θ3 , represents a multilayer perceptron used to extract gamma values.

[0112] The illuminance component is enhanced by the power of the gamma value to obtain the enhanced illuminance component, which is implemented using the following formula:

[0113]

[0114] Where,

[0115] i′ i , represents the enhanced illumination component, that is, the enhanced illumination component;

[0116] i i , represents the illumination component before enhancement;

[0117] γ i , represents the gamma value.

[0118] Different from the traditional method of directly brightening a 2D image, this embodiment performs a gamma transform on each Gaussian point to achieve adaptive enhancement of the illumination component, thereby changing the color attribute of each Gaussian point and ultimately obtaining a brightness-enhanced image through volume rendering.

[0119] In step S320 , the reflection component is multiplied by the enhanced illumination component to obtain the Gaussian point color, brightness enhanced images under different viewing angles are obtained through volume rendering, brightness enhancement loss is obtained based on the brightness enhanced image, and the brightness enhancement loss is used to optimize the multi-layer perceptron.

[0120] Multiply the reflection component by the enhanced illumination component to obtain the Gaussian point color, which is performed using the following formula:

[0121] c′ i =i′ i *r i (7)

[0122] i′ i , represents the enhanced illumination component;

[0123] c′ i , represents the Gaussian point color after the illumination component is enhanced;

[0124] r i , represents the reflection component.

[0125] Formula (7) obtains the adaptive enhancement result of the Gaussian point color attribute through a simple multiplication operation, and finally the illumination enhanced image under different viewing angles can be obtained through volume rendering.

[0126] In step S320, the brightness enhancement loss is obtained according to the brightness enhancement image, using the following formula:

[0127] L enhance =E[(C e -m) 2 ]

[0128] Where,

[0129] L enhance , represents the brightness enhancement loss;

[0130] E, means taking the average value;

[0131] C e , represents the brightness enhanced image;

[0132] m is a constant, for example, 0.55.

[0133] In some embodiments, step S320 further includes: rendering a reference image according to the reflection component, and compensating for the brightness enhancement loss using the difference between the reference image and the brightness enhancement image, specifically using the following formula:

[0134]

[0135] Where,

[0136] L enhance , represents the brightness enhancement loss;

[0137] E, means taking the average value;

[0138] C e , represents the brightness enhanced image;

[0139] m is a constant, for example, 0.55;

[0140] λ is a constant, such as 0.1;

[0141] R, represents the reference image;

[0142] var, represents variance;

[0143] α is a constant, such as 1.0.

[0144] In this embodiment, the first term of the brightness enhancement loss function “E[(C e -m) 2 ]" to brighten the scene; the second item of the function The significance of this is to use the image rendered with the reflection component to constrain the brightened scene to prevent the color from being too different from the original scene.

[0145] See also Figure 1 and Figure 6 The three-dimensional low-light scene brightening method based on 3D Gaussian sputtering provided in each embodiment of the present application is used to enhance the brightness of a three-dimensional low-light scene, and can maintain the geometric structure of the scene while improving the brightness of the scene. In step S100, the scene appearance and the geometric structure are decoupled, and the appearance and geometry of the three-dimensional scene are decomposed. Step S200 is used to decompose the appearance of the scene into an illumination component representing the lighting conditions of the scene and a reflection component representing the essential characteristics of the object. Step S300 is used to adaptively enhance the brightness of the three-dimensional scene while maintaining the geometric structure of the scene.

[0146] The embodiments of the present application can achieve adaptive brightening of a three-dimensional low-light scene while maintaining the scene's geometric structure by adaptively enhancing the illumination component of Gaussian points in space. Depending on the loss function, the training process can be further divided into two steps: (1) Reconstructing the three-dimensional scene using a low-light image, which completes step S200. (2) Improving the illumination component based on fixed scene geometric parameters to achieve adaptive brightness improvement of the scene, which completes step S300.

[0147] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0148] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a three-dimensional low-light scene brightening method based on 3D Gaussian sputtering is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0150] Step S100, decoupling the appearance geometry of a three-dimensional scene, includes steps S110 to S120. In step S110, 2D images of the same scene from different perspectives are input into the colmap software to obtain a point cloud for initializing a 3D Gaussian. In step S120, several pivot points are set in three-dimensional space. The pivot point attributes include pivot point position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the pivot points, geometric attributes for determining the geometric features of Gaussian points associated with the pivot points, and appearance attributes for determining the color features of Gaussian points associated with the pivot points. The initial position of the pivot point is determined based on the point cloud.

[0151] Step S200 decomposes the illumination component and the reflection component of the 3D scene, including steps S210 to S220. In step S210, the reflection component and illumination component involved in rendering are extracted from the appearance attributes, and the scaling attribute, rotation attribute, and opacity attribute involved in rendering are extracted from the geometric attributes. In step S220, the pivot point position attribute, Gaussian point positioning attribute, appearance attribute, and geometric attribute are optimized based on the difference between the 2D image and the 3D Gaussian rendered image under the corresponding viewing angle.

[0152] Step S300, adaptive brightness enhancement based on gamma transform, includes steps S310 to S320. In step S310, a multi-layer perceptron is used to extract the gamma value from the appearance attributes and enhance the illuminance component by a power of the gamma value to obtain an enhanced illuminance component. In step S320, the enhanced illuminance component is multiplied by the reflection component to obtain the Gaussian point color. Brightness-enhanced images at different viewing angles are obtained through volume rendering. A brightness enhancement loss is obtained based on the brightness-enhanced images and the brightness enhancement loss is used to optimize the multi-layer perceptron.

[0153] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:

[0154] Step S100, decoupling the appearance geometry of a three-dimensional scene, includes steps S110 to S120. In step S110, 2D images of the same scene from different perspectives are input into the colmap software to obtain a point cloud for initializing a 3D Gaussian. In step S120, several pivot points are set in three-dimensional space. The pivot point attributes include pivot point position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the pivot points, geometric attributes for determining the geometric features of Gaussian points associated with the pivot points, and appearance attributes for determining the color features of Gaussian points associated with the pivot points. The initial position of the pivot point is determined based on the point cloud.

[0155] Step S200 decomposes the illumination component and the reflection component of the 3D scene, including steps S210 to S220. In step S210, the reflection component and illumination component involved in rendering are extracted from the appearance attributes, and the scaling attribute, rotation attribute, and opacity attribute involved in rendering are extracted from the geometric attributes. In step S220, the pivot point position attribute, Gaussian point positioning attribute, appearance attribute, and geometric attribute are optimized based on the difference between the 2D image and the 3D Gaussian rendered image under the corresponding viewing angle.

[0156] Step S300, adaptive brightness enhancement based on gamma transform, includes steps S310 to S320. In step S310, a multi-layer perceptron is used to extract the gamma value from the appearance attributes and enhance the illuminance component by a power of the gamma value to obtain an enhanced illuminance component. In step S320, the enhanced illuminance component is multiplied by the reflection component to obtain the Gaussian point color. Brightness-enhanced images at different viewing angles are obtained through volume rendering. A brightness enhancement loss is obtained based on the brightness-enhanced images and the brightness enhancement loss is used to optimize the multi-layer perceptron.

[0157] In this embodiment, the computer program product includes a program code portion for executing the steps of the three-dimensional low-light scene brightening method based on 3D Gaussian sputtering in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for download with the aid of a hardware description language.

[0158] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.

[0160] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A three-dimensional low-light scene brightening method based on 3D Gaussian sputtering, characterized in that: include: Input 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing the 3D Gaussian. Setting a plurality of pivot points in a three-dimensional space, wherein attributes of the pivot points include pivot point position attributes, Gaussian point positioning attributes for determining positions of Gaussian points associated with the pivot points, geometric attributes for determining geometric features of Gaussian points associated with the pivot points, and appearance attributes for determining color features of Gaussian points associated with the pivot points, wherein an initial position of the pivot point is determined based on the point cloud; Extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute and opacity attribute involved in rendering from the geometric attributes; Optimizing the pivot point position attribute, the Gaussian point positioning attribute, the appearance attribute, and the geometric attribute based on a difference between the 2D image and the 3D Gaussian rendered image under corresponding viewing angles; Extracting a gamma value from the appearance attribute using a multi-layer perceptron, and enhancing the illumination component according to a power of the gamma value to obtain an enhanced illumination component; The reflection component is multiplied by the enhanced illumination component to obtain a Gaussian point color, brightness enhanced images under different viewing angles are obtained through volume rendering, brightness enhancement loss is obtained according to the brightness enhanced image, and the multilayer perceptron is optimized using the brightness enhancement loss.

2. The three-dimensional low-light scene brightening method according to claim 1, wherein: The attributes of the fulcrum also include fulcrum characteristic attributes, which are used to characterize the attributes of Gaussian points associated with the fulcrum. The shapes of the appearance attributes and geometric attributes are the same as the fulcrum characteristic attributes.

3. The three-dimensional low-light scene brightening method according to claim 1, wherein: The optimization process of optimizing the pivot point location attribute, the Gauss point positioning attribute, the appearance attribute, and the geometric attribute is implemented using a total loss: L total =λ1L1+λ2L d L total , represents the total loss; λ1 and λ2 are both constants; L1 represents the difference, which is the average of the absolute differences between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle; L d , represents the depth loss.

4. The three-dimensional low-light scene brightening method according to claim 3, wherein: The depth loss is obtained using the following formula: L d , represents the depth loss; i, j, represent any two Gaussian points; w i , represents the volume rendering weight; z j , represents the distance from the Gaussian point to the imaging plane.

5. The three-dimensional low-light scene brightening method according to claim 1, wherein: The illuminance component is enhanced according to the power of the gamma value to obtain an enhanced illuminance component, which is implemented using the following formula: Where, i′ i , represents the enhanced illumination component; i i , represents the illumination component before enhancement; γ i , represents the gamma value.

6. The three-dimensional low-light scene brightening method according to claim 1, wherein: The brightness enhancement loss is obtained according to the brightness enhancement image, and is implemented using the following formula: L enhance =E[(C e -m) 2 ] Where, L enhance , represents the brightness enhancement loss; E, means taking the average value; C e , represents the brightness enhanced image; m is a constant.

7. The three-dimensional low-light scene brightening method according to claim 1, wherein: The three-dimensional low-illumination scene brightening method further includes: A reference image is obtained by rendering according to the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image.

8. The three-dimensional low-light scene brightening method according to claim 1, wherein: The three-dimensional low-illumination scene brightening method further includes: A reference image is obtained by rendering the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image, which is specifically implemented using the following formula: Where, L enhance , represents the brightness enhancement loss; E, means taking the average value; C e , represents the brightness enhanced image; m, a constant; λ is a constant; R represents the reference image; var, represents variance; α is a constant.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instruction is executed by a processor, the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering are implemented as described in any one of claims 1 to 8.

Images