Three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, computer equipment and program product

Abstract

The invention relates to a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, computer equipment and a program product, the three-dimensional low-illumination scene brightening method comprises the following steps: a plurality of fulcrums are arranged in a three-dimensional space, the attributes of the fulcrums comprise fulcrum position attributes and Gaussian point positioning attributes used for determining the positions of Gaussian points associated with the fulcrums, and the attributes of the fulcrums comprise the fulcrum position attributes and the Gaussian point positioning attributes used for determining the positions of the Gaussian points associated with the fulcrums; the attribute determining module is used for determining appearance attributes and geometric attributes of Gaussian point features associated with fulcrums; extracting reflection components and illumination components participating in rendering according to the appearance attributes; optimizing a fulcrum position attribute, a Gaussian point positioning attribute, an appearance attribute and a geometric attribute according to the difference between the 2D image and the 3D Gaussian rendering image under the corresponding view angle; extracting a gamma value from the appearance attribute by using a multi-layer perceptron to obtain an enhanced illumination component; and multiplying the reflection component by the enhanced illumination component to obtain a Gaussian point color, obtaining brightness enhanced images under different visual angles through volume rendering, and obtaining a brightness enhancement loss optimization multilayer perceptron according to the brightness enhanced images.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, a computer device, and a program product. Background Art

[0002] Brightening three-dimensional low-light scenes is an important task in the field of vision and graphics. It can enhance the brightness of three-dimensional scenes with low brightness values ​​and adaptively obtain 2D images with normal brightness values ​​at any viewing angle. Given a series of low-light images from different viewing angles, the entire three-dimensional scene can be modeled and the brightness values ​​of the scene can be adaptively adjusted to obtain high-quality images. Since there is no prior image with normal illumination, scenes after adaptive brightening often produce poor rendering results. In order to improve the brightness of three-dimensional low-light scenes, the existing mainstream method is to retrain the three-dimensional scene after brightening the low-light image. However, they have problems such as poor multi-view consistency.

[0003] Neural Radiance Fields (NeRF) proposes to use neural networks to model 3D scenes. Due to its excellent performance in reconstructing geometry and appearance, it has become an excellent scene representation. Due to its reliability, there are currently many methods that use it to reconstruct 3D low-light scenes and brighten them.

[0004] RawNeRF (Nerf in the dark: High dynamic range view synthesis from noisy raw images) constructs a NeRF representation of the scene through noisy images directly obtained from the camera sensor to perform denoising and brightening at the same time. Aleth-NeRF (Aleth-nerf: Illumination adaptive nerf with concealing field assumption) decomposes low-light scenes into normal scenes and dark fields by introducing the concept of dark fields, and brightens the scene by eliminating dark fields. However, despite the valuable efforts, due to the limitations of the NeRF expression method, there is much room for improvement in the training time and rendering time of these works.

[0005] Recently, the introduction of 3D Gaussian Splatting (3DGS) has provided a new idea for 3D scene reconstruction. Compared with NeRF, 3DGS not only improves the reconstruction quality of 3D scenes, but also greatly speeds up the training and rendering speeds. Therefore, how to brighten low-light scenes constructed by 3DGS has become a research hotspot for technical issues. Summary of the invention

[0006] Based on this, it is necessary to provide a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering to address the above technical problems.

[0007] The present application provides a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, including:

[0008] Input 2D images of the same scene from different perspectives into the colmap software to obtain the point cloud used to initialize the 3D Gaussian;

[0009] A plurality of pivots are set in a three-dimensional space, wherein the attributes of the pivots include pivot position attributes, Gauss point positioning attributes for determining the positions of Gauss points associated with the pivots, geometric attributes for determining geometric features of Gauss points associated with the pivots, and appearance attributes for determining color features of Gauss points associated with the pivots, and the initialization positions of the pivots are determined according to the point cloud;

[0010] Extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute and opacity attribute involved in rendering from the geometric attributes;

[0011] Optimizing the pivot point position attribute, the Gauss point positioning attribute, the appearance attribute, and the geometric attribute according to the difference between the 2D image and the 3D Gaussian rendering image under the corresponding viewing angle;

[0012] Extracting a gamma value from the appearance attribute using a multi-layer perceptron, enhancing the illumination component according to a power of the gamma value, and obtaining an enhanced illumination component;

[0013] The reflection component is multiplied by the enhanced illumination component to obtain the Gaussian point color, brightness enhanced images under different viewing angles are obtained through volume rendering, brightness enhancement loss is obtained according to the brightness enhanced image, and the multi-layer perceptron is optimized using the brightness enhancement loss.

[0014] Optionally, the attributes of the fulcrum also include fulcrum characteristic attributes, and the fulcrum characteristic attributes are used to characterize the attributes of Gaussian points associated with the fulcrum, and the shapes of the appearance attributes and geometric attributes are the same as the fulcrum characteristic attributes.

[0015] Optionally, the optimization process of optimizing the pivot point position attribute, the Gauss point positioning attribute, the appearance attribute and the geometric attribute is implemented using a total loss:

[0016] L total =λ1L1+λ2L d

[0017] L total , represents the total loss;

[0018] λ1 and λ2 are both constants;

[0019] L1, represents the difference, which is the average of the absolute difference between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle;

[0020] L d , represents the depth loss.

[0021] Optionally, the depth loss is obtained using the following formula:

[0022]

[0023] L d , represents the depth loss;

[0024] i, j, represent any two Gaussian points;

[0025] w i , represents the volume rendering weight;

[0026] z j , represents the distance from the Gaussian point to the imaging plane.

[0027] Optionally, the illuminance component is enhanced according to the power of the gamma value to obtain an enhanced illuminance component, which is implemented using the following formula:

[0028]

[0029] In the formula,

[0030] i′ i , represents the enhanced illumination component;

[0031] i i , represents the illumination component before enhancement;

[0032] γ i , represents the gamma value.

[0033] Optionally, the brightness enhancement loss is obtained according to the brightness enhanced image, and is implemented using the following formula:

[0034] L enhance =E[(C e -m) 2 ]

[0035] In the formula,

[0036] L enhance , indicating brightness enhancement loss;

[0037] E, means taking the average value;

[0038] C e , represents the brightness enhanced image;

[0039] m is a constant.

[0040] Optionally, the three-dimensional low-illumination scene brightening method further includes:

[0041] A reference image is obtained by rendering according to the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image.

[0042] Optionally, the three-dimensional low-illumination scene brightening method further includes:

[0043] A reference image is obtained by rendering the reflection component, and the brightness enhancement loss is supplemented by using the difference between the reference image and the brightness enhancement image, which is specifically implemented by the following formula:

[0044]

[0045] In the formula,

[0046] L enhance , indicating brightness enhancement loss;

[0047] E, means taking the average value;

[0048] C e , represents the brightness enhanced image;

[0049] m, is a constant;

[0050] λ is a constant;

[0051] R, represents the reference image;

[0052] var, represents variance;

[0053] α is a constant.

[0054] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering described in the present application.

[0055] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering described in the present application.

[0056] The three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering in this application has at least the following effects:

[0057] This application intends to adopt the latest three-dimensional scene reconstruction technology as the basis, and use geometric appearance decoupling and illumination component gamma transform to achieve brightness enhancement, so as to effectively perform adaptive brightness enhancement on low-illuminance scenes while maintaining the three-dimensional scene geometric structure, including the use of 3D Gaussian sputtering and gamma transform to achieve adaptive brightening of three-dimensional low-illuminance scenes.

[0058] The present application provides and uses appearance attributes and geometric attributes separately, extracts the illumination component and reflection component of the scene from the appearance attributes, and enhances the illumination component, thereby improving the brightness of the three-dimensional scene while maintaining the geometric structure of the scene. The present application extracts the gamma value of the appearance attribute, and uses the extracted gamma value to adaptively enhance the illumination component, thereby better maintaining the three-dimensional consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the process of a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering in one embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of the process of a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering in one embodiment of the present application;

[0061] Figure 3 for Figure 1 Model architecture diagram of step S100;

[0062] Figure 4 for Figure 1 Model architecture diagram of step S200;

[0063] Figure 5 for Figure 1 Model architecture diagram of step S300;

[0064] Figure 6 This is a model overall architecture diagram of a three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering in one embodiment of the present application;

[0065] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] At present, three-dimensional low-light scene brightening based on neural radiation fields has achieved good results, but there are still problems such as long training time, slow rendering speed and poor three-dimensional consistency.

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] This application intends to adopt the latest three-dimensional scene reconstruction technology as the basis, and use geometric appearance decoupling and illumination component gamma transform to achieve brightness enhancement, so as to effectively perform adaptive brightness enhancement on low-illuminance scenes while maintaining the three-dimensional scene geometric structure, including the use of 3D Gaussian sputtering and gamma transform to achieve adaptive brightening of three-dimensional low-illuminance scenes.

[0069] See also Figure 1 An embodiment of the present application provides a three-dimensional low-illuminance scene brightening method based on 3D Gaussian sputtering, including step S100, three-dimensional scene appearance geometric decoupling; step S200, three-dimensional scene illumination component and reflection component decomposition; step S300, brightness adaptive enhancement based on gamma transform.

[0070] See also Figure 1 and Figure 2 , step S100, decoupling the appearance geometry of the three-dimensional scene, including steps S110 to S120. Step S110, inputting 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing a 3D Gaussian; step S120, setting a number of pivots in the three-dimensional space, the attributes of the pivots include pivot position attributes, Gauss point positioning attributes for determining the positions of Gaussian points associated with the pivots, geometric attributes for determining the geometric features of Gaussian points associated with the pivots, and appearance attributes for determining the color features of Gaussian points associated with the pivots, and the initialization position of the pivot is determined according to the point cloud.

[0071] Step S200, decomposition of illumination component and reflection component of a three-dimensional scene, including steps S210 to S220. Step S210, extracting reflection component and illumination component involved in rendering from appearance attributes, extracting scaling attribute, rotation attribute and opacity attribute involved in rendering from geometric attributes; Step S220, optimizing pivot position attribute, Gauss point positioning attribute, appearance attribute and geometric attribute according to the difference between the 2D image and the rendered image under the corresponding viewing angle of the 3D Gaussian.

[0072] Step S300, adaptive brightness enhancement based on gamma transformation, includes steps S310 to S320. Step S310, extracting gamma value from appearance attribute by multi-layer perceptron, enhancing illumination component by power of gamma value, obtaining enhanced illumination component; Step S320, multiplying reflection component by enhanced illumination component to obtain Gaussian point color, obtaining brightness enhanced image under different viewing angles by volume rendering, obtaining brightness enhancement loss according to brightness enhanced image, and optimizing multi-layer perceptron by brightness enhancement loss.

[0073] In step S120 of this embodiment, the appearance attributes and geometric attributes are provided and used separately to extract the illumination component and the reflection component of the scene from the appearance attributes, and the illumination component is enhanced in step S310 to improve the brightness of the three-dimensional scene while maintaining the geometric structure of the scene.

[0074] In this embodiment, step S220 executes step S310 and step S320 after completing the reconstruction of the three-dimensional scene. Step S310 and step S320 do not affect the reconstructed three-dimensional scene, but only change the rendering process of the three-dimensional scene. By extracting the gamma value of the appearance attribute, the illumination component is adaptively enhanced using the extracted gamma value, which can better maintain the three-dimensional consistency. At the same time, on the one hand, the brightening process only changes the rendering process, and will not affect the 3D Gaussian itself, reducing the loss of computing power; on the other hand, it can also conveniently meet the user's needs to restore low-light scenes.

[0075] In some embodiments, steps S100 to S300 and their sub-steps are explained in detail, and examples of optional embodiments are given.

[0076] See also Figure 2 and Figure 3 , conventional 3D Gaussian sputtering directly records properties such as position, scaling, rotation, color, opacity, etc. through Gaussian points. Step S100 includes steps S110 to S120. Step S110 can refer to the prior art.

[0077] Step S120 records Gaussian point attributes through the fulcrum, wherein the fulcrum position attribute is used to record the fulcrum position, and the Gaussian point positioning attribute is used to determine the position of the Gaussian associated with the fulcrum. It can be understood that the Gaussian point positioning attribute includes a vector pointing from a fulcrum to multiple Gaussian points, and this pointing relationship is that the fulcrum is associated with the Gaussian point. Compared with 3DGS, the Gaussian point attributes of this embodiment are all recorded by the relevant attributes of the fulcrum, and only the relevant attributes of the fulcrum need to be optimized during the optimization process. Specifically, the Gaussian point attributes associated with the fulcrum can be obtained through a multi-layer perceptron (MLP), which can achieve higher quality results while reducing computing power.

[0078] Both appearance attributes and geometric attributes are multi-dimensional feature vectors, such as 32-dimensional feature vectors. Before the optimization begins, this embodiment decouples the scene appearance and geometric structure by providing appearance attributes and geometric attributes respectively, and optimizes the appearance attributes and geometric attributes separately in the subsequent optimization process. This embodiment uses geometric attributes and appearance attributes (color attributes) as different attributes of the fulcrum to avoid changes in the scene geometry that may be caused by changing the appearance attributes during the brightening process.

[0079] Further, in step S120, the attributes of the fulcrum also include fulcrum feature attributes, which are used to characterize the attributes of Gaussian points associated with the fulcrum. The appearance attributes and geometric attributes are obtained by replacing the fulcrum feature attributes, and the shapes of the appearance attributes and geometric attributes are the same as the fulcrum feature attributes. From another perspective, a feature vector with the same shape as the fulcrum feature attribute is provided as the geometric attribute and the appearance attribute, and these two features are used to replace the original fulcrum feature attributes in each fulcrum of the scene.

[0080] See also Figure 2 and Figure 4 Step S200 includes steps S210 to S220. Step S220, extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute and opacity attribute involved in rendering from the geometric attributes.

[0081] After the appearance geometry decoupling stage, the scene appearance has been effectively extracted. Geometric features are used to extract the scaling attribute s, rotation attribute r, and opacity attribute o of the associated Gaussian points. Based on the basic theory of computer graphics, the appearance attributes of the object can be used to obtain the illumination attributes of the associated Gaussian points.

[0082] (i.e., illuminance component r) and reflection attribute (i.e., reflection component i). The illuminance component reflects the brightness distribution of the environment, while the reflection component reflects the essential characteristics of the object.

[0083] The extraction of the reflection component and the illumination component is performed, for example, by a multi-layer perceptron. Specifically, the appearance attribute is used as input and input into two different multi-layer perceptrons MLP to optimize the illumination attribute and the reflection attribute of the Gaussian point. As shown in the following formula:

[0084] i i =F θ1 (f j ) (1)

[0085] r i =F θ2 (f j ) (2)

[0086] f j , representing the appearance attributes;

[0087] i i , represents the illumination component of the Gaussian point associated with the fulcrum;

[0088] F θ1 , represents a multi-layer perceptron used to extract the illumination component;

[0089] r i , represents the reflection component of the Gaussian point associated with the support point;

[0090] F θ2 , represents a multi-layer perceptron used to extract the reflection component.

[0091] For the obtained illumination component and reflection component, the Gaussian color attribute can be obtained by a simple multiplication operation, and finally the image under different viewing angles can be obtained by volume rendering. It can be understood that in the process of brightening the three-dimensional low-illuminance scene, keeping the reflection component of the scene and only adjusting the illumination component can prevent the quality of the final scene generation from decreasing.

[0092] In step S220, according to the difference between the 2D image and the 3D Gaussian rendering image under the corresponding viewing angle, the optimization process of optimizing the pivot point position attribute, Gaussian point positioning attribute, appearance attribute and geometric attribute is implemented using the total loss:

[0093] L total =λ1L1+λ2L d (3)

[0094] L total , represents the total loss;

[0095] λ1 and λ2 are both constants, for example, λ1 is 1 and λ2 is 0.05;

[0096] L1, represents the difference between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle, which is the average of the absolute difference between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle;

[0097] L d , represents the depth loss.

[0098] This step uses equation (3) to optimize the scene and effectively reconstruct the three-dimensional low-light scene. Note that this step only involves three-dimensional scene reconstruction and does not improve the brightness of the scene.

[0099] The depth loss makes the Gaussian points as close to the surface of the object as possible, effectively alleviating the lack of constraints on the scene geometry structure in low-light images. See equation (4) for details.

[0100] L d =∑ i，j w i w j |z i -z j | (4)

[0101] L d , represents the depth loss;

[0102] i, j, represent any two Gaussian points;

[0103] w i , represents the volume rendering weight;

[0104] z j , represents the distance from the Gaussian point to the imaging plane.

[0105] This embodiment introduces a depth constraint in this step to reduce the number of Gaussian points in the wrong position. After the first two stages, we have achieved the reconstruction of the three-dimensional low-light scene. Next, we fix the geometric structure of the scene and only operate on the appearance part of the scene.

[0106] See also Figure 2 and Figure 5 Step S300 includes steps S310 to S320. Step S310: extracting a gamma value from the appearance attribute using a multi-layer perceptron, enhancing the illumination component according to the power of the gamma value, and obtaining an enhanced illumination component.

[0107] The gamma value is extracted from the appearance attribute using a multilayer perceptron, which is performed using equation (5). Equation (5) inputs the appearance feature into a multilayer perceptron to obtain the gamma value corresponding to the Gaussian point:

[0108] γ i =F θ3 (f j ) (5)

[0109] γ i , represents the gamma value of the Gaussian point associated with the pivot point;

[0110] f j , indicating appearance attributes;

[0111] F θ3 , represents a multilayer perceptron used to extract the gamma value.

[0112] The illuminance component is enhanced according to the power of the gamma value to obtain the enhanced illuminance component, which is implemented using the following formula:

[0113]

[0114] In the formula,

[0115] i′ i , represents the enhanced illumination component, that is, the enhanced illumination component;

[0116] i i , represents the illumination component before enhancement;

[0117] γ i , represents the gamma value.

[0118] Different from the conventional method of directly brightening a 2D image, this embodiment performs a gamma transform on each Gaussian point to achieve adaptive enhancement of the illumination component, thereby changing the color attribute of each Gaussian point, and finally obtaining a brightness-enhanced image through volume rendering.

[0119] Step S320, multiplying the reflection component and the enhanced illumination component to obtain the Gaussian point color, obtaining the brightness enhanced images under different viewing angles through volume rendering, obtaining the brightness enhancement loss according to the brightness enhanced image, and optimizing the multi-layer perceptron using the brightness enhancement loss.

[0120] Multiply the reflection component by the enhanced illumination component to obtain the Gaussian point color, using the following formula:

[0121] c′ i =i′ i *r i (7)

[0122] i′ i , represents the enhanced illumination component;

[0123] c′ i , represents the Gaussian point color after the illumination component is enhanced;

[0124] r i , represents the reflection component.

[0125] Formula (7) obtains the adaptive enhancement result of the Gaussian point color attribute through a simple multiplication operation, and finally the illumination enhanced image under different viewing angles can be obtained through volume rendering.

[0126] In step S320, the brightness enhancement loss is obtained according to the brightness enhancement image, using the following formula:

[0127] L enhance =E[(C e -m) 2 ]

[0128] In the formula,

[0129] L enhance , indicating brightness enhancement loss;

[0130] E, means taking the average value;

[0131] C e , represents the brightness enhanced image;

[0132] m is a constant, for example, 0.55.

[0133] In some implementations, step S320 further includes: obtaining a reference image by rendering according to the reflection component, and compensating for the brightness enhancement loss by using the difference between the reference image and the brightness enhancement image, which is specifically implemented by the following formula:

[0134]

[0135] In the formula,

[0136] L enhance , indicating brightness enhancement loss;

[0137] E, means taking the average value;

[0138] C e , represents the brightness enhanced image;

[0139] m is a constant, for example, 0.55;

[0140] λ is a constant, such as 0.1;

[0141] R, represents the reference image;

[0142] var, represents variance;

[0143] α is a constant, such as 1.0.

[0144] In this embodiment, the first term of the brightness enhancement loss function “E[(C e -m) 2 ]" to brighten the scene; the second term of the function The significance of this is to use the image rendered with the reflection component to constrain the brightened scene to prevent the color from being too different from the original scene.

[0145] See also Figure 1 and Figure 6 The three-dimensional low-light scene brightening method based on 3D Gaussian sputtering provided in each embodiment of the present application is used to enhance the brightness of a three-dimensional low-light scene, and can maintain the geometric structure of the scene while improving the brightness of the scene. Among them, step S100 decouples the scene appearance from the geometric structure, and decomposes the appearance and geometry of the three-dimensional scene. Step S200 is used to decompose the appearance of the scene into an illumination component representing the lighting conditions of the scene and a reflection component of the essential characteristics of the object. Step S300 is used to adaptively enhance the brightness of the three-dimensional scene while maintaining the geometric structure of the scene.

[0146] Each embodiment of the present application can realize adaptive brightening of a three-dimensional low-illuminance scene while maintaining the scene geometry by adaptively enhancing the illumination component of a Gaussian point in space. Depending on the loss function, the training process can be further divided into two steps: (1) Reconstructing the three-dimensional scene using a low-illuminance image, and completing step S200 after execution. (2) Improving the illumination component based on fixed scene geometry parameters, thereby achieving adaptive brightness improvement of the scene, and completing step S300 after execution.

[0147] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0148] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional low-light scene brightening method based on 3D Gaussian sputtering is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0150] Step S100, decoupling the appearance geometry of the three-dimensional scene, including steps S110 to S120. Step S110, inputting 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing the 3D Gaussian; step S120, setting a number of pivots in the three-dimensional space, the attributes of the pivots include pivot position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the pivots, geometric attributes for determining the geometric features of Gaussian points associated with the pivots, and appearance attributes for determining the color features of Gaussian points associated with the pivots, and the initialization position of the pivot is determined according to the point cloud.

[0151] Step S200, decomposition of illumination component and reflection component of three-dimensional scene, including steps S210 to S220. Step S210, extracting reflection component and illumination component involved in rendering from appearance attribute, extracting scaling attribute, rotation attribute and opacity attribute involved in rendering from geometric attribute; Step S220, optimizing pivot position attribute, Gauss point positioning attribute, appearance attribute and geometric attribute according to the difference between 2D image and 3D Gaussian rendering image under corresponding viewing angle.

[0152] Step S300, adaptive brightness enhancement based on gamma transformation, includes steps S310 to S320. Step S310, extracting gamma value from appearance attribute by multi-layer perceptron, enhancing illumination component by power of gamma value, obtaining enhanced illumination component; Step S320, multiplying reflection component by enhanced illumination component to obtain Gaussian point color, obtaining brightness enhanced image under different viewing angles by volume rendering, obtaining brightness enhancement loss according to brightness enhanced image, and optimizing multi-layer perceptron by brightness enhancement loss.

[0153] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0154] Step S100, decoupling the appearance geometry of the three-dimensional scene, including steps S110 to S120. Step S110, inputting 2D images of the same scene from different perspectives into the colmap software to obtain a point cloud for initializing the 3D Gaussian; step S120, setting a number of pivots in the three-dimensional space, the attributes of the pivots include pivot position attributes, Gaussian point positioning attributes for determining the positions of Gaussian points associated with the pivots, geometric attributes for determining the geometric features of Gaussian points associated with the pivots, and appearance attributes for determining the color features of Gaussian points associated with the pivots, and the initialization position of the pivot is determined according to the point cloud.

[0155] Step S200, decomposition of illumination component and reflection component of three-dimensional scene, including steps S210 to S220. Step S210, extracting reflection component and illumination component involved in rendering from appearance attribute, extracting scaling attribute, rotation attribute and opacity attribute involved in rendering from geometric attribute; Step S220, optimizing pivot position attribute, Gauss point positioning attribute, appearance attribute and geometric attribute according to the difference between 2D image and 3D Gaussian rendering image under corresponding viewing angle.

[0156] Step S300, adaptive brightness enhancement based on gamma transformation, includes steps S310 to S320. Step S310, extracting gamma value from appearance attribute by multi-layer perceptron, enhancing illumination component by power of gamma value, obtaining enhanced illumination component; Step S320, multiplying reflection component by enhanced illumination component to obtain Gaussian point color, obtaining brightness enhanced image under different viewing angles by volume rendering, obtaining brightness enhancement loss according to brightness enhanced image, and optimizing multi-layer perceptron by brightness enhancement loss.

[0157] In this embodiment, the computer program product includes a program code portion for executing the steps of the three-dimensional low-light scene brightening method based on 3D Gaussian sputtering in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or the functionality may be provided for download with the aid of a hardware description language.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0160] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering, characterized in that: include: Input 2D images of the same scene from different perspectives into the colmap software to obtain the point cloud used to initialize the 3D Gaussian; A plurality of pivots are set in a three-dimensional space, wherein the attributes of the pivots include pivot position attributes, Gauss point positioning attributes for determining the positions of Gauss points associated with the pivots, geometric attributes for determining geometric features of Gauss points associated with the pivots, and appearance attributes for determining color features of Gauss points associated with the pivots, and the initialization positions of the pivots are determined according to the point cloud; Extracting the reflection component and illumination component involved in rendering from the appearance attributes, and extracting the scaling attribute, rotation attribute and opacity attribute involved in rendering from the geometric attributes; Optimizing the pivot point position attribute, the Gauss point positioning attribute, the appearance attribute, and the geometric attribute according to the difference between the 2D image and the 3D Gaussian rendering image under the corresponding viewing angle; Extracting a gamma value from the appearance attribute using a multi-layer perceptron, enhancing the illumination component according to a power of the gamma value, and obtaining an enhanced illumination component; The reflection component is multiplied by the enhanced illumination component to obtain the Gaussian point color, brightness enhanced images under different viewing angles are obtained through volume rendering, brightness enhancement loss is obtained according to the brightness enhanced image, and the multi-layer perceptron is optimized using the brightness enhancement loss.

2. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The attributes of the fulcrum also include fulcrum characteristic attributes, which are used to characterize the attributes of Gaussian points associated with the fulcrum. The shapes of the appearance attributes and geometric attributes are the same as the fulcrum characteristic attributes.

3. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The optimization process of optimizing the pivot point location attribute, the Gauss point positioning attribute, the appearance attribute, and the geometric attribute is implemented using a total loss: L total =λ1L1+λ2L d L total , represents the total loss; λ1 and λ2 are both constants; L1, represents the difference, which is the average of the absolute difference between the 2D image and the 3D Gaussian rendered image at the corresponding viewing angle; L d , represents the depth loss.

4. The three-dimensional low-light scene brightening method according to claim 3, characterized in that: The depth loss is obtained using the following formula: L d , represents the depth loss; i, j, represent any two Gaussian points; w i , represents the volume rendering weight; z j , represents the distance from the Gaussian point to the imaging plane.

5. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The illumination component is enhanced according to the power of the gamma value to obtain an enhanced illumination component, which is implemented using the following formula: In the formula, i′ i , represents the enhanced illumination component; i i , represents the illumination component before enhancement; γ i , represents the gamma value.

6. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The brightness enhancement loss is obtained according to the brightness enhancement image, and is implemented using the following formula: L enhance =E[(C e -m) 2 ] In the formula, L enhance , indicating brightness enhancement loss; E, means taking the average value; C e , represents the brightness enhanced image; m is a constant.

7. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The three-dimensional low-illumination scene brightening method further includes: A reference image is obtained by rendering according to the reflection component, and the brightness enhancement loss is compensated by using the difference between the reference image and the brightness enhancement image.

8. The three-dimensional low-light scene brightening method according to claim 1, characterized in that: The three-dimensional low-illumination scene brightening method further includes: A reference image is obtained by rendering the reflection component, and the brightness enhancement loss is supplemented by using the difference between the reference image and the brightness enhancement image, which is specifically implemented by the following formula: In the formula, L enhance , indicating brightness enhancement loss; E, means taking the average value; C e , represents the brightness enhanced image; m, is a constant; λ is a constant; R, represents the reference image; var, represents variance; α is a constant.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instruction is executed by the processor, the steps of the three-dimensional low-illumination scene brightening method based on 3D Gaussian sputtering as described in any one of claims 1 to 8 are implemented.

Images