Weak light 3D Gaussian sputtering method based on color correction transformation

Abstract

The application discloses a weak light 3D Gaussian sputtering method based on color correction transformation, and steps are as follows: step 1, multi-angle images of weak light and normal light scenes are collected respectively, all 3D Gaussian ellipsoids in the corresponding scenes are fitted respectively, and original spherical harmonic color parameters of the corresponding scenes are obtained; step 2, a generation network is constructed; step 3, a discrimination network is constructed; step 4, a loss function is constructed; step 5, the generation network and the discrimination network weights are alternately updated according to the loss function, iterative training is completed, a trained network is obtained, original spherical harmonic color parameters of a weak light scene to be corrected are input into the trained network, corrected spherical harmonic color parameters are obtained, the original spherical harmonic color parameters are replaced by the corrected spherical harmonic color parameters, and a rendered image is synthesized. The method solves the problem of poor quality of a rendered picture after transformation of an existing method.

Description

Technical Field

[0001] This invention belongs to the field of low-light image processing technology, specifically relating to a low-light 3D Gaussian sputtering method based on color correction transformation. Background Technology

[0002] 3D Gaussian sputtering is currently the mainstream 3D scene reconstruction and rendering solution. It fits the scene's geometry and lighting color information using a large number of 3D Gaussian ellipsoids, and uses spherical harmonics to store the color and lighting parameters of each ellipsoid, achieving smooth rendering from any viewpoint. Existing technologies generally use linear transformations to correct the spherical harmonic color parameters. However, in low-light scenes such as indoor low-light environments or outdoor nighttime scenes, the captured images have low brightness and unbalanced contrast. Rendering with traditional linear spherical harmonic transformations results in poor image quality, such as cross-viewpoint color drift, loss of detail in dark areas, and overall color distortion. Summary of the Invention

[0003] The purpose of this invention is to provide a low-light 3D Gaussian sputtering method based on color correction transformation, which solves the problem of poor image quality after transformation in existing methods.

[0004] The technical solution adopted in this invention is a low-light 3D Gaussian sputtering method based on color correction transformation, and the steps are as follows: Step 1: Collect multi-angle images of low-light and normal-light scenes respectively, fit all 3D Gaussian ellipsoids in the corresponding scene respectively, and obtain the original spherical harmonic color parameters of the corresponding scene. Step 2, construct the generative network; Step 3: Construct the discriminant network; Step 4, construct the loss function; Step 5: Alternately update the generation network and determine the network weights according to the loss function to complete iterative training and obtain the trained network. Input the original spherical harmonic color parameters of the low-light scene to be corrected into the trained network to obtain the corrected spherical harmonic color parameters. Replace the original spherical harmonic color parameters with the corrected spherical harmonic color parameters and synthesize the rendered image.

[0005] The invention is further characterized by: The specific process of step 1 is as follows: Use a camera to take multi-angle pictures of low-light scenes and normal lighting scenes, and use a 3D Gaussian sputtering reconstruction tool to fit all 3D Gaussian ellipsoids in the low-light scene and the 3D Gaussian ellipsoids in the normal lighting scene respectively, to obtain the original spherical harmonic color parameters of the low-light scene and the original spherical harmonic color parameters of the normal lighting scene.

[0006] In step 2, the generated network consists of first-level encoding convolution, second-level encoding convolution, block-restricted self-attention module, first-level decoding convolution, second-level decoding convolution and residual connection; The first-level coding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels. The second-level coding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 64 output channels; The first-level decoding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels. The second-level decoding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 3 output channels; The process of generating a network is as follows: The original spherical harmonic color parameters for low-light scenes are transformed into a convolutional standard channel-first format through dimensionality permutation. Where SH represents the original spherical harmonic color parameters for a low-light scene, and X represents the feature after dimension permutation, with dimensions [N, 3, 16]. The dimension-permutated feature X is used as the input of the first-level encoding convolution. The output of the first-level encoding convolution is used as the input of the second-level encoding convolution. The output of the second-level encoding convolution is used as the input of the block-restricted self-attention module. The output of the block-restricted self-attention module and the output of the second-level encoding convolution are concatenated and used as the input of the first-level decoding convolution. The output of the first-level decoding convolution and the output of the first-level encoding convolution are concatenated and used as the input of the second-level decoding convolution. The output of the second-level decoding convolution is concatenated with the dimension-permutated feature X and used as the final output feature. The final output feature permutation dimension is restored to the spherical harmonic standard format. , among which, SH end This represents the corrected low-light spherical harmonic color parameters, with dimensions [N, 16, 3].

[0007] In step 2, the processing procedure of the block-restricted self-attention module is as follows: Step A: The output of the second-level coding convolution is split by a one-dimensional convolution of size 1×1 to obtain the query matrix Q, the key matrix K, and the value matrix V. The output channels of the query and key branches are both 1 / 2 of the input channels, and the output channels of the value branches are consistent with the input channels. Step B: Calculate the raw attention score; The expression is:

[0008] In the formula, S is the original attention score. To query the feature channel dimension; Step C: Use the mask matrix M to mask the cross-order scores to obtain the attention scores after masking the cross-order correlations; The expression is:

[0009] In the formula, M represents the block binary mask matrix, and S masked This represents the attention score after masking cross-order associations; This indicates that the scores for false positions in the block binary mask matrix are set to a minimum value; Step D: The softmax function is used to evaluate the attention score S after masking cross-order associations. masked Normalization is performed to obtain the normalized attention score A; The expression is:

[0010] Step E involves weighted fusion of the normalized attention score A and the value matrix V to obtain the intermediate features. .

[0011] In step C, the block binary mask matrix is:

[0012] In the formula, 1 represents a submatrix of all ones, 0 represents a submatrix of all zeros, all values ​​in the diagonal blocks are true, and all values ​​in the cross-order regions are false.

[0013] In step 3, the discriminant network consists of 4 independent discriminant branches; Each independent discriminant branch consists of a flattening layer, a first fully connected layer, a layer normalization layer, a LeakyReLU activation layer, a Dropout random deactivation layer, and a second fully connected layer. The processing procedure for the discriminant network is as follows: The corrected low-light spherical harmonic color parameters are all sliced ​​according to the inherent length of the spherical harmonics to obtain the sliced ​​features Z0, Z1, Z2, and Z3. The expression is:

[0014] In the formula, Z0 represents the 0th-order basic brightness feature in the corrected low-light spherical harmonic color parameters, which contains only one set of spherical harmonic color parameters; Z1 represents the 1st-order low-frequency global illumination feature in the corrected low-light spherical harmonic color parameters, which contains three sets of spherical harmonic color parameters; Z2 represents the 2nd-order mid-frequency color feature in the corrected low-light spherical harmonic color parameters, which contains five sets of spherical harmonic color parameters; and Z3 represents the 3rd-order high-frequency detail feature in the corrected low-light spherical harmonic color parameters, which contains seven sets of spherical harmonic color parameters. The split feature Z0 is used as the input to the first independent discriminant branch, the split feature Z1 is used as the input to the second independent discriminant branch, the split feature Z2 is used as the input to the third independent discriminant branch, and the split feature Z3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. ; The original spherical harmonic color parameters of the normal lighting scene are sliced ​​and split according to the inherent length of the spherical harmonics to obtain the split features N0, N1, N3, and N4.

[0015] In the formula, N0 represents the 0th-order basic brightness feature in the original spherical harmonic color parameters of a normal lighting scene, containing only 1 set of spherical harmonic color parameters; N1 represents the 1st-order low-frequency global illumination feature in the original spherical harmonic color parameters of a normal lighting scene, containing 3 sets of spherical harmonic color parameters; N2 represents the 2nd-order mid-frequency color feature in the original spherical harmonic color parameters of a normal lighting scene, containing 5 sets of spherical harmonic color parameters; and N3 represents the 3rd-order high-frequency detail feature in the original spherical harmonic color parameters of a normal lighting scene, containing 7 sets of spherical harmonic color parameters. The original spherical harmonic color parameters for a normal lighting scene; After splitting, feature N0 is used as the input to the first independent discriminant branch, feature N1 is used as the input to the second independent discriminant branch, feature N2 is used as the input to the third independent discriminant branch, and feature N3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. .

[0016] In step 4, the loss functions include adversarial loss function, image regression loss function, spatial contrast constraint loss function, and total loss function; The adversarial loss function consists of discriminative network loss and generative network loss. Determine network losses The expression is:

[0017] In the formula, For the first The original spherical harmonic color parameters of a normally lit scene are discriminated and output. For the first Discrimination output of low-light spherical harmonic color parameters after branch correction; The weight vector is the order of the decision; K is the number of independent decision branches, K=3; Calculate the average of batch samples corresponding to the original spherical harmonic color parameters in a normal lighting scene; Calculate the batch sample mean value corresponding to the corrected low-light spherical harmonic color parameters; The regularization coefficient; For cross-order illumination consistency regularization loss;

[0018] In the formula, Calculate the mean of a batch of samples; Generate network loss The expression is:

[0019] The expression for the image regression loss function is:

[0020] In the formula, For balancing weighting coefficients; For L1 pixel loss; For SSIM structural similarity loss; The original spherical harmonic color parameters of the low-light scene are rendered using 3DGS technology to obtain the original low-light rendering image. The corrected low-light spherical harmonic color parameters are then rendered using 3DGS technology to obtain the corrected low-light rendering image. Multi-angle images of the acquired low-light scene are processed using existing image enhancement algorithms to obtain color-enhanced images, which are then used for supervised training. The image regression loss is calculated using the expression of the image regression loss function on the original low-light rendered image and the acquired multi-angle images of the low-light scene, thus obtaining the low-light original image regression loss. ; The image regression loss is calculated using the expression of the image regression loss function on the corrected low-light rendered image and the color-enhanced image to obtain the enhanced image regression loss. ; Spatial contrast constraint loss function The expression is:

[0021]

[0022]

[0023] In the formula, SThis is the preset baseline contrast constant; The average brightness of all pixels in the original low-light rendered image; , , , These represent the brightness gradient convolution operators in the left, right, top, and bottom directions, respectively. K The total number of pixels sampled in the image; x, y () represents the image pixel space coordinates. x The coordinates are in the horizontal direction. y These are the vertical coordinates, and their values ​​cover all pixel positions in the image. To adjust the pixel brightness of the original rendered image in low light conditions. To correct the pixel brightness of the rendered image in low light; For frame-adaptive contrast coefficients; Total loss function The expression is: .

[0024] In step 5, the training process adopts an alternating training strategy, that is, first calculate the discriminant network loss to update the discriminant network, then calculate the total loss to update the generator network, and after iterating to a preset number of times, save the entire set of generator network and discriminant network weights to obtain the trained network.

[0025] The beneficial effects of this invention are: (1) The method of the present invention uses a one-dimensional spatial self-attention encoding and decoding generation network to directly perform nonlinear mapping on the 16-dimensional spherical harmonic color parameters, breaking through the limitations of the traditional linear transformation expression ability. It can autonomously learn the illumination correlation features between the 0th to 3rd order spherical harmonics, greatly improving the cross-view color drift problem in low-light scenes. It adopts a discriminant network with four independent branches to constrain four types of features: basic brightness features, low-frequency global illumination features, mid-frequency color features, and high-frequency detail features. The adversarial training constraints are more refined, and the corrected spherical harmonic illumination layers are more natural. The total loss simultaneously integrates four types of supervision signals: pixel error, image structure, local contrast, and adversarial constraints. While ensuring that the image color matches the true value, it prevents excessive enhancement of dark areas and brightness discontinuity in low-light scenes. (2) The method of the present invention directly acts on the 3D Gaussian spherical harmonic color parameters without modifying the underlying rendering logic of 3D Gaussian sputtering. It can be directly embedded into the existing reconstruction pipeline and is suitable for various low-light reconstruction scenarios such as indoor low light and nighttime outdoor scenes. It has strong versatility. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the network framework generated in the method of the present invention; Figure 3 This is a schematic diagram of the discrimination network framework in the method of the present invention; Figure 4 This is a comparison diagram of the low-light scene rendering effects of the method of the present invention and the traditional linear transformation in Embodiment 7 of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0028] Example 1 This invention relates to a low-light 3D Gaussian sputtering method based on color correction transformation, such as... Figure 1 As shown, the steps are as follows: Step 1: Collect multi-angle images of low-light and normal-light scenes respectively, fit all 3D Gaussian ellipsoids in the corresponding scenes respectively, and obtain the original spherical harmonic color parameters of the low-light scene and the original spherical harmonic color parameters of the normal-light scene. Step 2: Construct a generative network to generate spherical harmonic color parameters for the corrected low-light scene; Step 3: Construct the discriminant network; Step 4, construct the loss function; Step 5: Alternately update the generation network and determine the network weights according to the loss function to complete iterative training and obtain the trained network. Input the original spherical harmonic color parameters of the low-light scene to be corrected into the trained network to obtain the corrected spherical harmonic color parameters. Replace the original spherical harmonic color parameters with the corrected spherical harmonic color parameters and synthesize the rendered image.

[0029] Example 2 Based on Example 1, the specific process of step 1 is as follows: use a consumer-grade camera to take multi-angle pictures of nighttime outdoor scenes, low-light indoor scenes, and normal lighting scenes, and use a 3D Gaussian sputtering reconstruction tool to fit all 3D Gaussian ellipsoids in the low-light scene and the 3D Gaussian ellipsoids in the normal lighting scene to obtain the original spherical harmonic color parameters of the low-light scene and the original spherical harmonic color parameters of the normal lighting scene. Each 3D Gaussian ellipsoid corresponds to a set of spherical harmonic color parameters of order 0 to 3 with a total length of 16; The original spherical harmonic color parameters of normal lighting scenes are used for network training as ground truth samples; Low-light scenes include at least one of the following: nighttime streets, underground parking garages, tunnels, and twilight forests; The 3D Gaussian sputtering reconstruction tool is an existing technology, based on the 3D GaussianSplatting algorithm proposed by Kerbl et al. in 2023.

[0030] Example 3 Based on Example 2, in step 2, as follows: Figure 2 As shown, the generator network consists of first-level encoding convolution, second-level encoding convolution, block-restricted self-attention module, first-level decoding convolution, second-level decoding convolution and residual connection, which is adapted to the low-light spherical harmonic nonlinear color correction requirements. The first-level coding convolution consists of a one-dimensional convolutional layer (conv1d) and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels. The second-level coding convolution consists of a one-dimensional convolutional layer (conv1d) and a GELU activation function, with a kernel size of 3, padding of 1, and 64 output channels; The first-level decoding convolution consists of a one-dimensional convolutional layer (conv1d) and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels; The second-level decoding convolution consists of a one-dimensional convolutional layer (conv1d) and a GELU activation function, with a kernel size of 3, padding of 1, and 3 output channels; The processing procedure of the block-restricted self-attention module is as follows: Step A: The output of the second-level coding convolution (the feature dimension is [N,C,16], where N is the number of 3D Gaussian points, C is the number of feature channels, and 16 corresponds to the length of the 0th to 3rd order original spherical harmonic color parameters) is split by a one-dimensional convolution of size 1×1 to obtain the query matrix Q, the key matrix K, and the value matrix V. The output channels of the query and key branches are both 1 / 2 of the input channels, and the output channels of the value branches are consistent with the input channels. Step B: Calculate the raw attention score; The expression is:

[0031] In the formula, S is the original attention score. To query the feature channel dimension; Step C: Use the block binary mask matrix M to mask the cross-order scores to obtain the attention scores after masking the cross-order correlations; The expression is:

[0032] In the formula, S masked This represents the attention score after masking cross-order associations; This indicates that the scores for false positions in the block binary mask matrix are set to a minimum value; The block binary mask matrix is ​​as follows:

[0033] In the formula, 1 represents a submatrix of all ones, 0 represents a submatrix of all zeros, all values ​​in the diagonal block are true, and all values ​​in the cross-order region are false; The block binary mask matrix is ​​a 16×16 matrix. It is divided into independent calculation regions according to the four lengths [1,3,5,7] of the original spherical harmonic color parameters of the low-light scene. Cross-order feature matching is masked during the operation. Step D: The softmax function is used to evaluate the attention score S after masking cross-order associations. masked Normalization is performed to obtain the normalized attention score A; The expression is:

[0034] Step E involves weighted fusion of the normalized attention score A and the value matrix V to obtain the intermediate features. ; The process of generating a network is as follows: The original spherical harmonic color parameters (dimension [N, 16, 3]) of the low-light scene are transformed into a convolutional standard channel-first format by performing dimension permutation. Where SH represents the original spherical harmonic color parameters for a low-light scene, and X represents the feature after dimension permutation, with dimensions [N, 3, 16]. The dimension-permutated feature X is used as the input of the first-level encoding convolution, the output of the first-level encoding convolution is used as the input of the second-level encoding convolution, the output of the second-level encoding convolution is used as the input of the block-restricted self-attention module, the output of the block-restricted self-attention module is concatenated with the output of the second-level encoding convolution and used as the input of the first-level decoding convolution, the output of the first-level decoding convolution is concatenated with the output of the first-level encoding convolution and used as the input of the second-level decoding convolution, and the output of the second-level decoding convolution is concatenated with the dimension-permutated feature X and used as the final output feature. In the initial stage of training, the output of the second-level decoding convolution approaches 0, and the final output feature approaches the dimension-permutated feature X. The correction magnitude approaches 0, avoiding direct overexposure distortion of weak light spherical harmonics. The final output feature permutation dimension is restored to the spherical harmonic standard format. , among which, SH end This represents the corrected low-light spherical harmonic color parameters, with dimensions [N, 16, 3].

[0035] The generative network introduces a block-restricted self-attention module that only allows interaction of spherical harmonic features of the same order and blocks invalid cross-order associations. It specifically mines the effective illumination dependence of 0th to 3rd order spherical harmonics in low light and outputs low-light spherical harmonic color parameters after nonlinear correction.

[0036] Example 4 Based on Example 3, in step 3, as follows: Figure 3 As shown, the discriminant network consists of four independent discriminant branches; Each independent discriminant branch consists of a flattened layer, a first fully connected layer (Linear), a layer normalization layer (LayerNorm), a LeakyReLU activation layer, a Dropout random deactivation layer, and a second fully connected layer (Linear). The first fully connected layer in the first independent discriminative branch has an input dimension of 3 and an output dimension of 64 (the number of intermediate channels in this branch). The second fully connected layer has an input dimension of 64 (the number of intermediate channels in this branch) and a fixed output dimension of 1. The first fully connected layer in the second independent discriminant branch has an input dimension of 9 and an output dimension of 32 (the number of intermediate channels in this branch). The second fully connected layer has an input dimension of 32 (the number of intermediate channels in this branch) and an output dimension that is fixed at 1. The first fully connected layer in the third independent discriminative branch has an input dimension of 15 and an output dimension of 64 (the number of intermediate channels in this branch). The second fully connected layer has an input dimension of 64 (the number of intermediate channels in this branch) and a fixed output dimension of 1. The first fully connected layer in the fourth independent discriminant branch has an input dimension of 321 and an output dimension of 64 (the number of intermediate channels in this branch). The second fully connected layer has an input dimension of 64 (the number of intermediate channels in this branch) and a fixed output dimension of 1. The negative slope of the LeakyReLU activation layer is uniformly set to 0.2, and the Dropout inactivation probability is fixed at 0.1 to suppress training overfitting; The parameters of the four discrimination branches are completely independent and not shared; The processing procedure for the discriminant network is as follows: The corrected low-light spherical harmonic color parameters are all sliced ​​according to the inherent length of the spherical harmonics to obtain the sliced ​​features Z0, Z1, Z2, and Z3. The expression is:

[0037] In the formula, Z0 is the 0th-order basic brightness feature in the corrected low-light spherical harmonic color parameters, which contains only one set of spherical harmonic color parameters and corresponds to the basic brightness and darkness benchmark of the image. It is the feature most sensitive to human vision. Z1 is the 1st-order low-frequency global illumination feature in the corrected low-light spherical harmonic color parameters, which contains three sets of spherical harmonic color parameters and controls the large-scale light and shadow shift of the scene. Z2 is the 2nd-order mid-frequency color feature in the corrected low-light spherical harmonic color parameters, which contains five sets of spherical harmonic color parameters and determines the basic color transition effect of the object. Z3 is the 3rd-order high-frequency detail feature in the corrected low-light spherical harmonic color parameters, which contains seven sets of spherical harmonic color parameters and carries the light and shadow information of the object's edge and small texture. The split feature Z0 is used as the input to the first independent discriminant branch, the split feature Z1 is used as the input to the second independent discriminant branch, the split feature Z2 is used as the input to the third independent discriminant branch, and the split feature Z3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. ; These respectively characterize the confidence level that the corresponding order spherical harmonics conform to the normal illumination distribution; The original spherical harmonic color parameters of the normal lighting scene are sliced ​​and split according to the inherent length of the spherical harmonics to obtain the split features N0, N1, N3, and N4.

[0038] In the formula, N0 represents the 0th-order basic brightness feature in the original spherical harmonic color parameters of a normal lighting scene, containing only 1 set of spherical harmonic color parameters; N1 represents the 1st-order low-frequency global illumination feature in the original spherical harmonic color parameters of a normal lighting scene, containing 3 sets of spherical harmonic color parameters; N2 represents the 2nd-order mid-frequency color feature in the original spherical harmonic color parameters of a normal lighting scene, containing 5 sets of spherical harmonic color parameters; and N3 represents the 3rd-order high-frequency detail feature in the original spherical harmonic color parameters of a normal lighting scene, containing 7 sets of spherical harmonic color parameters. The original spherical harmonic color parameters for a normal lighting scene; After splitting, feature N0 is used as the input to the first independent discriminant branch, feature N1 is used as the input to the second independent discriminant branch, feature N2 is used as the input to the third independent discriminant branch, and feature N3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. .

[0039] Example 5 Based on Example 4, in step 4, the loss function includes adversarial loss function, image regression loss function, spatial contrast constraint loss function, and total loss function; The adversarial loss function consists of discriminative network loss and generative network loss. Determine network losses The expression is:

[0040] In the formula, For the first The original spherical harmonic color parameters of the branched normal lighting scene are discriminated and output, i=0,……,3; For the first The discriminant output of the low-light spherical harmonic color parameters after branch correction, i=0,……,3; Let i be the weight vector of order, i = 0, ..., 3. =1.0, =0.6, =0.4, =0.4, set according to the characteristics of human visual perception, to increase the constraint weight of the 0th order brightness feature; K is the number of independent discriminant branches, K=3; Calculate the average of batch samples corresponding to the original spherical harmonic color parameters in a normal lighting scene; Calculate the batch sample mean value corresponding to the corrected low-light spherical harmonic color parameters; The regularization coefficient is... ; A cross-order illumination consistency regularization loss is used as an additional constraint term in the discrimination loss. Only a weak constraint is applied to avoid disrupting the independent discrimination logic across different orders. This addresses the problem of fragmented illumination variation trends from orders 0 to 3 after weak light enhancement. Constrain the difference in the mean value of the output of different order spherical harmonics to ensure the continuity of the trend of brightness, lighting and detail changes after correction in low light scenes, and avoid the fragmentation of the rendered image layers.

[0041] In the formula, Calculate the mean of a batch of samples; Generate network loss The expression is:

[0042] The expression for the image regression loss function is:

[0043] In the formula, For balancing weighting coefficients; For L1 pixel loss; For SSIM structural similarity loss; The original spherical harmonic color parameters of the low-light scene are rendered using 3DGS technology to obtain the original low-light rendering image. The corrected low-light spherical harmonic color parameters are then rendered using 3DGS technology to obtain the corrected low-light rendering image. The multi-angle images of the acquired low-light scene are processed using existing image enhancement algorithms to obtain color-enhanced images, which are then used for supervised training. Existing image enhancement algorithms can be generated using mature existing image enhancement algorithms such as EnlightenGAN (Jiang Y, et al. EnlightenGAN: Deep Light Enhancement without Paired Supervision. IEEE TIP, 2021)); The image regression loss is calculated using the expression of the image regression loss function on the original low-light rendered image and the acquired multi-angle images of the low-light scene, thus obtaining the low-light original image regression loss. ; The image regression loss is calculated using the expression of the image regression loss function on the corrected low-light rendered image and the color-enhanced image to obtain the enhanced image regression loss. ; Spatial contrast constraint loss function The expression is:

[0044]

[0045]

[0046] In the formula, S This is the preset baseline contrast constant; The average brightness of all pixels in the original low-light rendered image; , , , These represent the brightness gradient convolution operators in the left, right, top, and bottom directions, respectively. K The total number of pixels sampled in the image; x, y () represents the image pixel space coordinates. x The coordinates are in the horizontal direction. y These are the vertical coordinates, and their values ​​cover all pixel positions in the image. To adjust the pixel brightness of the original rendered image in low light conditions. To correct the pixel brightness of the rendered image in low light; For frame-adaptive contrast coefficients; Frame-adaptive contrast coefficient The brightness of the image changes dynamically with the overall brightness of the original image; the lower the brightness of the image in low light, the better. The larger the value, the stronger the contrast constraint, thereby constraining the local brightness and darkness changes between the original image and the enhanced image, and avoiding brightness banding and contrast distortion in the rendered image. Total loss function The expression is: .

[0047] Example 6 Based on Example 5, in step 5, the training process adopts an alternating training strategy, that is, first calculate the discriminant network loss to update the discriminant network, then calculate the total loss to update the generator network, and after iterating to a preset number of times, save the entire set of generator network and discriminant network weights to obtain the trained network.

[0048] When training the discriminant network, the network loss is determined. As an optimization objective, the discriminant output of the original spherical harmonic color parameters in a normal lighting scene is simultaneously constrained to approach 1, while the discriminant output of the corrected low-light spherical harmonic color parameters approaches -1. When training the generative network, only the loss function of the generative network is considered. As an objective of the adversarial optimization, the discrimination outputs of each order of the corrected low-light spherical harmonic color parameters are made as close as possible to the discrimination scores of the original spherical harmonic color parameters in a normal lighting scene.

[0049] During training, the network training hyperparameters were uniformly configured, including the learning rate for various 3D Gaussian parameters, the Gaussian denser iteration interval, and the start and end iteration limits. Specific parameter values ​​are as follows: initial learning rate for position parameters: 0.00004; termination learning rate: 0.000002; delay multiplier: 0.01; maximum number of steps: 10000; spherical harmonic feature learning rate: 0.0025; opacity learning rate: 0.05; scaling parameter learning rate: 0.005; rotation parameter learning rate: 0.001; Gaussian denser interval: 100 steps; opacity reset interval: 3000 steps; denser initial iteration: 500 steps; denser termination iteration: 6000 steps; position gradient threshold: 0.00035; structural similarity loss weight: 0.2. A GPU with 24GB of VRAM and the ADAM optimizer were selected for training, with a total of 10000 training steps. The composite rendered images were completed using the 3D Gaussian Splatting standard rendering pipeline proposed by Kerbl et al. in 2023 to synthesize images from new perspectives.

[0050] Example 7 Images were synthesized and rendered using Aleth-NeRF, Luminance-GS, and the method of this invention for the shrub, sofa, bike, and chair scenes in the LOM dataset, respectively. Figure 4As shown, the method of the present invention effectively suppresses cross-viewpoint color drift and artifacts; compared with Luminance-GS, the method of the present invention enhances detail clarity while maintaining natural lighting transition, and avoids color distortion caused by over-enhancement; it verifies the effectiveness of the present invention in directly acting on spherical harmonic parameters and adapting to low-light scenes without modifying the underlying rendering logic.

Claims

1. A low-light 3D Gaussian sputtering method based on color correction transformation, characterized in that, The steps are as follows: Step 1: Collect multi-angle images of low-light and normal-light scenes respectively, fit all 3D Gaussian ellipsoids in the corresponding scene respectively, and obtain the original spherical harmonic color parameters of the corresponding scene. Step 2, construct the generative network; Step 3: Construct the discriminant network; Step 4, construct the loss function; Step 5: Alternately update the generation network and determine the network weights according to the loss function to complete iterative training and obtain the trained network. Input the original spherical harmonic color parameters of the low-light scene to be corrected into the trained network to obtain the corrected spherical harmonic color parameters. Replace the original spherical harmonic color parameters with the corrected spherical harmonic color parameters and synthesize the rendered image.

2. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 1, characterized in that, The specific process of step 1 is as follows: Use a camera to take multi-angle pictures of low-light scenes and normal lighting scenes, and use a 3D Gaussian sputtering reconstruction tool to fit all 3D Gaussian ellipsoids in the low-light scene and the 3D Gaussian ellipsoids in the normal lighting scene respectively, to obtain the original spherical harmonic color parameters of the low-light scene and the original spherical harmonic color parameters of the normal lighting scene.

3. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 1, characterized in that, In step 2, the generated network consists of first-level encoding convolution, second-level encoding convolution, block-restricted self-attention module, first-level decoding convolution, second-level decoding convolution and residual connection; The first-level coding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels. The second-level coding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 64 output channels; The first-level decoding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 32 output channels. The second-level decoding convolution consists of a one-dimensional convolutional layer and a GELU activation function, with a kernel size of 3, padding of 1, and 3 output channels; The process of generating a network is as follows: The original spherical harmonic color parameters for low-light scenes are transformed into a convolutional standard channel-first format through dimensionality permutation. Where SH represents the original spherical harmonic color parameters for a low-light scene, and X represents the feature after dimension permutation, with dimensions [N, 3, 16]. The dimension-permutated feature X is used as the input of the first-level encoding convolution. The output of the first-level encoding convolution is used as the input of the second-level encoding convolution. The output of the second-level encoding convolution is used as the input of the block-restricted self-attention module. The output of the block-restricted self-attention module and the output of the second-level encoding convolution are concatenated and used as the input of the first-level decoding convolution. The output of the first-level decoding convolution and the output of the first-level encoding convolution are concatenated and used as the input of the second-level decoding convolution. The output of the second-level decoding convolution is concatenated with the dimension-permutated feature X and used as the final output feature. The final output feature permutation dimension is restored to the spherical harmonic standard format. , among which, SH end This represents the corrected low-light spherical harmonic color parameters, with dimensions [N, 16, 3].

4. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 3, characterized in that, In step 2, the processing procedure of the block-restricted self-attention module is as follows: Step A: The output of the second-level coding convolution is split by a one-dimensional convolution of size 1×1 to obtain the query matrix Q, the key matrix K, and the value matrix V. The output channels of the query and key branches are both 1 / 2 of the input channels, and the output channels of the value branches are consistent with the input channels. Step B: Calculate the raw attention score; The expression is: In the formula, S is the original attention score. To query the feature channel dimension; Step C: Use the block binary mask matrix M to mask the cross-order scores to obtain the attention scores after masking the cross-order correlations; The expression is: In the formula, S masked This represents the attention score after masking cross-order associations; This indicates that the scores for false positions in the block binary mask matrix are set to a minimum value; Step D: Use the softmax function to evaluate the attention score S after masking cross-order associations. masked Normalization is performed to obtain the normalized attention score A; The expression is: Step E involves weighted fusion of the normalized attention score A and the value matrix V to obtain the intermediate features. .

5. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 4, characterized in that, In step C, the block binary mask matrix is: In the formula, 1 represents a submatrix of all ones, 0 represents a submatrix of all zeros, all values ​​in the diagonal blocks are true, and all values ​​in the cross-order regions are false.

6. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 4, characterized in that, In step 3, the discriminant network consists of four independent discriminant branches; Each independent discriminant branch consists of a flattening layer, a first fully connected layer, a layer normalization layer, a LeakyReLU activation layer, a Dropout random deactivation layer, and a second fully connected layer. The processing procedure for the discriminant network is as follows: The corrected low-light spherical harmonic color parameters are all sliced ​​according to the inherent length of the spherical harmonics to obtain the sliced ​​features Z0, Z1, Z2, and Z3. The expression is: In the formula, Z0 represents the 0th-order basic brightness feature in the corrected low-light spherical harmonic color parameters, which contains only one set of spherical harmonic color parameters; Z1 represents the 1st-order low-frequency global illumination feature in the corrected low-light spherical harmonic color parameters, which contains three sets of spherical harmonic color parameters; Z2 represents the 2nd-order mid-frequency color feature in the corrected low-light spherical harmonic color parameters, which contains five sets of spherical harmonic color parameters; and Z3 represents the 3rd-order high-frequency detail feature in the corrected low-light spherical harmonic color parameters, which contains seven sets of spherical harmonic color parameters. The split feature Z0 is used as the input to the first independent discriminant branch, the split feature Z1 is used as the input to the second independent discriminant branch, the split feature Z2 is used as the input to the third independent discriminant branch, and the split feature Z3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. ; The original spherical harmonic color parameters of the normal lighting scene are sliced ​​and split according to the inherent length of the spherical harmonics to obtain the split features N0, N1, N3, and N4. In the formula, N0 represents the 0th-order basic brightness feature in the original spherical harmonic color parameters of a normal lighting scene, containing only 1 set of spherical harmonic color parameters; N1 represents the 1st-order low-frequency global illumination feature in the original spherical harmonic color parameters of a normal lighting scene, containing 3 sets of spherical harmonic color parameters; N2 represents the 2nd-order mid-frequency color feature in the original spherical harmonic color parameters of a normal lighting scene, containing 5 sets of spherical harmonic color parameters; and N3 represents the 3rd-order high-frequency detail feature in the original spherical harmonic color parameters of a normal lighting scene, containing 7 sets of spherical harmonic color parameters. The original spherical harmonic color parameters for a normal lighting scene; After splitting, feature N0 is used as the input to the first independent discriminant branch, feature N1 is used as the input to the second independent discriminant branch, feature N2 is used as the input to the third independent discriminant branch, and feature N3 is used as the input to the fourth independent discriminant branch. The output of the first independent discriminant branch is the first single-dimensional discriminant score. The output of the second independent discriminant branch is the second single-dimensional discriminant score. The output of the third independent discriminant branch is the third single-dimensional discriminant score. The output of the fourth independent discriminant branch is the fourth single-dimensional discriminant score. .

7. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 6, characterized in that, In step 4, the loss functions include adversarial loss function, image regression loss function, spatial contrast constraint loss function, and total loss function; The adversarial loss function consists of discriminative network loss and generative network loss. Determine network losses The expression is: In the formula, For the first The original spherical harmonic color parameters of a normally lit scene are discriminated and output. For the first Discrimination output of low-light spherical harmonic color parameters after branch correction; The weight vector is the order of the decision; K is the number of independent decision branches, K=3; Calculate the average of batch samples corresponding to the original spherical harmonic color parameters in a normal lighting scene; Calculate the batch sample mean value corresponding to the corrected low-light spherical harmonic color parameters; The regularization coefficient; For cross-order illumination consistency regularization loss; In the formula, Calculate the mean of a batch of samples; Generate network loss The expression is: The expression for the image regression loss function is: In the formula, For balancing weighting coefficients; For L1 pixel loss; For SSIM structural similarity loss; The original spherical harmonic color parameters of the low-light scene are rendered using 3DGS technology to obtain the original low-light rendering image. The corrected low-light spherical harmonic color parameters are then rendered using 3DGS technology to obtain the corrected low-light rendering image. Multi-angle images of the acquired low-light scene are processed using existing image enhancement algorithms to obtain color-enhanced images, which are then used for supervised training. The image regression loss is calculated using the expression of the image regression loss function on the original low-light rendered image and the acquired multi-angle images of the low-light scene, thus obtaining the low-light original image regression loss. ; The image regression loss is calculated using the expression of the image regression loss function on the corrected low-light rendered image and the color-enhanced image to obtain the enhanced image regression loss. ; Spatial contrast constraint loss function The expression is: In the formula, S This is the preset baseline contrast constant; The average brightness of all pixels in the original low-light rendered image; , , , These represent the brightness gradient convolution operators in the left, right, top, and bottom directions, respectively. K The total number of pixels sampled in the image; x, y () represents the image pixel space coordinates. x The coordinates are in the horizontal direction. y These are the vertical coordinates, and their values ​​cover all pixel positions in the image. To adjust the pixel brightness of the original rendered image in low light conditions. To correct the pixel brightness of the low-light rendered image; For frame-adaptive contrast coefficients; Total loss function The expression is: 。 8. The low-light 3D Gaussian sputtering method based on color correction transformation according to claim 7, characterized in that, In step 5, the training process adopts an alternating training strategy, that is, first calculate the discriminant network loss to update the discriminant network, then calculate the total loss to update the generator network, and after iterating to a preset number of times, save the entire set of generator network and discriminant network weights to obtain the trained network.

Images