A low-light scene image three-dimensional reconstruction method, device and electronic equipment based on a 3D Gaussian sphere

Abstract

The application discloses a low-light scene image three-dimensional reconstruction method and device based on a 3D Gaussian ball and electronic equipment, and the method comprises the following steps: 1) acquiring a low-light scene image; 2) initializing information of a 3D Gaussian ball in a 3D space according to camera parameters and volume rendering technology, obtaining an initial position, an initial radius and an initial color of the 3D Gaussian ball, and writing the initialized 3D Gaussian ball information into an enhanced image; 3) constructing a light adjustment network based on a convolutional neural network with a fusion attention mechanism; 4) inputting the enhanced image obtained in the step 2) into the light adjustment network, and predicting a light weight value of each 3D Gaussian ball by using the light adjustment network; 5) performing three-dimensional scene reconstruction by using a three-dimensional reconstruction algorithm based on the initial information of the 3D Gaussian ball and the light weight value, and obtaining a three-dimensional scene reconstruction view C under normal light nor The application can dynamically adjust light by using the light adjustment network, and realizes real restoration of a low-light scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, apparatus, and electronic device for three-dimensional reconstruction of low-light scene images based on a 3D Gaussian sphere. Background Technology

[0002] In the field of computer vision, 3D reconstruction technology is crucial for building 3D models of the real world from 2D images. Traditional Neural Radiance Field (NeRF) techniques can synthesize new views with high fidelity when processing high-quality input images. However, NeRF has significant limitations; it is highly dependent on the quality of the input image. In low-light scenes, the poor visibility and inaccurate pixel information of the input image make it difficult to capture high-fidelity color information and recover scene details well, ultimately resulting in low-quality 3D reconstructions.

[0003] 3D (3-Dimensional) Gaussian spheres, as an emerging 3D reconstruction technology, possess unique advantages. When dealing with complex scene structures, they can more efficiently represent the geometric and appearance information of the scene by flexibly adjusting various parameters of the Gaussian sphere (such as position, radius, and color). Compared with NeRF, 3D Gaussian spheres offer significant improvements in storage and computational efficiency, especially when processing large-scale scenes, where their advantages are even more pronounced. However, 3D Gaussian spheres also have shortcomings. They are not refined enough in terms of lighting processing, and their simulation of lighting effects in low-light scenes is not accurate enough, making it difficult to reproduce the realistic lighting atmosphere and color accuracy in reconstructed images.

[0004] Current 3D reconstruction techniques cannot adequately balance reconstruction accuracy, lighting effects, and computational efficiency in low-light scenarios, necessitating a new approach to address these issues. Summary of the Invention

[0005] The purpose of this invention is to provide a method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, comprising the following steps:

[0006] 1) Acquire low-light scene images and evaluate image quality. If the image quality is unsatisfactory, perform image enhancement processing on the low-light scene images to obtain enhanced images.

[0007] 2) Initialize the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and write the initialized 3D Gaussian sphere information into the enhanced image;

[0008] 3) Construct an illumination regulation network based on a convolutional neural network with a fusion attention mechanism;

[0009] 4) Input the enhanced image obtained in step 2) into the illumination adjustment network, and use the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0010] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight values, a 3D scene reconstruction algorithm is used to reconstruct the 3D scene and obtain the reconstructed 3D scene view C under normal illumination. nor .

[0011] Furthermore, image quality is characterized by the image's entropy value E, contrast C, and mean brightness L;

[0012] When the entropy value E is within the preset range [E min E max Outside of [C], the contrast ratio C is within the preset range. min C max Outside of [L], or the average brightness L is within the preset range [L] min ,L max Outside of these parameters, the image quality is substandard.

[0013] When the entropy value is too low or too high, there will be a lack of detail and too much noise. When the contrast is too low or too high, the details in the bright and dark areas will be lost. When the average brightness is too low or too high, it will be too dark or too bright.

[0014] The entropy value E, contrast ratio C, and mean luminance L are shown below:

[0015]

[0016] In the formula, p i I is the probability of a pixel with gray value i appearing in the image; I(x,y) is the pixel value at (x,y) in the image. M is the average pixel value of the image, and M and N are the dimensions of the image;

[0017] Furthermore, in step 1), the steps for image enhancement processing of low-light scene images include:

[0018] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0019] 1.2) The processed image blocks are merged using bilinear interpolation to obtain an enhanced image.

[0020] Furthermore, step 2), the step of obtaining the initial position of the 3D Gaussian sphere in 3D space, includes:

[0021] The camera's position o and viewing direction d in three-dimensional space are obtained through camera calibration technology;

[0022] Using the principle of ray casting in volume rendering technology, a ray is emitted from the camera position o along the viewing direction d, and the point where the ray intersects with the scene is taken as the initial position of the 3D Gaussian sphere.

[0023] Furthermore, the initial radius of the 3D Gaussian sphere α is the adjustment coefficient; S is the average object size in the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0024] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channel and lightness channel information in HSV color space.

[0025] Furthermore, the illumination adjustment network includes multiple convolutional layers, pooling layers, and fully connected layers;

[0026] Convolutional layers are used to extract local features of an image;

[0027] Pooling layers are used to reduce the dimensionality of feature maps;

[0028] Fully connected layers are used to integrate the extracted features and output the final prediction result.

[0029] Furthermore, in step 4), the step of predicting the illumination weight value for each 3D Gaussian sphere using an illumination adjustment network includes:

[0030] 4.1) Obtain the camera's position O in 3D space as the initialization starting point, and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td in the enhanced image; t ∈ R;

[0031] 4.2) Using the Monte Carlo simulation method, N points P are randomly sampled on the ray R(t). j j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in a scene;

[0032] For each sampling point P j The probability P of light propagation at that sampling point is calculated based on the occlusion status and material properties of the sampling point in the scene. trans (P i );

[0033] Propagation probability P trans (P i As shown below:

[0034] P trans (P i ) = w reflect ·P reflect +w refract ·P refract +wabsorb ·P absorb +w scatter ·P scatter (4)

[0035] In the formula, w reflect w refract w absorb w scatter P represents the weight; reflect P refract P absorb P scatter These are the probability of reflection, the probability of refraction, the probability of absorption, and the probability of scattering.

[0036] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where, sampling point P j Initial light intensity I init (P i )=I0×P trans (P i I0 is the initially set light intensity;

[0037] 4.4) Perform comprehensive calculations on the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, thus completing the local illumination initialization of the enhanced image;

[0038] 4.5) The convolutional layers of the illumination adjustment network are used to extract features from the enhanced image to obtain multiple feature maps F;

[0039] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0040] 4.7) The global feature vector G is processed using two fully connected layers to obtain the attention weight vector A;

[0041] 4.8) The attention weight vector a is weighted and fused with the feature map F to obtain the feature map F after the attention mechanism is applied. att ;

[0042] 4.8) Utilize the remaining convolutional layers and fully connected layers to process the feature map F att The process is performed to obtain the lighting weight value for each 3D Gaussian sphere.

[0043] Furthermore, the 3D scene reconstruction view C under normal lighting conditions nor As shown below:

[0044]

[0045] In the formula, (x,y,z) represents the position of a pixel in 3D space; (x i ,yi ,z i ) represents the center position of the Gaussian sphere; w i D represents the illumination weight value for the i-th 3D Gaussian sphere. r A represents the ambient light term and the diffuse reflection term. C i The initial color of the Gaussian sphere; r i Let be the initial radius of the Gaussian sphere.

[0046] An apparatus for three-dimensional reconstruction of low-light scene images based on the aforementioned low-light scene image method includes a camera, an image enhancement module, a 3D Gaussian sphere initialization module, an illumination adjustment module, and a three-dimensional reconstruction module;

[0047] The camera acquires images of low-light scenes and transmits them to the image enhancement module;

[0048] The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is enhanced to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image.

[0049] The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image.

[0050] The illumination adjustment module stores an illumination adjustment network;

[0051] The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere.

[0052] The 3D reconstruction module, based on the initial information of a 3D Gaussian sphere and illumination weight values, uses a 3D reconstruction algorithm to reconstruct the 3D scene, obtaining a 3D scene reconstruction view C under normal lighting. nor .

[0053] An electronic device that performs the method described.

[0054] The technical advantages of this invention are undeniable. It combines the high efficiency of 3D Gaussian spheres in representing scene structures with the precise control of illumination by a lighting adjustment network, thus overcoming the shortcomings of 3D Gaussian spheres in low-light processing. During reconstruction, the 3D Gaussian sphere can be used to quickly and accurately construct the scene geometry, while the lighting adjustment network dynamically adjusts the illumination, achieving realistic restoration of low-light scenes, effectively preserving scene details, and improving the accuracy and quality of 3D reconstruction of low-light scene images. Furthermore, compared to the traditional NeRF method, the application of 3D Gaussian spheres improves computational efficiency and reduces the time and computational resources required for reconstruction. Attached Figure Description

[0055] Figure 1 Flowchart for 3D reconstruction of low-light scene images based on 3D Gaussian sphere

[0056] Figure 2 This is a flowchart of the construction and weight prediction process for the illumination regulation network. Detailed Implementation

[0057] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0058] Example 1:

[0059] See Figures 1 to 2 A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere includes the following steps:

[0060] 1) Acquire low-light scene images and evaluate image quality. If the image quality is unsatisfactory, perform image enhancement processing on the low-light scene images to obtain enhanced images.

[0061] 2) Initialize the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and write the initialized 3D Gaussian sphere information into the enhanced image;

[0062] 3) Construct an illumination regulation network based on a convolutional neural network with a fusion attention mechanism;

[0063] 4) Input the enhanced image obtained in step 2) into the illumination adjustment network, and use the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0064] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight values, a 3D scene reconstruction algorithm is used to reconstruct the 3D scene and obtain the reconstructed 3D scene view C under normal illumination.nor .

[0065] Image quality is characterized by the image's entropy value E, contrast C, and mean brightness L;

[0066] When the entropy value E is within the preset range [E min E max Outside of [C], the contrast ratio C is within the preset range. min C max Outside of [L], or the average brightness L is within the preset range [L] min ,L max Outside of these parameters, the image quality is substandard.

[0067] The entropy value E, contrast ratio C, and mean luminance L are shown below:

[0068]

[0069] In the formula, p i I is the probability of a pixel with gray value i appearing in the image; I(x,y) is the pixel value at (x,y) in the image. M is the average pixel value of the image, and M and N are the dimensions of the image;

[0070] Step 1) includes the following steps for image enhancement processing of low-light scene images:

[0071] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0072] 1.2) The processed image blocks are merged using bilinear interpolation to obtain an enhanced image.

[0073] Step 2), the steps for obtaining the initial position of the 3D Gaussian sphere in 3D space include:

[0074] The camera's position o and viewing direction d in three-dimensional space are obtained through camera calibration technology;

[0075] Using the principle of ray casting in volume rendering technology, a ray is emitted from the camera position o along the viewing direction d, and the point where the ray intersects with the scene is taken as the initial position of the 3D Gaussian sphere.

[0076] Initial radius of a 3D Gaussian sphere α is the adjustment coefficient; S is the average object size in the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0077] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channel and lightness channel information in HSV color space.

[0078] The illumination adjustment network includes multiple convolutional layers, pooling layers, and fully connected layers;

[0079] Convolutional layers are used to extract local features of an image;

[0080] Pooling layers are used to reduce the dimensionality of feature maps;

[0081] Fully connected layers are used to integrate the extracted features and output the final prediction result.

[0082] Step 4), which involves using an illumination adjustment network to predict the illumination weight value for each 3D Gaussian sphere, includes:

[0083] 4.1) Obtain the camera's position O in 3D space as the initialization starting point, and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td in the enhanced image; t ∈ R;

[0084] 4.2) Using the Monte Carlo simulation method, N points P are randomly sampled on the ray R(t). j j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in a scene;

[0085] For each sampling point P j The probability P of light propagation at that sampling point is calculated based on the occlusion status and material properties of the sampling point in the scene. trans (P i );

[0086] Propagation probability P trans (P i As shown below:

[0087] P trans (P i ) = w reflect ·P reflect +w refract ·P refract +w absorb ·P absorb +w scatter ·P scatter (4)

[0088] In the formula, w reflect w refract w absorb w scatter P represents the weight; reflect P refract P absorb P scatter These are the probability of reflection, the probability of refraction, the probability of absorption, and the probability of scattering.

[0089] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where, sampling point P j Initial light intensity I init (P i )=I0×P trans (P i I0 is the initially set light intensity;

[0090] 4.4) Perform comprehensive calculations on the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, thus completing the local illumination initialization of the enhanced image;

[0091] 4.5) The convolutional layers of the illumination adjustment network are used to extract features from the enhanced image to obtain multiple feature maps F;

[0092] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0093] 4.7) The global feature vector G is processed using two fully connected layers to obtain the attention weight vector A;

[0094] 4.8) The attention weight vector A is weighted and fused with the feature map F to obtain the feature map F after the attention mechanism is applied. att ;

[0095] 4.8) Utilize the remaining convolutional layers and fully connected layers to process the feature map F att The process is performed to obtain the lighting weight value for each 3D Gaussian sphere.

[0096] 3D scene reconstruction view C under normal lighting nor As shown below:

[0097]

[0098] In the formula, (x,y,z) represents the position of a pixel in 3D space; (x i ,y i ,z i ) represents the center position of the Gaussian sphere; w i D represents the illumination weight value for the i-th 3D Gaussian sphere. r A represents the ambient light term and the diffuse reflection term. C i The initial color of the Gaussian sphere; r i Let be the initial radius of the Gaussian sphere.

[0099] Example 2:

[0100] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere includes the following steps:

[0101] 1) Acquire low-light scene images and evaluate image quality. If the image quality is unsatisfactory, perform image enhancement processing on the low-light scene images to obtain enhanced images.

[0102] 2) Initialize the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and write the initialized 3D Gaussian sphere information into the enhanced image;

[0103] 3) Construct an illumination regulation network based on a convolutional neural network with a fusion attention mechanism;

[0104] 4) Input the enhanced image obtained in step 2) into the illumination adjustment network, and use the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0105] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight values, a 3D scene reconstruction algorithm is used to reconstruct the 3D scene and obtain the reconstructed 3D scene view C under normal illumination. nor .

[0106] Example 3:

[0107] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as in Embodiment 2, further wherein the image quality is characterized by the image entropy value E, contrast C, and average brightness L.

[0108] When the entropy value is too low or too high, there will be a lack of detail and too much noise. When the contrast is too low or too high, the details in the bright and dark areas will be lost. When the average brightness is too low or too high, it will be too dark or too bright.

[0109] The entropy value E, contrast ratio C, and mean luminance L are shown below:

[0110]

[0111] In the formula, p i I is the probability of a pixel with gray value i appearing in the image; I(x,y) is the pixel value at (x,y) in the image. M and N are the average pixel values ​​of the image, and M and N are the dimensions of the image; Example 4:

[0112] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as any one of embodiments 2-3, further comprising, in step 1), the step of image enhancement processing of the low-light scene image including:

[0113] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0114] 1.2) The processed image blocks are merged using bilinear interpolation to obtain an enhanced image.

[0115] Example 5:

[0116] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as any one of embodiments 2-4, further comprising the following steps in step 2) for obtaining the initial position of the 3D Gaussian sphere in 3D space:

[0117] The camera's position o and viewing direction d in three-dimensional space are obtained through camera calibration technology;

[0118] Using the principle of ray casting in volume rendering technology, a ray is emitted from the camera position o along the viewing direction d, and the point where the ray intersects with the scene is taken as the initial position of the 3D Gaussian sphere.

[0119] Example 6:

[0120] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as any one of embodiments 2-5, further wherein the initial radius of the 3D Gaussian sphere is... α is the adjustment coefficient; S is the average object size in the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0121] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channel and lightness channel information in HSV color space.

[0122] Example 7:

[0123] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as any one of embodiments 2-6, further wherein the illumination adjustment network includes multiple convolutional layers, pooling layers and fully connected layers;

[0124] Convolutional layers are used to extract local features of an image;

[0125] Pooling layers are used to reduce the dimensionality of feature maps;

[0126] Fully connected layers are used to integrate the extracted features and output the final prediction result.

[0127] Example 8:

[0128] A method for 3D reconstruction of low-light scene images based on 3D Gaussian spheres, with the same technical content as any one of embodiments 2-7, further comprising, in step 4), the step of predicting the illumination weight value of each 3D Gaussian sphere using an illumination adjustment network, including:

[0129] 4.1) Obtain the camera's position O in 3D space as the initialization starting point, and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td in the enhanced image; t ∈ R;

[0130] 4.2) Using the Monte Carlo simulation method, N points P are randomly sampled on the ray R(t). j j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in a scene;

[0131] For each sampling point P j The probability P of light propagation at that sampling point is calculated based on the occlusion status and material properties of the sampling point in the scene. trans (P i );

[0132] According to sampling point (P) i The occlusion status and material of the object are used to calculate the reflection probability (P) using the Fresnel equation. reflect ) and refraction probability (P refract The absorption probability (P) is calculated according to the Beer-Lambert law. absorb Considering that light may scatter on certain materials, the Henyey-Greenstein phase function can be used to derive the scattering probability (P). scatter Finally, the propagation probability is calculated using a weighted method:

[0133] P trans (P i ) = w reflect ·P reflect +w refract ·P refract +w absorb ·P absorb +w scatter ·P scatter

[0134] Where w represents the weight, reflecting the likelihood of each interaction method, and their sum equals 1.

[0135] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where, sampling point P j Initial light intensity I init (P i )=I0×P trans (P i I0 is the initially set light intensity;

[0136] 4.4) Perform comprehensive calculations on the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, thus completing the local illumination initialization of the enhanced image;

[0137] 4.5) The convolutional layers of the illumination adjustment network are used to extract features from the enhanced image to obtain multiple feature maps F;

[0138] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0139] 4.7) The global feature vector G is processed using two fully connected layers to obtain the attention weight vector A;

[0140] 4.8) The attention weight vector A is weighted and fused with the feature map F to obtain the feature map F after the attention mechanism is applied. att ;

[0141] 4.8) Utilize the remaining convolutional layers and fully connected layers to process the feature map F att The process is performed to obtain the lighting weight value for each 3D Gaussian sphere.

[0142] Example 9:

[0143] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, with the same technical content as any one of embodiments 2-8, further comprising a 3D scene reconstruction view C under normal lighting. nor As shown below:

[0144]

[0145] In the formula, (x,y,z) represents the position of a pixel in 3D space; (x i ,y i ,z i ) represents the center position of the Gaussian sphere; w i D represents the illumination weight value for the i-th 3D Gaussian sphere. r A represents the ambient light term and the diffuse reflection term.

[0146] Example 10:

[0147] An apparatus for three-dimensional reconstruction of low-light scene images based on any one of embodiments 1-9 includes a camera, an image enhancement module, a 3D Gaussian sphere initialization module, an illumination adjustment module, and a three-dimensional reconstruction module;

[0148] The camera acquires images of low-light scenes and transmits them to the image enhancement module;

[0149] The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is enhanced to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image.

[0150] The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image.

[0151] The illumination adjustment module stores an illumination adjustment network;

[0152] The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere.

[0153] The 3D reconstruction module, based on the initial information of a 3D Gaussian sphere and illumination weight values, uses a 3D reconstruction algorithm to reconstruct the 3D scene, obtaining a 3D scene reconstruction view C under normal lighting. nor .

[0154] Example 11:

[0155] An electronic device that performs the method described in any one of Examples 1-9.

[0156] Example 12:

[0157] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, comprising the following steps:

[0158] 1. Image preprocessing and 3D Gaussian sphere initialization

[0159] After acquiring the low-light image set, image quality is first assessed. Image quality is comprehensively judged by calculating the image's entropy value (E), contrast (C), and mean brightness (L). Entropy value... Where p i It is the probability of a pixel with grayscale value i appearing in an image; contrast. I(x,y) is the pixel value at (x,y) in the image. The average pixel value of the image, where M and N are the image dimensions; the average brightness value.

[0160] If the evaluation results indicate poor image quality, an adaptive histogram equalization algorithm is used to enhance the image. This algorithm divides the image into multiple small blocks, performs histogram equalization on each block, and then merges the processed blocks using bilinear interpolation, thereby enhancing the overall brightness and contrast of the image while preserving image details.

[0161] Next, camera calibration technology is used to obtain the camera's position o and viewing direction d in 3D space. Utilizing the ray casting principle in volume rendering technology, rays are emitted from the camera position o along the viewing direction d, and the point where the rays intersect with the scene is the position corresponding to an image pixel in 3D space.

[0162] Determine the initial radius r of the 3D Gaussian sphere i In this process, not only are the image's depth information D and the scene's average object size S considered, but also the image's texture complexity T. Texture complexity is measured by calculating the entropy value of the image's gray-level co-occurrence matrix. The formula is used... Where α is the adjustment coefficient used to control the overall size of the radius, and β is the texture influence coefficient used to adjust the degree of influence of texture complexity on the radius.

[0163] For the initial color C of the Gaussian sphere i A multi-channel fusion approach is employed. First, the image is converted from the RGB color space to the HSV color space, and the luminance channel information from both the RGB and HSV color spaces is extracted. These four channels are then weighted and fused, with the weights dynamically adjusted based on the image's brightness distribution and color saturation. For example, for areas with low brightness and high saturation, the weight of the RGB channel is appropriately increased; for areas with high brightness and low saturation, the weight of the luminance channel is increased. By comprehensively considering information from different channels in this way, a more accurate initial Gaussian sphere color C is obtained. i Complete the initialization of the 3D Gaussian sphere.

[0164] 2. Construction and Weight Prediction of Illumination Regulation Network

[0165] In the illumination adjustment network construction and weight prediction stage, we use a convolutional neural network that incorporates an attention mechanism to predict illumination weights for low-light images. The specific algorithm processing is as follows:

[0166] 2.1. Network Architecture: The illumination adjustment network adopts a convolutional neural network architecture, consisting of multiple convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract local features from the image, while pooling layers reduce the dimensionality of the feature maps, decreasing computational cost and preventing overfitting. Multiple convolutional and pooling layers are stacked alternately to progressively extract high-level image features. At the end of the network, a fully connected layer integrates the extracted features and outputs the final prediction result.

[0167] 2.2 Local Illumination Initialization: The camera's position O in 3D space is obtained as the initialization starting point, and the camera's viewing direction D is also obtained. Based on this, the ray equation R(t) = O + tD (t∈R) is constructed. This equation is used to determine the propagation path of the light ray originating from the camera in 3D space. Using the Monte Carlo simulation method, N points P are randomly sampled on the ray. i (i = 1, 2, ..., N). These sampling points are used to simulate the propagation of light in the scene to obtain more accurate lighting information. For each sampling point P iThe probability P of light propagation at a point is calculated based on its occlusion in the scene and its material properties. trans (P i Occlusion can be determined by comparing the positional relationship between the sampling point and objects in the scene. If the sampling point is located inside an object or is occluded by an object, the probability of light propagation is low. Material properties determine the reflection, refraction, and absorption characteristics of light when it encounters that material. For example, for metallic materials, light reflection is strong, resulting in a low propagation probability; for transparent materials, the probability of light propagation is relatively high. Using this information, a suitable light propagation probability model is employed to calculate P. trans (P i Based on the calculated light propagation probability, the light intensity at each sampling point is adjusted. Initial illumination intensity I init (P i It can be achieved through formula I init (P i )=I0×P trans (P i The initial light intensity is calculated, where I0 is the initially set light intensity. By comprehensively calculating the light intensity of all sampling points, the initial light intensity distribution at each location point is obtained, thus completing the local lighting initialization.

[0168] 2.3. Attention Mechanism Integration: The low-light image, processed by image preprocessing and 3D Gaussian sphere initialization, is input into the illumination adjustment network. First, convolutional layers extract features from the image, obtaining a series of feature maps F. Then, global average pooling is performed on these feature maps, compressing each feature map into a global feature vector G, enabling the network to acquire global information about the image. Two fully connected layers process the global feature vector G. The first fully connected layer maps the global feature vector to a low-dimensional space, and the second fully connected layer maps it back to the original dimension, obtaining the attention weight vector A. Each element in the attention weight vector A corresponds to the importance of each position in the feature map F. In this way, the network can automatically learn the importance of different regions in the image. The attention weight vector A and the feature map F are then weighted and fused. Specifically, the feature value at each position in the feature map F is multiplied by the corresponding attention weight to obtain the weighted feature map F. att In this way, the network can pay more attention to important regions in the image during subsequent processing, improving the accuracy of illumination weight prediction. The feature map F after attention mechanism processing... att After a series of convolutional and fully connected layers, the final output is the illumination weight value w for each 3D Gaussian sphere. i These weight values ​​will be used in the subsequent 3D reconstruction process to adjust the lighting contribution of each Gaussian sphere to the final reconstructed image, thereby optimizing the lighting effects in low-light scenes.

[0169] Through the above algorithm processing, the illumination adjustment network can effectively initialize the local illumination of low-light images and accurately predict the illumination weight value of each 3D Gaussian sphere, providing key illumination information for subsequent 3D reconstruction and helping to improve the quality of 3D reconstruction of low-light scene images.

[0170] 3. Three-dimensional reconstruction based on 3D Gaussian sphere

[0171] During the generation of a reconstructed 3D scene view under normal lighting, based on the color C of the 3D Gaussian sphere... i Location information and illumination weight value w i Using improved rendering formulas

[0172]

[0173] Rendering calculations are performed to obtain a reconstructed 3D scene view C with normal lighting. nor When generating reconstructed views of low-light 3D scenes, further adjust the lighting weight value w. i Simulate the lighting effects under low-light conditions to obtain a reconstructed 3D scene view C under low-light conditions. low (Formula same as C) nor ), utilizing perceptual loss, structural similarity loss, and color consistency loss to adjust C. i w i r i The 3D reconstruction algorithm is optimized using the influence range of sample points. Perceptual loss is calculated by comparing the differences between the reconstructed image and the real image in the feature space of the pre-trained convolutional neural network; structural similarity loss uses the Structural Similarity Index (SSIM) to measure the structural similarity of the images; color consistency loss ensures that the colors of the reconstructed image remain consistent under different lighting conditions by calculating and summing the differences between different color channels.

Claims

1. A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, characterized in that, Includes the following steps: Step 1) Obtain low-light scene images and evaluate image quality. If the image quality is unsatisfactory, perform image enhancement processing on the low-light scene images to obtain enhanced images. Step 2) Initialize the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and write the initialized 3D Gaussian sphere information into the enhanced image. Step 3) Construct an illumination regulation network based on a convolutional neural network with a fusion attention mechanism; Step 4) Input the enhanced image obtained in Step 2) into the illumination adjustment network and use the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere; Step 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight values, a 3D scene reconstruction algorithm is used to reconstruct the 3D scene and obtain a 3D scene reconstruction view under normal illumination. ; Step 4), which involves using an illumination adjustment network to predict the illumination weight value for each 3D Gaussian sphere, includes: Step 4.1) Obtain the position of the camera in three-dimensional space. Using this as the initial starting point and obtaining the camera's viewing direction d, ray equations are constructed in the enhanced image. ; ; Step 4.2) Using the Monte Carlo simulation method, in the ray Up-random sampling Points ; ; using sampling points Simulate the propagation of light in a scene; For each sampling point The probability of light propagation at a sampling point is calculated based on the occlusion status and material properties of that sampling point in the scene. ; Propagation probability As shown below: （4） In the formula, , , , Indicates weight; , , , These are the probability of reflection, the probability of refraction, the probability of absorption, and the probability of scattering. Step 4.3) Adjust the light intensity at each sampling point to obtain the initial illumination intensity; where, sampling point initial light intensity ; The initial light intensity setting; Step 4.4) Perform a comprehensive calculation on the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, thus completing the local illumination initialization of the enhanced image; Step 4.5) Use the convolutional layers of the illumination adjustment network to extract features from the enhanced image, obtaining multiple feature maps. ; Step 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector. ; Step 4.7) Use two fully connected layers to process the global feature vector. The process is performed to obtain the attention weight vector. ; Step 4.8) Transfer the attention weight vector With feature map Weighted fusion is performed to obtain the feature map after attention mechanism processing. ; Step 4.9) Utilize the remaining convolutional and fully connected layers to process the feature map. The process is performed to obtain the lighting weight value for each 3D Gaussian sphere.

2. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, Image quality is measured by the image's entropy value. Contrast and average brightness Characterization; When entropy value Located within the preset range In addition, contrast Located within the preset range In addition, or average brightness Located within the preset range Otherwise, the image quality is substandard; Among them, entropy value Contrast and average brightness They are shown below: （1） （2） （3） In the formula, The grayscale value in the image The probability of a pixel appearing; Is the image in Pixel value at that location, It is the average pixel value of the image. and It refers to the size of the image.

3. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, Step 1) includes the following steps for image enhancement processing of low-light scene images: Step 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block; Step 1.2) Merge the processed image blocks using bilinear interpolation to obtain the enhanced image.

4. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, Step 2), the steps for obtaining the initial position of the 3D Gaussian sphere in 3D space include: The camera's position in three-dimensional space is obtained through camera calibration technology. and observation direction ; Using the principle of ray casting in volume rendering technology, from the camera position Along the direction of observation Emit a ray of light, and use the point where the ray intersects the scene as the initial position of the 3D Gaussian sphere.

5. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, Initial radius of a 3D Gaussian sphere ; This is the adjustment coefficient; The average object size in the scene; This refers to the depth information of the image; This is the texture influence coefficient; The texture complexity of the image; The initial color of the 3D Gaussian sphere It is obtained by weighted fusion of RGB channel and lightness channel information in HSV color space.

6. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, The illumination adjustment network includes multiple convolutional layers, pooling layers, and fully connected layers; Convolutional layers are used to extract local features of an image; Pooling layers are used to reduce the dimensionality of feature maps; Fully connected layers are used to integrate the extracted features and output the final prediction result.

7. The method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere according to claim 1, characterized in that, 3D scene reconstruction view under normal lighting As shown below: （5） In the formula, The position of a pixel in 3D space; The center of the Gaussian sphere; Let i be the illumination weight value for the i-th 3D Gaussian sphere; , For ambient light and diffuse reflection; The initial color of the Gaussian sphere; Let be the initial radius of the Gaussian sphere.

8. An apparatus for three-dimensional reconstruction of low-light scene images based on the method described in any one of claims 1-7, characterized in that, Includes a camera, an image enhancement module, a 3D Gaussian sphere initialization module, an illumination adjustment module, and a 3D reconstruction module; The camera acquires images of low-light scenes and transmits them to the image enhancement module; The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is enhanced to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image. The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in 3D space according to camera parameters and volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image. The illumination adjustment module stores an illumination adjustment network; The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere. The 3D reconstruction module, based on the initial information of a 3D Gaussian sphere and illumination weight values, uses a 3D reconstruction algorithm to reconstruct the 3D scene and obtain a reconstructed 3D scene view under normal lighting. .

9. An electronic device, characterized in that, Perform the method according to any one of claims 1-7.

Images