Low-illumination scene image three-dimensional reconstruction method and device based on 3D Gaussian ball, and electronic equipment

Abstract

The invention discloses a low-illumination scene image three-dimensional reconstruction method and device based on a 3D Gaussian ball, and electronic equipment. The method comprises the following steps: 1) obtaining a low-illumination scene image; 2) initializing information of a 3D Gaussian ball in a 3D space according to camera parameters and a volume rendering technology to obtain an initial position, an initial radius and an initial color of the 3D Gaussian ball, and writing the initialized 3D Gaussian ball information into the enhanced image; 3) constructing an illumination adjustment network based on the convolutional neural network fused with the attention mechanism; 4) inputting the enhanced image obtained in the step 2) into an illumination adjustment network, and predicting an illumination weight value of each 3D Gaussian ball by using the illumination adjustment network; and 5) based on the initial information of the 3D Gaussian ball and the illumination weight value, performing three-dimensional scene reconstruction by using a three-dimensional reconstruction algorithm to obtain a three-dimensional scene reconstruction view Cnor under normal illumination. According to the invention, the illumination can be dynamically adjusted through the illumination adjusting network, and the real restoration of the low-illumination scene is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and electronic equipment for three-dimensional reconstruction of low-light scene images based on a 3D Gaussian sphere. Background Art

[0002] In the field of computer vision, 3D reconstruction technology is crucial for constructing 3D models of the real world from 2D images. Traditional Neural Radiance Field (NeRF) technology can synthesize new views with high fidelity when processing high-quality input images. However, NeRF has significant limitations and is highly dependent on the quality of the input images. In low-light scenarios, the input image has poor visibility and inaccurate pixel information, making it difficult to capture high-fidelity color information and poorly restore scene details, ultimately resulting in low-quality 3D reconstruction.

[0003] As an emerging 3D reconstruction technology, the 3D (3-Dimensional) Gaussian sphere has unique advantages. When processing complex scene structures, it can more efficiently represent the geometric and appearance information in the scene by flexibly adjusting the various parameters of the Gaussian sphere (such as position, radius, color, etc.). Compared with NeRF, the 3D Gaussian sphere has significantly improved storage and computational efficiency, especially when processing large-scale scenes. Its advantages are more obvious. However, the 3D Gaussian sphere also has shortcomings. It is not sophisticated enough in lighting processing, and its simulation of lighting effects in low-light scenes is not accurate enough. It is difficult to restore the true lighting atmosphere and color accuracy in the reconstructed image.

[0004] Current 3D reconstruction technologies cannot effectively balance reconstruction accuracy, lighting processing effects, and computational efficiency in low-light scenarios. A new method is urgently needed to solve these problems. Summary of the Invention

[0005] The present invention aims to provide a method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, comprising the following steps:

[0006] 1) Acquire a low-light scene image and evaluate the image quality. If the image quality is unqualified, perform image enhancement processing on the low-light scene image to obtain an enhanced image;

[0007] 2) Initializing the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and writing the initialized 3D Gaussian sphere information into the enhanced image;

[0008] 3) Constructing a light regulation network based on a convolutional neural network integrated with an attention mechanism;

[0009] 4) Inputting the enhanced image obtained in step 2) into the illumination adjustment network, and using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0010] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight value, the 3D scene is reconstructed using the 3D reconstruction algorithm to obtain the 3D scene reconstruction view C under normal illumination. nor .

[0011] Furthermore, the image quality is characterized by the image entropy value E, contrast C and brightness mean L;

[0012] When the entropy value E is within the preset range [E min ,E max ], the contrast C is within the preset range [C min ,C max ], or the average brightness L is within the preset range [L min ,L max ], the image quality is unqualified;

[0013] When the entropy value is too low or too high, there will be a lack of details and excessive noise. When the contrast is too low or too high, the details of the bright and dark parts will be lost. When the average brightness is too low or too high, it will be too dark or too bright.

[0014] Among them, the entropy value E, contrast C and brightness mean L are as follows:

[0015]

[0016] Where p i is the probability of the pixel with gray value i appearing in the image; I(x,y) is the pixel value of the image at (x,y), is the average pixel value of the image, M and N are the sizes of the image;

[0017] Furthermore, in step 1), the step of performing image enhancement processing on the low-light scene image includes:

[0018] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0019] 1.2) The processed image blocks are merged by bilinear interpolation method to obtain the enhanced image.

[0020] Furthermore, in step 2), the step of obtaining the initial position of the 3D Gaussian sphere in the 3D space includes:

[0021] Obtain the camera's position o and viewing direction d in three-dimensional space through camera calibration technology;

[0022] Using the ray casting principle in volume rendering technology, light is emitted from the camera position o along the observation direction d, and the point where the light intersects the scene is used as the initial position of the 3D Gaussian sphere.

[0023] Furthermore, the initial radius of the 3D Gaussian sphere α is the adjustment coefficient; S is the average object size of the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0024] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channels and lightness channel information in HSV color space.

[0025] Furthermore, the light adjustment network includes multiple convolutional layers, pooling layers and fully connected layers;

[0026] The convolutional layer is used to extract local features of the image;

[0027] The pooling layer is used to reduce the dimension of the feature map;

[0028] The fully connected layer is used to integrate the extracted features and output the final prediction results.

[0029] Furthermore, in step 4), the step of using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere includes:

[0030] 4.1) Obtain the camera's position O in 3D space as the initialization starting point and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td; t∈R in the enhanced image;

[0031] 4.2) Using the Monte Carlo simulation method, randomly sample N points P on the ray R(t) j ; j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in the scene;

[0032] For each sampling point P j , calculate the propagation probability P of light at this point based on the occlusion of the sampling point in the scene and the material properties trans (P i );

[0033] Propagation probability P trans (P i ) is as follows:

[0034] P trans (P i )=w reflect ·P reflect +w refract ·P refract +wabsorb ·P absorb +w scatter ·P scatter (4)

[0035] Where w reflect 、w refract 、w absorb 、w scatter represents weight; P reflect 、P refract 、P absorb 、P scatter is the probability of reflection, refraction, absorption and scattering;

[0036] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where the sampling point P j The initial light intensity I init (P i )=I0×P trans (P i );I0 is the initial light intensity;

[0037] 4.4) Comprehensively calculate the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, completing the local illumination initialization of the enhanced image;

[0038] 4.5) Using the convolutional layer of the illumination adjustment network to extract features from the enhanced image, a plurality of feature maps F are obtained;

[0039] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0040] 4.7) Use two fully connected layers to process the global feature vector G to obtain the attention weight vector A;

[0041] 4.8) Perform weighted fusion of the attention weight vector a and the feature map F to obtain the feature map F processed by the attention mechanism att ;

[0042] 4.8) Use the remaining convolutional layers and fully connected layers to transform the feature map F att Processing is performed to obtain the lighting weight value of each 3D Gaussian sphere.

[0043] Furthermore, the 3D scene reconstruction view C under normal illumination nor As shown below:

[0044]

[0045] Where (x, y, z) is the position of the pixel in 3D space; (x i ,yi ,z i ) is the center position of the Gaussian sphere; w i is the illumination weight value of the i-th 3D Gaussian sphere; D r , A is the ambient light term and diffuse reflection term. C i is the initial color of the Gaussian sphere; r i is the initial radius of the Gaussian sphere.

[0046] A device based on the method for 3D reconstruction of low-light scene images, comprising a camera, an image enhancement module, a 3D Gaussian sphere initialization module, an illumination adjustment module, and a 3D reconstruction module;

[0047] The camera acquires low-light scene images and transmits them to the image enhancement module;

[0048] The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is subjected to image enhancement processing to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image.

[0049] The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and the volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image;

[0050] The illumination adjustment module stores an illumination adjustment network;

[0051] The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0052] The 3D reconstruction module uses the 3D reconstruction algorithm to reconstruct the 3D scene based on the initial information of the 3D Gaussian sphere and the illumination weight value, and obtains the 3D scene reconstruction view C under normal illumination. nor .

[0053] An electronic device executes the method.

[0054] The technical effect of the present invention is unquestionable. The present invention combines the high efficiency of the 3D Gaussian sphere in representing scene structure with the precise control of illumination by the illumination adjustment network, thus making up for the shortcomings of the 3D Gaussian sphere in low-light processing. During the reconstruction process, the 3D Gaussian sphere can be used to quickly and accurately construct the scene geometry, and the illumination can be dynamically adjusted through the illumination adjustment network to achieve a true restoration of the low-light scene, effectively retaining scene details, and improving the accuracy of the three-dimensional reconstruction of low-light scene images and the quality of the reconstructed image. At the same time, compared with the traditional NeRF method, the application of the 3D Gaussian sphere improves computational efficiency and reduces the time and computing resources required for reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart for 3D reconstruction of low-light scene images based on 3D Gaussian sphere

[0056] Figure 2 Flowchart for building the light regulation network and predicting weights. DETAILED DESCRIPTION

[0057] The present invention will be further described below with reference to the following examples, but it should not be understood that the scope of the present invention is limited to the following examples. Without departing from the above technical ideas of the present invention, various substitutions and modifications can be made according to common technical knowledge and customary means in the art, and all should be included in the scope of protection of the present invention.

[0058] Example 1:

[0059] See also Figures 1 to 2 A 3D Gaussian sphere-based 3D reconstruction method for low-light scene images comprises the following steps:

[0060] 1) Acquire a low-light scene image and evaluate the image quality. If the image quality is unqualified, perform image enhancement processing on the low-light scene image to obtain an enhanced image;

[0061] 2) Initializing the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and writing the initialized 3D Gaussian sphere information into the enhanced image;

[0062] 3) Constructing a light regulation network based on a convolutional neural network integrated with an attention mechanism;

[0063] 4) Inputting the enhanced image obtained in step 2) into the illumination adjustment network, and using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0064] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight value, the 3D scene is reconstructed using the 3D reconstruction algorithm to obtain the 3D scene reconstruction view C under normal illumination.nor .

[0065] Image quality is characterized by the image entropy value E, contrast C and brightness mean L;

[0066] When the entropy value E is within the preset range [E min ,E max ], the contrast C is within the preset range [C min ,C max ], or the average brightness L is within the preset range [L min ,L max ], the image quality is unqualified;

[0067] Among them, the entropy value E, contrast C and brightness mean L are as follows:

[0068]

[0069] Where p i is the probability of the pixel with gray value i appearing in the image; I(x,y) is the pixel value of the image at (x,y), is the average pixel value of the image, M and N are the sizes of the image;

[0070] In step 1), the step of performing image enhancement processing on the low-light scene image includes:

[0071] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0072] 1.2) The processed image blocks are merged by bilinear interpolation method to obtain the enhanced image.

[0073] In step 2), the step of obtaining the initial position of the 3D Gaussian sphere in the 3D space includes:

[0074] Obtain the camera's position o and viewing direction d in three-dimensional space through camera calibration technology;

[0075] Using the ray casting principle in volume rendering technology, light is emitted from the camera position o along the observation direction d, and the point where the light intersects the scene is used as the initial position of the 3D Gaussian sphere.

[0076] Initial radius of the 3D Gaussian sphere α is the adjustment coefficient; S is the average object size of the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0077] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channels and lightness channel information in HSV color space.

[0078] The light adjustment network includes multiple convolutional layers, pooling layers and fully connected layers;

[0079] The convolutional layer is used to extract local features of the image;

[0080] The pooling layer is used to reduce the dimension of the feature map;

[0081] The fully connected layer is used to integrate the extracted features and output the final prediction results.

[0082] In step 4), the step of using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere includes:

[0083] 4.1) Obtain the camera's position O in 3D space as the initialization starting point and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td; t∈R in the enhanced image;

[0084] 4.2) Using the Monte Carlo simulation method, randomly sample N points P on the ray R(t) j ; j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in the scene;

[0085] For each sampling point P j , calculate the propagation probability P of light at this point based on the occlusion of the sampling point in the scene and the material properties trans (P i );

[0086] Propagation probability P trans (P i ) is as follows:

[0087] P trans (P i )=w reflect ·P reflect +w refract ·P refract +w absorb ·P absorb +w scatter ·P scatter (4)

[0088] Where w reflect 、w refract 、w absorb 、w scatter represents weight; P reflect 、P refract 、P absorb 、P scatter is the probability of reflection, refraction, absorption and scattering;

[0089] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where the sampling point P j The initial light intensity I init (P i )=I0×P trans (P i );I0 is the initial light intensity;

[0090] 4.4) Comprehensively calculate the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, completing the local illumination initialization of the enhanced image;

[0091] 4.5) Using the convolutional layer of the illumination adjustment network to extract features from the enhanced image, a plurality of feature maps F are obtained;

[0092] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0093] 4.7) Use two fully connected layers to process the global feature vector G to obtain the attention weight vector A;

[0094] 4.8) Perform weighted fusion of the attention weight vector A and the feature map F to obtain the feature map F processed by the attention mechanism att ;

[0095] 4.8) Use the remaining convolutional layers and fully connected layers to transform the feature map F att Processing is performed to obtain the lighting weight value of each 3D Gaussian sphere.

[0096] 3D scene reconstruction view C under normal lighting nor As shown below:

[0097]

[0098] Where (x, y, z) is the position of the pixel in 3D space; (x i ,y i ,z i ) is the center position of the Gaussian sphere; w i is the illumination weight value of the i-th 3D Gaussian sphere; D r , A is the ambient light term and diffuse reflection term. C i is the initial color of the Gaussian sphere; r i is the initial radius of the Gaussian sphere.

[0099] Example 2:

[0100] A 3D Gaussian sphere-based 3D reconstruction method for low-light scene images comprises the following steps:

[0101] 1) Acquire a low-light scene image and evaluate the image quality. If the image quality is unqualified, perform image enhancement processing on the low-light scene image to obtain an enhanced image;

[0102] 2) Initializing the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and writing the initialized 3D Gaussian sphere information into the enhanced image;

[0103] 3) Constructing a light regulation network based on a convolutional neural network integrated with an attention mechanism;

[0104] 4) Inputting the enhanced image obtained in step 2) into the illumination adjustment network, and using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0105] 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight value, the 3D scene is reconstructed using the 3D reconstruction algorithm to obtain the 3D scene reconstruction view C under normal illumination. nor .

[0106] Example 3:

[0107] A 3D Gaussian sphere-based 3D reconstruction method for low-light scene images. The technical content is the same as that of Example 2. Furthermore, the image quality is characterized by the image entropy value E, contrast C, and brightness mean L.

[0108] When the entropy value is too low or too high, there will be a lack of details and excessive noise. When the contrast is too low or too high, the details of the bright and dark parts will be lost. When the average brightness is too low or too high, it will be too dark or too bright.

[0109] Among them, the entropy value E, contrast C and brightness mean L are as follows:

[0110]

[0111] Where p i is the probability of the pixel with gray value i appearing in the image; I(x,y) is the pixel value of the image at (x,y), is the average pixel value of the image, M and N are the sizes of the image; Example 4:

[0112] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, the technical content of which is the same as any one of Examples 2-3, further comprising the step of performing image enhancement processing on the low-light scene image in step 1) comprising:

[0113] 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block;

[0114] 1.2) The processed image blocks are merged by bilinear interpolation method to obtain the enhanced image.

[0115] Example 5:

[0116] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, the technical content of which is the same as any one of Examples 2-4, further comprising the step of obtaining the initial position of the 3D Gaussian sphere in 3D space in step 2) comprising:

[0117] Obtain the camera's position o and viewing direction d in three-dimensional space through camera calibration technology;

[0118] Using the ray casting principle in volume rendering technology, light is emitted from the camera position o along the observation direction d, and the point where the light intersects the scene is used as the initial position of the 3D Gaussian sphere.

[0119] Example 6:

[0120] A 3D reconstruction method for low-light scene images based on a 3D Gaussian sphere, the technical content of which is the same as any one of Examples 2-5, further, the initial radius of the 3D Gaussian sphere α is the adjustment coefficient; S is the average object size of the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image;

[0121] The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channels and lightness channel information in HSV color space.

[0122] Example 7:

[0123] A method for 3D reconstruction of low-light scene images based on a 3D Gaussian sphere, the technical content of which is the same as any one of Examples 2-6, further comprising a light adjustment network comprising a plurality of convolutional layers, pooling layers, and fully connected layers;

[0124] The convolutional layer is used to extract local features of the image;

[0125] The pooling layer is used to reduce the dimension of the feature map;

[0126] The fully connected layer is used to integrate the extracted features and output the final prediction results.

[0127] Example 8:

[0128] A method for 3D reconstruction of low-light scene images based on 3D Gaussian spheres, the technical content of which is the same as any one of Examples 2-7, further comprising the step of predicting the illumination weight value of each 3D Gaussian sphere using an illumination adjustment network in step 4) comprising:

[0129] 4.1) Obtain the camera's position O in 3D space as the initialization starting point and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td; t∈R in the enhanced image;

[0130] 4.2) Using the Monte Carlo simulation method, randomly sample N points P on the ray R(t) j ; j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in the scene;

[0131] For each sampling point P j , calculate the propagation probability P of light at this point based on the occlusion of the sampling point in the scene and the material properties trans (P i );

[0132] According to the sampling point (P i )'s occlusion and material, and the reflection probability (P reflect ) and refraction probability (P refract ), the absorption probability (P absorb ), considering that light may be scattered on some materials, the Henyey-Greenstein phase function can be used to obtain the scattering probability (P scatter ), and finally the propagation probability is calculated using a weighted approach:

[0133] P trans (P i )=w reflect ·P reflect +w refract ·P refract +w absorb ·P absorb +w scatter ·P scatter

[0134] Where w represents the weight, which reflects the possibility of each interaction mode, and their sum is equal to 1.

[0135] 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where the sampling point P j The initial light intensity I init (P i )=I0×P trans (P i );I0 is the initial light intensity;

[0136] 4.4) Comprehensively calculate the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, completing the local illumination initialization of the enhanced image;

[0137] 4.5) Using the convolutional layer of the illumination adjustment network to extract features from the enhanced image, a plurality of feature maps F are obtained;

[0138] 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G;

[0139] 4.7) Use two fully connected layers to process the global feature vector G to obtain the attention weight vector A;

[0140] 4.8) Perform weighted fusion of the attention weight vector A and the feature map F to obtain the feature map F processed by the attention mechanism att ;

[0141] 4.8) Use the remaining convolutional layers and fully connected layers to transform the feature map F att Processing is performed to obtain the lighting weight value of each 3D Gaussian sphere.

[0142] Example 9:

[0143] A 3D Gaussian sphere-based method for 3D reconstruction of low-light scene images, the technical content of which is the same as any one of Embodiments 2-8, further, a 3D scene reconstruction view C under normal illumination nor As shown below:

[0144]

[0145] Where (x, y, z) is the position of the pixel in 3D space; (x i ,y i ,z i ) is the center position of the Gaussian sphere; w i is the illumination weight value of the i-th 3D Gaussian sphere; D r , A is the ambient light term and diffuse reflection term.

[0146] Example 10:

[0147] A device for three-dimensional reconstruction of low-light scene images based on the method described in any one of Embodiments 1-9, comprising a camera, an image enhancement module, a 3D Gaussian sphere initialization module, an illumination adjustment module, and a three-dimensional reconstruction module;

[0148] The camera acquires low-light scene images and transmits them to the image enhancement module;

[0149] The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is subjected to image enhancement processing to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image.

[0150] The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and the volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image;

[0151] The illumination adjustment module stores an illumination adjustment network;

[0152] The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere;

[0153] The 3D reconstruction module uses the 3D reconstruction algorithm to reconstruct the 3D scene based on the initial information of the 3D Gaussian sphere and the illumination weight value, and obtains the 3D scene reconstruction view C under normal illumination. nor .

[0154] Example 11:

[0155] An electronic device that executes the method described in any one of embodiments 1-9.

[0156] Example 12:

[0157] A 3D Gaussian sphere-based 3D reconstruction method for low-light scene images, comprising the following steps:

[0158] 1. Image preprocessing and 3D Gaussian sphere initialization

[0159] After obtaining the low-light image set, the image quality is evaluated first. The image quality is comprehensively judged by calculating the entropy value E, contrast C and brightness mean L of the image. where p i is the probability of a pixel with gray value i appearing in the image; contrast I(x,y) is the pixel value of the image at (x,y), is the average pixel value of the image, M and N are the size of the image; the average brightness

[0160] If the evaluation results indicate poor image quality, the image is enhanced using an adaptive histogram equalization algorithm. This algorithm divides the image into multiple small blocks, performs histogram equalization on each block, and then merges the processed blocks using bilinear interpolation. This enhances the overall brightness and contrast of the image while preserving image details.

[0161] Next, camera calibration techniques are used to obtain the camera's position o and viewing direction d in 3D space. Using the ray casting principle of volume rendering, rays are cast from the camera position o along the viewing direction d. The points where the rays intersect the scene are the locations corresponding to the image pixels in 3D space.

[0162] In determining the initial radius r of the 3D Gaussian sphere i When , not only the depth information D of the image and the average object size S of the scene are combined, but also the texture complexity T of the image is considered. Texture complexity is measured by calculating the entropy value of the gray-level co-occurrence matrix of the image. Using the formula Among them, α is the adjustment coefficient, which is used to control the overall size of the radius, and β is the texture influence coefficient, which is used to adjust the influence of texture complexity on the radius.

[0163] For the initial color C of the Gaussian sphere i , using a multi-channel fusion approach. First, convert the image from RGB color space to HSV color space, and extract the RGB channel and the lightness channel information in the HSV color space respectively. Perform weighted fusion on these four channels, and the weights are dynamically adjusted according to the brightness distribution and color saturation of the image. For example, for areas with low brightness and high saturation, appropriately increase the weight of the RGB channel; for areas with high brightness and low saturation, increase the weight of the lightness channel. In this way, by comprehensively considering the information of different channels, a more accurate Gaussian sphere initial color C is obtained. i , complete the initialization of the 3D Gaussian sphere.

[0164] 2. Lighting adjustment network construction and weight prediction

[0165] In the illumination adjustment network construction and weight prediction phase, we use a convolutional neural network integrated with an attention mechanism to predict illumination weights for low-light images. The specific algorithm processing process is as follows:

[0166] 2.1. Network Architecture: The lighting control network utilizes a convolutional neural network architecture, consisting of multiple convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract local features from the image, while pooling layers reduce the dimensionality of the feature map, reducing computational effort and preventing overfitting. Multiple convolutional and pooling layers are stacked alternately to gradually extract high-level features from the image. At the end of the network, a fully connected layer integrates the extracted features and outputs the final prediction result.

[0167] 2.2 Local illumination initialization: Get the camera's position O in 3D space as the initialization starting point, and get the camera's viewing direction D. Based on this, construct the ray equation R(t) = O + tD(t∈R), which is used to determine the propagation path of the light from the camera in 3D space. Using the Monte Carlo simulation method, randomly sample N points P on the ray. i (i=1,2,…,N). These sampling points are used to simulate the propagation of light in the scene to obtain more accurate lighting information. For each sampling point P i, calculate the propagation probability P of light at this point based on its occlusion in the scene and material properties trans (P i The occlusion situation can be determined by comparing the positional relationship between the sampling point and the objects in the scene. If the sampling point is located inside the object or is blocked by the object, the probability of light propagation is low. The material properties determine the reflection, refraction, and absorption characteristics of the light when it encounters the material. For example, for metal materials, the light reflection is strong and the propagation probability is low; for transparent materials, the light propagation probability is relatively high. Using this information, a suitable light propagation probability model is used to calculate P trans (P i ). According to the calculated light propagation probability, the light intensity of each sampling point is adjusted. Initial light intensity I init (P i ) can be obtained by formula I init (P i )=I0×P trans (P i ), where I0 is the initial light intensity. By comprehensively calculating the light intensity of all sampling points, the initial light intensity distribution of each location is obtained, completing the local light initialization.

[0168] 2.3. Integration of attention mechanism: The low-light image after image preprocessing and 3D Gaussian sphere initialization steps is input into the light adjustment network. First, the convolution layer extracts features from the image to obtain a series of feature maps F. Then, these feature maps are subjected to global average pooling operations to compress each feature map into a global feature vector G, so that the network can obtain the global information of the image. The global feature vector G is processed by two fully connected layers. The first fully connected layer maps the global feature vector to a low-dimensional space, and the second fully connected layer maps it back to the original dimension to obtain the attention weight vector A. Each element in the attention weight vector A corresponds to the importance of each position in the feature map F. In this way, the network can automatically learn the importance of different areas in the image. The attention weight vector A is weightedly fused with the feature map F. Specifically, the feature value of each position in the feature map F is multiplied by the corresponding attention weight to obtain the weighted feature map F. att In this way, the network can pay more attention to the important areas in the image in subsequent processing, improving the accuracy of illumination weight prediction. The feature map F after attention mechanism processing att , and then processed by a series of convolutional layers and fully connected layers, the final output is the illumination weight value w of each 3D Gaussian sphere i These weight values ​​will be used in the subsequent 3D reconstruction process to adjust the illumination contribution of each Gaussian sphere to the final reconstructed image, thereby optimizing the illumination effect for low-light scenes.

[0169] Through the above algorithm processing, the illumination adjustment network can effectively initialize the local illumination of low-light images and accurately predict the illumination weight value of each 3D Gaussian sphere, providing key illumination information for subsequent 3D reconstruction, which helps to improve the quality of 3D reconstruction of low-light scene images.

[0170] 3. 3D reconstruction based on 3D Gaussian sphere

[0171] In the process of generating the reconstruction view of the normal lighting 3D scene, according to the color C of the 3D Gaussian ball i , position information and lighting weight value w i , using the improved rendering formula

[0172]

[0173] Perform rendering calculation to obtain the normal lighting 3D scene reconstruction view C nor When generating the reconstruction view of the low-light 3D scene, further adjust the lighting weight value w i , simulate the lighting effect in low-light environment and obtain the low-light 3D scene reconstruction view C low (Formula is the same as C nor ). Using perceptual loss, structural similarity loss and color consistency loss to adjust C i 、w i 、r i The 3D reconstruction algorithm is optimized based on the influence range of the sample points. The perceptual loss is calculated by comparing the difference between the reconstructed image and the real image in the feature space of a pre-trained convolutional neural network. The structural similarity loss uses the Structural Similarity Index (SSIM) to measure the structural similarity of the images. The color consistency loss calculates and sums the differences between different color channels to ensure that the color of the reconstructed image remains consistent under different lighting conditions.

Claims

1. A 3D Gaussian sphere-based 3D reconstruction method for low-light scene images, characterized in that: The following steps are involved: 1) Acquire a low-light scene image and evaluate the image quality. If the image quality is unqualified, perform image enhancement processing on the low-light scene image to obtain an enhanced image; 2) Initializing the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and volume rendering technology to obtain the initial position, initial radius and initial color of the 3D Gaussian sphere, and writing the initialized 3D Gaussian sphere information into the enhanced image; 3) Constructing a light regulation network based on a convolutional neural network integrated with an attention mechanism; 4) Inputting the enhanced image obtained in step 2) into the illumination adjustment network, and using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere; 5) Based on the initial information of the 3D Gaussian sphere and the illumination weight value, the 3D scene is reconstructed using the 3D reconstruction algorithm to obtain the 3D scene reconstruction view C under normal illumination. nor .

2. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: Image quality is characterized by the image entropy value E, contrast C and brightness mean L; When the entropy value E is within the preset range [E min ,E max ], the contrast C is within the preset range [C min ,C max ], or the average brightness L is within the preset range [L min ,L max ], the image quality is unqualified; Among them, the entropy value E, contrast C and brightness mean L are as follows: Where p i is the probability of the pixel with gray value i appearing in the image; I(x,y) is the pixel value of the image at (x,y), is the average pixel value of the image, M and N are the dimensions of the image.

3. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: In step 1), the step of performing image enhancement processing on the low-light scene image includes: 1.1) Divide the low-light scene image into multiple image blocks and perform histogram equalization on each image block; 1.2) The processed image blocks are merged by bilinear interpolation method to obtain the enhanced image.

4. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: In step 2), the step of obtaining the initial position of the 3D Gaussian sphere in the 3D space includes: Obtain the camera's position o and viewing direction d in three-dimensional space through camera calibration technology; Using the ray casting principle in volume rendering technology, light is emitted from the camera position o along the observation direction d, and the point where the light intersects the scene is used as the initial position of the 3D Gaussian sphere.

5. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: Initial radius of the 3D Gaussian sphere α is the adjustment coefficient; S is the average object size of the scene; D is the depth information of the image; β is the texture influence coefficient; T is the texture complexity of the image; The initial color C of the 3D Gaussian sphere i It is obtained by weighted fusion of RGB channels and lightness channel information in HSV color space.

6. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: The light adjustment network includes multiple convolutional layers, pooling layers and fully connected layers; The convolutional layer is used to extract local features of the image; The pooling layer is used to reduce the dimension of the feature map; The fully connected layer is used to integrate the extracted features and output the final prediction results.

7. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: In step 4), the step of using the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere includes: 4.1) Obtain the camera's position O in 3D space as the initialization starting point and obtain the camera's viewing direction d, thereby constructing the ray equation R(t) = O + td; t∈R in the enhanced image; 4.2) Using the Monte Carlo simulation method, randomly sample N points P on the ray R(t) j ; j = 1, 2, ..., N; using sampling point P j Simulate the propagation of light in the scene; For each sampling point P j , calculate the propagation probability P of light at this point based on the occlusion of the sampling point in the scene and the material properties trans (P i ); Propagation probability P trans (P i ) is as follows: P trans (P i )=w reflect ·P reflect +w refract ·P refract +w absorb ·P absorb +w scatter ·P scatter (4) Where w reflect 、w refract 、w absorb 、w scatter represents weight; P reflect 、P refract 、P absorb 、P scatter is the probability of reflection, refraction, absorption and scattering; 4.3) Adjust the light intensity at each sampling point to obtain the initial light intensity; where the sampling point P j The initial light intensity I init (P i )=I0×P trans (P i );I0 is the initial light intensity; 4.4) Comprehensively calculate the illumination intensity of all sampling points to obtain the initial illumination intensity distribution at each location point, completing the local illumination initialization of the enhanced image; 4.5) Using the convolutional layer of the illumination adjustment network to extract features from the enhanced image, a plurality of feature maps F are obtained; 4.6) Perform global average pooling on these feature maps to compress each feature map into a global feature vector G; 4.7) Use two fully connected layers to process the global feature vector G to obtain the attention weight vector a; 4.8) Perform weighted fusion of the attention weight vector A and the feature map F to obtain the feature map F processed by the attention mechanism att ; 4.8) Use the remaining convolutional layers and fully connected layers to transform the feature map F att Processing is performed to obtain the lighting weight value of each 3D Gaussian sphere.

8. The method for 3D reconstruction of low-light scene images based on 3D Gaussian sphere according to claim 1, characterized in that: 3D scene reconstruction view C under normal lighting nor As shown below: Where (x, y, z) is the position of the pixel in 3D space; (x i ,y i ,z i ) is the center position of the Gaussian sphere; w i is the illumination weight value of the i-th 3D Gaussian sphere; D r , A is the ambient light term and diffuse reflection term; C i is the initial color of the Gaussian sphere; r i is the initial radius of the Gaussian sphere.

9. A device based on the method for 3D reconstruction of low-light scene images according to any one of claims 1 to 8, characterized in that: Including camera, image enhancement module, 3D Gaussian sphere initialization module, illumination adjustment module, and 3D reconstruction module; The camera acquires low-light scene images and transmits them to the image enhancement module; The image enhancement module evaluates the image quality. If the image quality is unqualified, the low-light scene image is subjected to image enhancement processing to obtain an enhanced image. If the image quality is qualified, the low-light scene image is used as the enhanced image. The 3D Gaussian sphere initialization module initializes the information of the 3D Gaussian sphere in the 3D space according to the camera parameters and the volume rendering technology, obtains the initial position, initial radius and initial color of the 3D Gaussian sphere, and writes the initialized 3D Gaussian sphere information into the enhanced image; The illumination adjustment module stores an illumination adjustment network; The illumination adjustment module inputs the enhanced image with 3D Gaussian sphere information into the illumination adjustment network, and uses the illumination adjustment network to predict the illumination weight value of each 3D Gaussian sphere; The 3D reconstruction module uses the 3D reconstruction algorithm to reconstruct the 3D scene based on the initial information of the 3D Gaussian sphere and the illumination weight value, and obtains the 3D scene reconstruction view C under normal illumination. nor .

10. An electronic device, characterized in that: Execute the method according to any one of claims 1 to 8.

Images