Camera image three-dimensional reconstruction method, device, equipment and medium

Abstract

The invention discloses a camera image three-dimensional reconstruction method and device, equipment and a medium, and relates to the technical field of computer vision, and the method comprises the steps: constructing a distortion model, a vignetting model and a fuzzy model, in the three-dimensional reconstruction, firstly, utilizing the constructed distortion model, vignetting model and fuzzy model to obtain a three-dimensional reconstruction model; different aberrations of a camera lens are predicted and added into an image to form an aberration image added with the aberration of the camera lens, then three-dimensional reconstruction is carried out by utilizing the aberration image with corrected aberration, and correction and three-dimensional reconstruction of the aberration of the camera lens are carried out step by step in the process. Different from a method of continuously predicting lens parameters and reconstructing in a reconstruction process at the present stage, the method has the advantages that the influence of multi-view inconsistency caused by aberration on reconstruction is greatly reduced, the quality of a reconstructed object is improved, the difference between a three-dimensional model of the reconstructed object and a real scene is small, and the three-dimensional information of the object is clearly displayed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a camera image three-dimensional reconstruction method, device, equipment and medium. Background Art

[0002] 3D reconstruction refers to the restoration of 3D information of a scene (shape, texture, etc.) from natural images taken from multiple perspectives of the same scene; 3D reconstruction of objects is a common scientific problem and core technology in the fields of computer-aided geometric design (CAGD), computer graphics (CG), computer animation, computer vision, medical image processing, scientific computing and virtual reality, digital media creation, etc.; there are two main methods for generating three-dimensional representations of objects in computers; one is to use geometric modeling software to generate a three-dimensional geometric model of an object under human control through human-computer interaction, and the other is to obtain the geometric shape of a real object through certain means. This technology is relatively mature, and generally uses curves and surfaces with mathematical expressions to represent geometric shapes; the other is the 3D reconstruction process, which uses mathematical processes and computer technology to restore the three-dimensional information (shape, etc.) of an object using two-dimensional projection, including steps such as data acquisition, preprocessing, point cloud stitching and feature analysis.

[0003] At present, in the task of 3D reconstruction, the aberration caused by the lens design of real cameras destroys the assumption of ideal imaging, resulting in a decrease in the consistency between multi-view images, which in turn affects the quality of reconstruction. Therefore, the correction of aberrations has become a top priority. The specific manifestations of aberrations in images include blur, dispersion, distortion and vignetting; blur will cause the edges and details of objects in the image to be unclear, reducing the overall sharpness of the image, making it difficult to accurately capture the fine structure of the scene during 3D reconstruction; dispersion (chromatic aberration) will cause light of different wavelengths to deviate from the focus to varying degrees in the image, resulting in colored stripes or halos on the edges, destroying the color consistency of the image; distortion will cause the geometric shape of the image to be distorted, turning straight lines into curves, or the shape of objects into deformation, especially in fisheye lenses or wide-angle lenses, affecting the geometric accuracy of 3D reconstruction; vignetting refers to the phenomenon that the surroundings of the image are darker than the central part, which will cause uneven image brightness, making it difficult to accurately capture the details in the brightness change area during 3D reconstruction.

[0004] At present, in the three-dimensional reconstruction of objects, images of the object from different perspectives are directly used for three-dimensional reconstruction. During the reconstruction, the camera parameters are predicted and the aberrations of the images from different perspectives are directly corrected according to the camera parameters in the three-dimensional reconstruction to complete the three-dimensional reconstruction of the object. However, this method continuously predicts and corrects the aberrations during the three-dimensional reconstruction process. During the correction process, the multi-perspective inconsistency of the predicted aberrations will greatly reduce the quality of the reconstructed object, causing the reconstructed three-dimensional model to often be quite different from the real scene, making it difficult to reflect the three-dimensional information of the object. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, equipment and medium for three-dimensional reconstruction of camera images, which can solve the problem in the prior art that the current method continuously predicts and corrects aberrations during the three-dimensional reconstruction process. During the correction process, the multi-view inconsistency of the predicted aberrations will greatly reduce the quality of the reconstructed object, causing the reconstructed three-dimensional model to often be significantly different from the real scene, making it difficult to reflect the three-dimensional information of the object.

[0006] An embodiment of the present invention provides a camera image 3D reconstruction method, comprising the following steps:

[0007] Get natural images taken by cameras from multiple perspectives of the same scene;

[0008] Construct distortion model, vignetting model and blur model;

[0009] For each natural image, the distortion model is used to predict the offset of the camera lens aberration represented by each coordinate point of the natural image, and the natural image is distorted according to the offset to obtain a distorted image; the vignetting model is used to predict the brightness gain of each area in the distorted image representing the camera lens aberration, and the vignetting of the distorted image is adjusted according to the brightness gain to obtain a bright and dark image; the fuzzy model is used to predict the point spread function of each area in the bright and dark images representing the camera lens aberration, and the fuzzy dispersion fitting is performed on the bright and dark images according to the point spread function to obtain various aberration images under multiple viewing angles in the same scene;

[0010] The three-dimensional scene is reconstructed according to each aberration image.

[0011] Preferably, the construction of the distortion model includes:

[0012] The structure of the distortion model is sequentially: a first 1×1 convolutional layer, a first ReLU activation function, a second 1×1 convolutional layer, a second ReLU activation function, a third 1×1 convolutional layer, and a first Tanh activation function;

[0013] The first 1×1 convolution layer in the distortion model is used to map the input features from 2D coordinates to a high-dimensional feature space; the first ReLU activation function is used to increase the nonlinear expression; the second 1×1 convolution layer is used to map the features to the target dimension; the second ReLU activation function is used to increase the nonlinear expression again; the third 1×1 convolution layer is used to map the processed features to the output optical flow vector dimension, which includes a horizontal offset and a vertical offset; the first Tanh activation function is used to normalize the output value to the [-1,1] interval, which represents the normalized image coordinate system.

[0014] Preferably, the construction of the vignetting model includes:

[0015] The vignetting model is formed by sequentially arranging a plurality of learnable two-dimensional Gaussian functions, and the structure of the vignetting model is non-network;

[0016] Each two-dimensional Gaussian function is configured with a learnable parameter, which is used to learn the center position and diffusion range in the image when processing the corresponding image.

[0017] Preferably, the construction of the fuzzy model includes:

[0018] The structure of the blur model is a 1×1 convolution layer + a ReLU activation function in the first layer, three residual modules in the second layer, and a 1×1 convolution layer + a ReLU activation function in the third layer.

[0019] The 1×1 convolutional layer in the first layer of the fuzzy model is used to map from the input channel to the intermediate feature space, and a ReLU activation function is used to increase the nonlinear expression;

[0020] The first residual module in the second layer of the fuzzy model is a 1×1 convolutional layer, which is used to adjust the number of channels or feature distribution; the second residual module is a normalization layer BatchNorm, which is used to stabilize training; the third residual module is a ReLU activation function, which is used to increase nonlinear expression again;

[0021] The 1×1 convolution layer and a ReLU activation function in the third layer of the blur model are used to generate different output feature channels respectively, and each output feature channel corresponds to a specific point spread function PSF feature.

[0022] Preferably, the optimization of the distortion model, the vignetting model and the blur model includes:

[0023] Obtain the error loss between a natural image and a distorted image, obtain the error loss between a natural image and a bright and dark image, and obtain the error loss between a natural image and an aberration image;

[0024] According to the error loss between the natural image and the distorted image, the error loss between the natural image and the bright and dark images, and the error loss between the natural image and the aberration image, gradient backpropagation is performed to cyclically update the parameters of the distortion model, the vignetting model and the blur model to optimize the distortion model, the vignetting model and the blur model.

[0025] The embodiment of the present invention further provides a camera image 3D reconstruction device, comprising:

[0026] The acquisition module is used to obtain natural images taken by cameras from multiple perspectives in the same scene;

[0027] Model building module, used to build distortion model, vignetting model and blur model;

[0028] A processing module is used to predict, for each natural image, the offset of each coordinate point of the natural image representing the camera lens aberration using a distortion model, distort the natural image according to the offset, and obtain a distorted image; predict the brightness gain of each area in the distorted image representing the camera lens aberration using a vignetting model, adjust the vignetting of the distorted image according to the brightness gain, and obtain a bright and dark image; predict the point spread function of each area in the bright and dark image representing the camera lens aberration using a fuzzy model, perform fuzzy dispersion fitting on the bright and dark image according to the point spread function, and obtain various aberration images under multiple viewing angles in the same scene;

[0029] The reconstruction module is used to reconstruct a three-dimensional scene according to each aberration image.

[0030] An embodiment of the present invention further provides an electronic device, including a memory and a processor;

[0031] The memory is used to store computer programs;

[0032] The processor is used to implement the steps of the above-mentioned camera image three-dimensional reconstruction method when executing the computer program stored in the memory.

[0033] An embodiment of the present invention further provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the steps of the camera image three-dimensional reconstruction method as described above are implemented.

[0034] The embodiments of the present invention provide a camera image 3D reconstruction method, device, equipment and medium. Compared with the prior art, the advantages thereof are as follows:

[0035] The present invention constructs a distortion model, a vignetting model and a blur model, uses the distortion model to predict the offset of the camera lens aberration represented by each coordinate point in the image, and distorts the image according to the offset; uses the vignetting model to predict the brightness gain of the camera lens aberration represented by each area in the image, and adjusts the brightness of the image according to the brightness gain; uses the blur model to predict the point spread function of the camera lens aberration represented by each area in the image, and performs fuzzy dispersion fitting on the image according to the point spread function, thereby obtaining an aberration image, and finally uses the aberration images of the object at different viewing angles to reconstruct the three-dimensional scene; in the three-dimensional reconstruction of the present invention, firstly, the constructed distortion model, vignetting model and blur model are used to predict different aberrations of the camera lens and add them to the image, This forms an aberration image with the camera lens aberration added. This operation also enhances the visual effects and features between multi-view images, thereby improving the consistency between multi-view images. The aberration image with corrected aberration is then used for 3D reconstruction. This process performs the correction of camera lens aberration and 3D reconstruction in steps. Unlike the current method of continuously predicting lens parameters and reconstructing during the reconstruction process, it offsets the inconsistency of multi-view image correction aberrations caused by continuously predicting lens parameters for images from different viewpoints during the reconstruction process, thereby greatly reducing the impact of multi-view inconsistency caused by aberration on reconstruction, improving the quality of the reconstructed object, and the 3D model of the reconstructed object is slightly different from the real scene, clearly displaying the 3D information of the object.

[0036] In addition, the present invention accurately predicts the aberration of the camera lens before the three-dimensional reconstruction task begins by constructing a distortion model, a vignetting model and a blur model. Compared with the current method of continuously predicting and adding in three-dimensional reconstruction, the accuracy of the geometric structure of the reconstructed object model is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the overall process of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0038] Figure 2 A simplified flowchart of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of aberration model processing of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a three-dimensional reconstruction example of a camera image three-dimensional reconstruction method provided by an embodiment of the present invention;

[0041] Figure 5 A network structure and processing schematic diagram of a distortion model of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0042] Figure 6 A network structure and processing diagram of a vignetting model of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0043] Figure 7 A network structure and processing schematic diagram of a fuzzy model of a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0044] Figure 8 A schematic diagram of reconstructing an image scene taken by a camera in a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0045] Fig. 9 A schematic diagram of reconstructing an image scene taken by a drone using a camera image 3D reconstruction method provided by an embodiment of the present invention;

[0046] Fig.10 A schematic diagram of reconstructing an image scene captured by a fisheye lens in a camera image 3D reconstruction method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0048] See also Figure 1 , an embodiment of the present invention provides a camera image three-dimensional reconstruction method, comprising the following steps:

[0049] Step 1: The standard reconstruction pipeline processes the image into a clear image.

[0050] Step 2: Input the clear image into the aberration model for processing; specifically including:

[0051] ① Input the clear image into the distortion model for image distortion; specifically, the optical flow represented by the network is used, that is, the input spatial information outputs the offset of each pixel and the image is distorted according to the offset.

[0052] ② Input the result of image distortion into the vignetting model for brightness adjustment; specifically, adjust the image brightness based on the vignetting represented by the mixed Gaussian to fit the uneven brightness.

[0053] ③ Input the brightness-adjusted image into the blur model to fit the blurred dispersion; specifically, input the spatial information into the PSF fitting network to generate a pixel-by-pixel point spread function, and apply deconvolution to the image to fit the degradation effect caused by optical aberrations.

[0054] Step 3: Calculate the loss by combining the aberration fitting result with the training data.

[0055] Step 4: Perform back-propagation gradient operation to update the scene reconstruction and aberration model.

[0056] Step 5: Repeat steps 1 to 4 until the scene reconstruction is complete.

[0057] The present invention is based on (3D Gaussian Splatting, 3D-GS), which represents 3D Gaussian speckle, and uses a group of Gaussian primitives to represent the geometric and appearance information of a three-dimensional scene. The spatial position, shape and color of each primitive are continuously optimized during the training process.

[0058] The present invention adopts the idea of ​​joint optimization to calibrate the aberration in the 3D reconstruction task, thereby reducing the influence of aberration on the reconstruction consistency. Specifically, it consists of two parts: a standard reconstruction pipeline and an aberration model. The simplified flowchart is as follows: Figure 2 As shown in the figure, the standard reconstruction pipeline uses multi-view consistency constraints to make the reconstructed scene tend to be clear and distortion-free. However, real images usually contain various aberrations. Based on this, the present invention designs an aberration model to fit the different aberrations that actually exist in the lens by adding calibrated aberration effects to the clear image. Finally, the loss between the fitted aberration and the actual image is calculated, and the gradient is back-propagated to optimize the aberration model and scene reconstruction at the same time.

[0059] The present invention effectively distinguishes clear reconstruction results from training images with various aberrations by introducing an aberration model; that is, aberrations are added to clear reconstruction results through the aberration model, so that when calculating the loss, the quality of the reconstructed scene will not be affected by the aberrations; the aberration model can calibrate the aberrations in the image, so that the scene reconstruction can extract useful information from these training images with aberrations without being interfered by the aberrations; therefore, even if the training data is full of various aberrations, through the processing of the aberration model, the framework can still effectively use these data for training, indirectly use the images with aberrations, and at the same time ensure the quality of the reconstruction results.

[0060] According to the characteristics of different aberrations, the present invention designs a special network for fitting and constructs a corresponding workflow based on the actual imaging process; Figure 3As shown in the figure, it is demonstrated how to fit the clear image with the real image through the calibrated aberration model, so as to simulate the imaging process of the real camera; the aberration model processes different aberration types through three independent modules: distortion, vignetting and blur; the present invention utilizes the powerful fitting ability of the neural network to construct three independent aberration processing modules, which are respectively responsible for the correction of distortion, vignetting and blur. Each module designs a corresponding network structure according to different aberration characteristics, but the overall framework remains similar, and all use 1×1 convolutional layers to perform the main operations of the model. Although the overall structures of the three are similar, we have made corresponding adjustments at the detail level according to the tasks they respectively handle to ensure the effective processing of different aberrations. For example, the activation function of the last layer is determined according to the characteristics of different aberrations.

[0061] Specifically:

[0062] 1. Distortion model.

[0063] The network structure and processing process of the distortion model are as follows Figure 5 As shown; its network layers from left to right are:

[0064] ①1×1 convolution layer: The first layer is a 1×1 convolution layer, which maps the input features from 2D coordinates to a high-dimensional feature space; the number of output channels is adjustable to improve feature expression capabilities.

[0065] ②ReLU activation function: The activation function is ReLU, which increases nonlinear expression capabilities and avoids linear mapping.

[0066] ③1×1 convolution layer: The second 1×1 convolution layer is used to further process the features and compress or map them to the target dimension; this design is suitable for lightweight networks and avoids complex convolution operations with a large receptive field.

[0067] ④ReLU activation function: Use the ReLU activation function again to activate nonlinearity.

[0068] ⑤1×1 convolution layer: The third 1×1 convolution layer is used to map the processed features to the output optical flow vector dimension (usually two-dimensional: horizontal and vertical offset).

[0069] ⑥Tanh activation function: The last layer uses the Tanh activation function to normalize the output value to the [-1,1] interval; here [-1,1] refers to the normalized image coordinate system, that is, the center of the image is the origin and the boundary is 1 or -1.

[0070] The processing flow of the distortion model is:

[0071] ① Input: 2D coordinate grid; its input is a two-dimensional coordinate grid with a shape of (H, W, 2), and each coordinate point represents the (x, y) position on the image plane.

[0072] ② Output: offset (optical flow) of each coordinate; the output is a two-dimensional vector with the same shape as the input grid, with a shape of (H, W, 2), which represents the optical flow offset of each coordinate point; each offset value is predicted by the network, usually corresponding to the horizontal offset (u) and vertical offset (v) of the optical flow.

[0073] 2. Vignette model.

[0074] The network structure and processing process of the vignetting model are as follows: Figure 6 As shown; its structure is non-network, but it can learn parameters, learning the mean and standard deviation of the Gaussian function; including:

[0075] ① Multiple Gaussian functions: These Gaussian distributions are learned with different weights, center locations, and diffusion ranges.

[0076] ② Adding all Gaussian functions together can obtain the final brightness gain (vignetting), that is, the brightness change of each pixel.

[0077] 3. Fuzzy model.

[0078] The network structure and processing process of the fuzzy model are as follows Figure 7 Shown; including:

[0079] ① Input: such as Figure 7 The leftmost cube in the figure has a 3D tensor of shape (H, W, C_{\text{input}}), where H and W represent the height and width of the grid; C_in represents the number of input channels, which can be the original dimensions of the coordinates (e.g., x, y) or features embedded in a high-dimensional space.

[0080] ②The first stage is a 1×1 convolution layer and a ReLU activation function. The first layer uses a 1×1 convolution to map the input channel to the intermediate feature space. The feature of each position is based only on itself (without spatial receptive field expansion). This structure is suitable for point-by-point feature mapping. The ReLU activation function introduces nonlinear characteristics to improve the network’s expression capabilities.

[0081] ③The second stage consists of three residual modules in the middle. The first residual module is a 1×1 convolutional layer, which is used to further process features and adjust the number of channels or feature distribution. The second residual module is a normalization layer, BatchNorm, which is used to stabilize training. The third residual module is a ReLU activation function, which is used to increase nonlinear expression again.

[0082] The residual module retains the original features through skip connections while superimposing deeper feature representations; the three residual modules indicate that this residual structure is repeated three times, aiming to gradually enhance the feature expression.

[0083] ④ Output branch; its network contains multiple 1×1 convolutional layers and branches of ReLU activation functions, which are used to generate different output feature channels. Each output branch corresponds to a specific point spread function (PSF) feature.

[0084] ⑤ The final output (point spread function, equivalent to the blur kernel of each coordinate) inputs eleven three-dimensional tensors with a shape of (H, W, C_in), where H and W are consistent with the input grid, C_in represents the number of output channels, and outputs the PSF of three channels (RGB).

[0085] The processing flow is:

[0086] Input: An encoding of a spatial location (usually a coordinate grid such as (x,y) or a high-dimensional embedding thereof).

[0087] Output: Point spread function (PSF) corresponding to each position, usually represented as an image patch or a three-channel feature map.

[0088] The distortion model, vignetting model and blur model are integrated. The distortion model is used to repair geometric distortion and remove distortion; the vignetting model is used to correct brightness distribution and eliminate brightness attenuation in edge areas; the blur model is used to enhance edge details and restore clear images; through the serial processing of these three models, a high-quality image with true geometry, clear details and uniform brightness is finally obtained; this design strictly follows the physical characteristics of optical problems to ensure that the correction effect is accurate and efficient.

[0089] 4. Calculate the loss by combining the aberration fitting result with the training data.

[0090] ①The results of aberration fitting.

[0091] The goal of the model is to generate a set of parameters that can accurately describe these aberrations (such as the distorted displacement field, the blurred point spread function, and the brightness gain map of the vignetting); these parameters are used in the output of the renderer to add aberrations to the aberration-free image through the fitted aberration parameters to generate a simulated aberrated image; then, by comparing the aberration-free image with the training data (containing the real aberrations), the difference between the two is calculated.

[0092] ②Calculate the loss.

[0093] The difference is measured by a loss function (such as mean squared error or perceptual loss) and is used to guide model optimization so that the aberration parameters it generates can more accurately reproduce the true optical properties in the training images; this process ensures that the model can capture and reproduce the aberration characteristics during training while complying with multi-view consistency.

[0094] ③Back propagation.

[0095] Finally, through back-propagation, the results of the loss function are used to optimize the model parameters, thereby gradually improving the accuracy of the model in correcting optical aberrations. This process is iterated repeatedly throughout the training process until the aberration correction result fitted by the model converges to a smaller value with the actual annotation error of the training data, thus completing the model training.

[0096] 5. Update scene reconstruction and aberration models.

[0097] ① Loss calculation: The output of the model is obtained through forward propagation, and the loss function is used to measure the error between the output and the true value.

[0098] ② Gradient calculation: Through back propagation, the chain rule is used to calculate the gradient of the loss for each parameter layer by layer, and the error information is propagated from the output layer to the input layer layer by layer.

[0099] ③ Parameter update: Use optimization algorithms (such as gradient descent) to adjust model parameters according to the gradient, so that the error is gradually reduced until the model converges.

[0100] When updating the scene and aberration models, the gradient back-propagation mechanism is utilized. Since the properties of the scene and the characteristics of the aberration model are represented by learnable parameters in deep learning, these parameters can be optimized by calculating the gradient of the error and performing back-propagation, ultimately achieving accurate modeling of the scene and optical aberrations.

[0101] like Figure 8 , Fig. 9 and Fig.10 As shown, the results of 3D reconstruction using images taken by a camera, an image taken by a drone, and an image taken by a fisheye lens are shown respectively; Figure 8 The left side shows the image captured by the camera, the middle shows the result of three-dimensional reconstruction using 3D-GS software, and the right side shows the result of three-dimensional reconstruction using the method of the present invention; Fig. 9 The left side shows the image taken by the drone, the middle shows the result of three-dimensional reconstruction using the 3D-GS software, and the right side shows the result of three-dimensional reconstruction using the method of the present invention; Fig.10The left side shows the result of 3D reconstruction using 3D-GS software, the middle shows the result of 3D reconstruction using the method of the present invention, and the right side shows the image taken by the fisheye lens. It can be seen that in the case of multiple aberrations existing at the same time, the present invention can still reconstruct the scene with high fidelity.

[0102] The embodiment of the present invention further provides a camera image 3D reconstruction device, comprising:

[0103] The acquisition module is used to obtain natural images taken by cameras from multiple perspectives in the same scene.

[0104] Model building module, used to build distortion model, vignetting model and blur model.

[0105] The processing module is used to predict, for each natural image, the offset of each coordinate point of the natural image representing the camera lens aberration using a distortion model, distort the natural image according to the offset, and obtain a distorted image; predict the brightness gain of each area in the distorted image representing the camera lens aberration using a vignetting model, adjust the vignetting of the distorted image according to the brightness gain, and obtain a bright and dark image; predict the point spread function of each area in the bright and dark images representing the camera lens aberration using a fuzzy model, perform fuzzy dispersion fitting on the bright and dark images according to the point spread function, and obtain various aberration images under multiple viewing angles in the same scene.

[0106] The reconstruction module is used to reconstruct a three-dimensional scene according to each aberration image.

[0107] An embodiment of the present invention further provides an electronic device, including a memory and a processor.

[0108] The memory is used to store computer programs.

[0109] When the processor is used to execute the computer program stored in the memory, the steps of the above-mentioned method for three-dimensional reconstruction of camera images are implemented.

[0110] An embodiment of the present invention further provides a computer-readable storage medium for storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for three-dimensional reconstruction of camera images are implemented.

[0111] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A camera image three-dimensional reconstruction method, characterized in that: The following steps are involved: Get natural images taken by cameras from multiple perspectives of the same scene; Construct distortion model, vignetting model and blur model; For each natural image, the distortion model is used to predict the offset of the camera lens aberration represented by each coordinate point of the natural image, and the natural image is distorted according to the offset to obtain a distorted image; the vignetting model is used to predict the brightness gain of each area in the distorted image representing the camera lens aberration, and the vignetting of the distorted image is adjusted according to the brightness gain to obtain a bright and dark image; The fuzzy model is used to predict the point spread function of the camera lens aberration in each area of ​​the bright and dark images, and the fuzzy dispersion fitting is performed on the bright and dark images according to the point spread function to obtain the aberration images under multiple viewing angles in the same scene. The three-dimensional scene is reconstructed according to each aberration image.

2. The camera image 3D reconstruction method according to claim 1, characterized in that: The construction of the distortion model includes: The structure of the distortion model is sequentially: a first 1×1 convolutional layer, a first ReLU activation function, a second 1×1 convolutional layer, a second ReLU activation function, a third 1×1 convolutional layer, and a first Tanh activation function; The first 1×1 convolution layer in the distortion model is used to map the input features from 2D coordinates to a high-dimensional feature space; the first ReLU activation function is used to increase the nonlinear expression; the second 1×1 convolution layer is used to map the features to the target dimension; the second ReLU activation function is used to increase the nonlinear expression again; the third 1×1 convolution layer is used to map the processed features to the output optical flow vector dimension, which includes a horizontal offset and a vertical offset; the first Tanh activation function is used to normalize the output value to the [-1,1] interval, which represents the normalized image coordinate system.

3. The camera image 3D reconstruction method according to claim 1, characterized in that: The construction of the vignetting model includes: The vignetting model is formed by sequentially arranging a plurality of learnable two-dimensional Gaussian functions, and the structure of the vignetting model is non-network; Each two-dimensional Gaussian function is configured with a learnable parameter, which is used to learn the center position and diffusion range in the image when processing the corresponding image.

4. The camera image 3D reconstruction method according to claim 1, characterized in that: The construction of the fuzzy model includes: The structure of the blur model is a 1×1 convolution layer + a ReLU activation function in the first layer, three residual modules in the second layer, and a 1×1 convolution layer + a ReLU activation function in the third layer. The 1×1 convolutional layer in the first layer of the fuzzy model is used to map from the input channel to the intermediate feature space, and a ReLU activation function is used to increase the nonlinear expression; The first residual module in the second layer of the fuzzy model is a 1×1 convolutional layer, which is used to adjust the number of channels or feature distribution; the second residual module is a normalization layer BatchNorm, which is used to stabilize training; the third residual module is a ReLU activation function, which is used to increase nonlinear expression again; The 1×1 convolution layer and a ReLU activation function in the third layer of the blur model are used to generate different output feature channels respectively, and each output feature channel corresponds to a specific point spread function PSF feature.

5. The camera image 3D reconstruction method according to claim 1, characterized in that: The optimization of the distortion model, the vignetting model and the blur model includes: Obtain the error loss between a natural image and a distorted image, obtain the error loss between a natural image and a bright and dark image, and obtain the error loss between a natural image and an aberration image; According to the error loss between the natural image and the distorted image, the error loss between the natural image and the bright and dark images, and the error loss between the natural image and the aberration image, gradient backpropagation is performed to cyclically update the parameters of the distortion model, the vignetting model and the blur model to optimize the distortion model, the vignetting model and the blur model.

6. A camera image three-dimensional reconstruction device, characterized in that: include: The acquisition module is used to obtain natural images taken by cameras from multiple perspectives in the same scene; Model building module, used to build distortion model, vignetting model and blur model; A processing module is used to predict, for each natural image, the offset of each coordinate point of the natural image representing the camera lens aberration using a distortion model, distort the natural image according to the offset, and obtain a distorted image; predict the brightness gain of each area in the distorted image representing the camera lens aberration using a vignetting model, adjust the vignetting of the distorted image according to the brightness gain, and obtain a bright and dark image; The fuzzy model is used to predict the point spread function of the camera lens aberration in each area of ​​the bright and dark images, and the fuzzy dispersion fitting is performed on the bright and dark images according to the point spread function to obtain the aberration images under multiple viewing angles in the same scene. The reconstruction module is used to reconstruct a three-dimensional scene according to each aberration image.

7. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is used to implement the steps of a camera image three-dimensional reconstruction method as claimed in any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the steps of a camera image three-dimensional reconstruction method as claimed in any one of claims 1 to 5.

Images