Light field three-dimensional imaging method and system based on neural network

Abstract

The present invention provides a light field three-dimensional imaging method and system based on a neural network. The method includes: using a microlens array light field camera to collect multi-view light field images to form a multi-view data set; constructing a convolutional neural network model for inputting light field images and outputting two-dimensional depth maps, which includes a convolution layer, a residual module, an SPP module and an attention mechanism module in sequence, for extracting features from all information of the light field image and retaining features of key information; training and testing the neural network model with the data set; inputting the light field image of the target scene into the neural network model to obtain a two-dimensional depth map of the target scene; transforming the image coordinates of the two-dimensional depth map to obtain a three-dimensional reconstructed image. The present invention adopts multi-view information fusion, realizes end-to-end learning through a neural network, directly generates a depth map for the image, can reliably restore the three-dimensional structure of various scenes, and produce accurate three-dimensional reconstruction results; simplifies the network calculation amount and enhances the robustness of the algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of light field three-dimensional imaging, and in particular relates to a light field three-dimensional imaging method based on a neural network and a light field three-dimensional imaging system. Background Art

[0002] As an important means of perceiving the real world, three-dimensional imaging technology performs well in acquiring information. In recent years, it has been widely used in autonomous driving, virtual reality, face recognition, medical health and other fields.

[0003] Light field cameras are an advanced technology for obtaining three-dimensional information that has attracted much attention in recent years. By installing a microlens array on its image sensor, it can capture light in different directions at the same time. These microlenses allow the incident light to form tiny sub-images on the image plane. Each sub-image contains information about a point on the object scene, including its position, direction, and intensity. By merging all sub-images, a multi-dimensional light field image can be formed. By analyzing the multi-dimensional image, the depth, shape, and position information of the three-dimensional scene can be restored.

[0004] The current research methods for restoring 3D depth information of light fields mainly include the following categories:

[0005] 1. 3D imaging method based on single view: This method realizes 3D reconstruction of objects by searching for distinguishable feature points in the image, such as corners and edges. However, due to the uncertainty of 2D images in 3D space, it is difficult to restore the precise geometric structure from a single image, and the reconstruction effect is poor.

[0006] 2. 3D imaging method based on multi-view geometry. This method combines the light field camera and multi-view geometry principles, which can easily utilize surrounding information. The reconstruction accuracy is determined by feature extraction. The existing image local feature algorithm can only extract simple texture features and cannot obtain the overall information of the target, which restricts the reconstruction accuracy.

[0007] 3. Monocular 3D imaging method based on structured light method. By projecting specially coded light patterns or light strips onto the photographed object, the depth information of the object is calculated by observing the deformation or change of the light patterns. For high-brightness or reflective objects, the structured light method is easily interfered with, and there is a problem of reduced accuracy for long-distance depth measurement, resulting in inaccurate depth measurement. Multiple stripe light sources need to be projected during shooting, and the reconstruction speed is greatly restricted.

[0008] 4. Focus stack algorithm. This algorithm uses different focus images in the image sequence to determine the depth position of each point in the scene by calculating the clarity evaluation value of the image. It has poor adaptability to transparent, reflective and weak texture scenes, is easily disturbed by noise, and has relatively limited processing for moving objects, resulting in limitations in its application in complex scenes.

[0009] 5. Light field 3D depth recovery algorithm based on deep learning. This algorithm establishes a known real depth label data set and trains to learn the mapping relationship between the light field image and the corresponding depth map. However, it uses the feature information in the horizontal, vertical and diagonal directions of the light field epipolar plane image, and does not use all the information of the sub-aperture image. It also needs to process redundant information from different perspectives, which requires more computational overhead and increases the amount of calculation.

[0010] In summary, the existing light field 3D imaging methods have the problems of poor reconstruction effect, low reconstruction accuracy, limited reconstruction speed, limited application in complex scenes, failure to utilize all the information of sub-aperture images and large amount of calculation. Summary of the invention

[0011] The purpose of the present invention is to solve the shortcomings of existing light field 3D imaging methods, such as poor reconstruction effect, low reconstruction accuracy, limited reconstruction speed, limited application in complex scenes, failure to utilize all information of sub-aperture images and large amount of calculation, and to provide a light field 3D imaging method and a light field 3D imaging system based on a neural network, which generate a high-completeness depth map for the light field image with the help of an efficient feature fusion method and spatial attention, and then perform a 3D visualization process.

[0012] To achieve the above purpose, the technical solution provided by the present invention is:

[0013] A light field three-dimensional imaging method based on a neural network is special in that it includes the following steps:

[0014] Step 1, using a microlens array light field camera to collect multi-view light field images to form a multi-view data set;

[0015] Step 2, constructing a convolutional neural network model, which is used to input a light field image and output its two-dimensional depth map, and includes a convolutional layer, a residual module, an SPP module and an attention mechanism module in sequence, which is used to extract features from all information of the input light field image and retain features of key information in the light field image;

[0016] Step 3, using the formed multi-view data set to train and test the constructed neural network model;

[0017] Step 4, inputting the light field image of the target scene into the trained neural network model to obtain a two-dimensional depth map of the target scene;

[0018] Step 5: transform the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

[0019] Further, step 3 includes the following steps: according to the image depth estimation value y and the image depth actual value y * The difference determines the loss function:

[0020] L(y * , y) = λL1 + L2

[0021] In the formula, the loss value L1 of the depth information is:

[0022]

[0023] The structural similarity loss function L2 is:

[0024]

[0025] In the formula, λ is the loss value weight of the depth information, N is the number of image pixels, and SSIM is the structural similarity function.

[0026] Furthermore, step 3 includes the following sub-steps:

[0027] Step 3.1, dividing the formed multi-view dataset into a training set and a test set;

[0028] Step 3.2, training the constructed neural network model using the divided training set;

[0029] Step 3.3, use the divided test set to verify the trained neural network model.

[0030] Furthermore, step 5 includes the following sub-steps: using the two-dimensional depth map of the target scene to infer the three-dimensional spatial coordinates corresponding to each pixel in the image, converting the image coordinates of the two-dimensional depth map into the three-dimensional point cloud coordinates corresponding to each pixel, and realizing the three-dimensional reconstructed image of the target scene:

[0031] X w =Z·(u-u0)d x / f

[0032] Y w =Z·(v-v0)d y / f

[0033] Z w =Z

[0034] In the formula, (X w ,Y w ,Z w ) is the world coordinate, (u0,v0) is the coordinate of the pixel point in the center of the image, (u,v) is the coordinate of the pixel point in the image, Z is the image depth value to be obtained, f is the focal length of the camera, d x and d y is the camera parameter.

[0035] A light field three-dimensional imaging system, which is special in that it includes:

[0036] A microlens array light field camera is used to collect light field images of target scenes;

[0037] A data processing module, used to input the collected light field image into the above-mentioned convolutional neural network model and output a two-dimensional depth map of the target scene;

[0038] The three-dimensional reconstruction module is used to transform the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

[0039] The advantages of the present invention are:

[0040] 1. The present invention uses a microlens array light field camera to collect multi-view light field images, and constructs a convolutional neural network model including a convolution layer, a residual module, an SPP module, and an attention mechanism module, which extracts features from all information of the light field image and retains features of key information, then calculates the depth value to output a two-dimensional depth map, and then performs coordinate transformation to obtain a three-dimensional reconstructed image. Therefore, the present invention adopts multi-view information fusion to improve information acquisition, and also implements end-to-end learning through the network to directly generate a depth map for the collected image, avoiding the need to design complex feature extraction and matching algorithms for different types of scenes. It has strong robustness in complex scenes, can reliably restore the real three-dimensional structure of various scenes, and produce accurate three-dimensional reconstruction results;

[0041] 2. The neural network structure of the present invention includes a convolutional layer, a residual module, an SPP module and an attention mechanism module. The network can utilize all the information of the light field image and dynamically select and focus on the key information in the input data, thereby effectively simplifying the computational complexity of the network, improving the performance of the neural network, retaining the sub-aperture light field image with a higher amount of information for depth estimation, and enhancing the robustness of the overall algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The features and advantages of the present invention will become more easily understood through the following description with reference to the accompanying drawings, in which:

[0043] Figure 1 is a flow chart of the neural network-based light field three-dimensional imaging method of the present invention;

[0044] Figure 2 is a diagram of an imaging system for collecting light field images in the method of the present invention;

[0045] Figure 3 It is a structural principle diagram of a neural network model in the method of the present invention;

[0046] Figure 4It is a structural principle diagram of the spatial pyramid pooling (SPP) module used by the neural network model in the method of the present invention;

[0047] Figure 5 It is a flow chart of the attention mechanism module used in the neural network model in the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in detail below with reference to the accompanying drawings by means of exemplary embodiments of the present invention. It should be noted that the following detailed description of the present invention is only for the purpose of illustration, and is not intended to limit the present invention.

[0049] The present invention provides a light field three-dimensional imaging method and a light field three-dimensional imaging system based on a neural network, which generate a high-completeness depth map for the light field image with the help of an efficient feature fusion method and spatial attention, and then perform three-dimensional reconstruction.

[0050] Reference Figure 1 , as an exemplary embodiment of the present invention, a light field three-dimensional imaging method based on a neural network comprises the following steps:

[0051] Step S1, using a microlens array light field camera to collect multi-view light field images to form a multi-view data set;

[0052] Step S2, constructing a convolutional neural network model, the neural network model is used to input a light field image and output its two-dimensional depth map, and sequentially includes a convolution layer, a residual module, an SPP module and an attention mechanism module, which is used to extract features from all information of the input light field image and retain features of key information in the light field image. The structure of the network model will be described in detail below;

[0053] Step S3, using the formed multi-view data set to train and test the constructed neural network model;

[0054] Step S4, inputting the light field image of the target scene into the trained neural network model to obtain a two-dimensional depth map of the target scene;

[0055] Step S5, transforming the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

[0056] like Figure 2 As shown, in step S1, the acquisition imaging system of the microlens array light field camera includes a main lens, a detector and a microlens array. The microlens array is composed of multiple independently distributed lenses, and the sub-aperture image is composed of pixels at the same position under different microlenses in the order of the microlens array. The sub-lenses have the same focal length but a smaller aperture, and the sub-aperture image has a larger depth of field range.

[0057] Reference Figure 3 For the convolutional neural network model in step S2, the residual module enhances the feasibility of neural network training to deal with the problem of gradient explosion, making it easier for the model to achieve convergence in the training process. The feature extraction of the convolutional layer and the residual module enables the network to learn relatively simple basic textures and local features, accurately capturing the local texture and semantics in the image. Through the application of the SPP module, the network is guided to learn the contextual semantic information of the target object in the scene more deeply. After passing through the SPP module, the target feature evolves into a complex feature with global semantic information of the scene, carrying the contextual semantic information of the target object. The multi-view feature outputs a preliminary depth map of the current scene through the subsequent matching stage. The attention mechanism module learns the features of different depths and retains the features of the overlapping areas in the images of different perspectives. Then, a more complete depth map is obtained through the regression function, and the network learns the accurate three-dimensional depth information in the image.

[0058] The structure of the SPP module is as follows Figure 4 As shown in the figure, a source image and multiple reference images are all passed through the SPP module, using the same network parameters. This module is used to expand the context information of the scene. The SPP model structure uses four average pooling layers of different sizes to compress features. The output of the pooling layer is connected to a 1×1 convolution layer to reduce the feature dimension, and the upsampled feature map is used. Finally, the feature maps of all levels are spliced ​​and output. The entire process is completely independent of the size of the input, so feature maps of any size can be processed.

[0059] The structure of the attention mechanism module is shown in Figure 5 The purpose of learning the attention mechanism is to learn important information from different modules and select different aperture perspectives. Since the sub-aperture contains a large amount of redundant data and the light field image is highly symmetrical, the attention module searches for important perspective images to simplify the computational complexity. The attention mechanism enables the network to retain richer detail features and achieve high-integrity 3D scene reconstruction.

[0060] In step S3, the intensity map of each field of view is input to the model, and 8 images can be input as a batch for calculation each time. The training network forward propagates to calculate the loss function, and the back propagation uses the optimizer to update the network parameters. The specific data flow transmission process of the network is that each sub-aperture image first passes through the residual module, and the extended convolution machine is introduced to give the neural network a larger receptive field. The obtained feature image is passed to the spatial pyramid pooling (SPP) module, and finally the attention map is used to predict the disparity of the central perspective through the disparity regression model, and the objective function is used to convert the feature map into a probability value to obtain the depth map corresponding to the reference image.

[0061] Specifically, step S3 includes the following sub-steps:

[0062] Step S3.1, dividing the formed multi-view data set into a training set and a test set, and 80 scenes can be selected for training and 20 scenes for testing;

[0063] Step S3.2, training the constructed neural network model using the divided training set;

[0064] Step S3.3, use the divided test set to verify the trained neural network model.

[0065] In particular, in step S3, the optimizer selects Adam, and the loss function is based on the image depth estimation value y and the image depth actual value y * The difference is determined by:

[0066] L(y * , y) = λL1 + L2

[0067] In the formula, the loss value L1 of the depth information is:

[0068]

[0069] The structural similarity loss function L2 is:

[0070]

[0071] Where λ is the loss value weight of the depth information, N is the number of image pixels, and SSIM is the structural similarity function. The structural similarity is calculated based on the brightness and contrast between images, which can retain the details in the depth map.

[0072] In some embodiments of the present invention, step S5 includes the following sub-steps: using the two-dimensional depth map of the target scene to infer the three-dimensional spatial coordinates corresponding to each pixel in the image, converting the image coordinates of the two-dimensional depth map into the three-dimensional point cloud coordinates corresponding to each pixel, to achieve a three-dimensional reconstructed image of the target scene:

[0073] X w =Z·(u-u0)d x / f

[0074] Y w =Z·(v-v0)d y / f

[0075] Z w =Z

[0076] In the formula, (X w ,Y w ,Z w ) is the world coordinate, (u0,v0) is the coordinate of the pixel point in the center of the image, (u,v) is the coordinate of the pixel point in the image, Z is the image depth value to be obtained, f is the focal length of the camera, dx and d y is the camera parameter.

[0077] However, it should be understood that the image 3D visualization process in step S5 can be implemented by any other means besides the above-mentioned point cloud 3D display, including voxelization, normal vector mapping, virtual viewpoint generation, etc. The specific selection of these methods depends on the application scenario and requirements.

[0078] As an exemplary embodiment of the present invention, a light field 3D imaging system may include:

[0079] A microlens array light field camera is used to collect light field images of target scenes;

[0080] A data processing module, used to input the collected light field image into the above-mentioned convolutional neural network model and output a two-dimensional depth map of the target scene;

[0081] The three-dimensional reconstruction module is used to transform the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

[0082] The light field three-dimensional imaging system provided by the present invention can implement the technical solution described in the above method embodiment. The implementation principle of each module in the system can refer to the corresponding content in the above method embodiment.

[0083] As described above, the present invention uses a microlens array light field camera to collect multi-view light field images, uses a convolutional neural network model to extract features from all the information of the light field image and retain features for key information, then calculates the depth value to output a two-dimensional depth map, and then performs coordinate transformation to obtain a three-dimensional reconstructed image. Therefore, the present invention adopts multi-view information fusion to improve information acquisition, and also realizes end-to-end learning through the network to directly generate a depth map for the collected image, avoiding the need to design complex feature extraction and matching algorithms for different types of scenes. It has strong robustness in complex scenes, can reliably restore the real three-dimensional structure of various scenes, and produce accurate three-dimensional reconstruction results. In addition, the network of the present invention can utilize all the information of the light field image and dynamically select and focus on key information in the input data, thereby effectively simplifying the computational complexity of the network, improving the performance of the neural network, retaining sub-aperture light field images with higher information content for depth estimation, and enhancing the robustness of the overall algorithm.

[0084] The features mentioned and / or shown in the above description of the exemplary embodiments of the present invention may be combined in the same or similar manner into one or more other embodiments, combined with the features in other embodiments or substituted for the corresponding features in other implementations. The technical solutions obtained by these combinations or substitutions shall also be deemed to be included in the protection scope of the present invention.

Claims

1. A light field three-dimensional imaging method based on a neural network, characterized in that: The following steps are involved: Step 1, using a microlens array light field camera to collect multi-view light field images to form a multi-view data set; Step 2, constructing a convolutional neural network model, the neural network model is used to input a light field image and output its two-dimensional depth map, and includes a convolution layer, a residual module, an SPP module and an attention mechanism module in sequence, which is used to extract features from all the information of the input light field image and retain features of key information in the light field image, the convolution layer and the residual module are used to extract local texture and semantics in the light field image, the SPP module is used to guide the network to learn the contextual semantic information of the target object in the scene more deeply, and evolve the target features into complex features with global semantic information of the scene, and the attention mechanism module is used to select different aperture perspectives to retain features of overlapping areas in images of different perspectives; Step 3, using the formed multi-view data set to train and test the constructed neural network model; Step 4, inputting the light field image of the target scene into the trained neural network model to obtain a two-dimensional depth map of the target scene; Step 5: transform the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

2. The method for light field three-dimensional imaging based on a neural network according to claim 1, characterized in that: The step 3 comprises the following steps: according to the image depth estimation value y and the image depth actual value y * The difference determines the loss function: <h2 style=";text-align:left;direction:ltr">L(y<h2 style=";text-align:left;direction:ltr"> * <h2 style=";text-align:left;direction:ltr"> ,y)=λL1+L2 In the formula, the loss value L1 of the depth information is: The structural similarity loss function L2 is: In the formula, λ is the loss value weight of the depth information, N is the number of image pixels, and SSIM is the structural similarity function.

3. The neural network-based light field three-dimensional imaging method according to claim 1 or 2, characterized in that: The step 3 includes the following sub-steps: Step 3.1, dividing the formed multi-view dataset into a training set and a test set; Step 3.2, training the constructed neural network model using the divided training set; Step 3.3, use the divided test set to verify the trained neural network model.

4. The neural network-based light field three-dimensional imaging method according to claim 1 or 2, characterized in that: The step 5 includes the following sub-steps: using the two-dimensional depth map of the target scene to infer the three-dimensional spatial coordinates corresponding to each pixel in the image, converting the image coordinates of the two-dimensional depth map into the three-dimensional point cloud coordinates corresponding to each pixel, so as to realize the three-dimensional reconstruction image of the target scene: X w =Z·(u-u0)d x / f Y w =Z·(v-v0)d y / f WITH w =Z In the formula, (X w ,Y w ,Z w ) is the world coordinate, (u0,v0) is the coordinate of the pixel point in the center of the image, (u,v) is the coordinate of the pixel point in the image, Z is the image depth value to be obtained, f is the focal length of the camera, d x and d y is the camera parameter.

5. A light field three-dimensional imaging system, characterized in that: include: A microlens array light field camera is used to collect light field images of target scenes; A data processing module, configured to input the acquired light field image into the convolutional neural network model of claim 1 and output a two-dimensional depth map of the target scene; The three-dimensional reconstruction module is used to transform the image coordinates of the two-dimensional depth map of the target scene to obtain a three-dimensional reconstructed image of the target scene.

Images