Light field three-dimensional reconstruction method, device, electronic device and computer storage medium

Abstract

The present invention relates to a light field 3D reconstruction method, device, electronic device, and computer storage medium. The method comprises: collecting light field data of a target object using a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image; inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image; calculating disparity information of the target object based on the phase information of the light field image, and determining the depth information of the target object based on the disparity information and a predetermined mapping relationship between disparity and depth in a scene; and performing light field 3D reconstruction of the target object based on the super-resolution light field image and the depth information. The present invention can perform high-precision light field reconstruction of the target object based on the trained optical flow network model, thereby improving the efficiency and accuracy of light field 3D reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a light field three-dimensional reconstruction method, device, electronic equipment and computer storage medium. Background Art

[0002] With the continuous advancement of my country's intelligent manufacturing strategy, the precision requirements in the production and manufacturing field have also increased. Optical three-dimensional reconstruction, as a non-contact measurement method, has been widely used in reverse engineering, medical equipment, cultural relics and archaeology and other fields.

[0003] Different from traditional imaging, light field imaging is an imaging technology based on computational imaging theory and image rendering theory. In addition to the two-dimensional spatial resolution of traditional imaging, it can also simultaneously record the angular resolution of the scene. With a single shot, it can obtain images of the scene from different perspectives, enabling the acquisition of four-dimensional information about angles and space. Because of its ability to record four-dimensional information, it can use angle and spatial information to achieve rapid three-dimensional reconstruction of objects. As a new type of imaging acquisition device, light field cameras can simultaneously record the direction and intensity of light in a scene, allowing for multi-perspective views of the scene with a single shot, acquiring four-dimensional information about angles and space.

[0004] However, existing light field 3D reconstruction methods are mostly passive measurements, relying primarily on features such as the shape and texture of the object being measured, making it difficult to maintain accurate and stable reconstruction results. Furthermore, the resolution of the imaging sensor of a light field camera is fixed, and obtaining images from multiple perspectives will reduce the resolution of each sub-aperture image. Existing technologies typically upsample low-resolution images to the same size as the high-resolution image before inputting them into the network for calculation. However, this approach leads to the problem of performing convolution operations on large scales within the network, significantly increasing computational cost and time complexity. Therefore, how to improve the resolution of each sub-aperture image while maintaining multiple perspectives is an urgent problem that needs to be solved in high-precision light field 3D reconstruction. Summary of the Invention

[0005] In view of this, it is necessary to provide a light field 3D reconstruction method, device, electronic device and computer storage medium to achieve the purpose of improving the resolution of each sub-aperture image while maintaining multiple perspectives, so as to improve the accuracy of 3D reconstruction.

[0006] In order to solve the above problems, the present invention provides a light field 3D reconstruction method in a first aspect, comprising:

[0007] Collecting light field data of a target object based on a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image;

[0008] Inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image;

[0009] Calculating disparity information of the target object based on phase information of the light field image, and determining depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in the scene;

[0010] Perform light field three-dimensional reconstruction of the target object according to the super-resolution light field image and the depth information.

[0011] Furthermore, the pre-trained optical flow network model includes an optical flow network and a sub-pixel convolutional network;

[0012] The optical flow network is trained using a training set composed of high-resolution light field images, and the pixel mean square error of the reconstructed image is used as an objective function for training the optical flow network;

[0013] The composite loss function of the optical flow network and the sub-pixel convolutional network is determined as the loss function of the optical flow network model.

[0014] Further, the light field image includes a light field sub-aperture image;

[0015] Inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image includes:

[0016] Utilizing the optical flow network to calculate sub-pixel offsets between adjacent light field sub-aperture images, and obtaining offset-registered light field sub-aperture images;

[0017] The sub-pixel convolutional network is used to perform feature calculation on the offset-registered light field sub-aperture image to obtain the super-resolution light field image.

[0018] Furthermore, the method of calculating the sub-pixel offset between adjacent light field sub-aperture images using the optical flow network to obtain offset-registered light field sub-aperture images includes:

[0019] Calculating first features of the adjacent light field sub-aperture images using a first convolutional layer and a first upsampling layer to obtain an initial optical flow;

[0020] Performing a warping transformation on the initial optical flow and the light field sub-aperture image at the target position to obtain an initial deformed image of the light field sub-aperture image at the target position;

[0021] Utilizing a second convolutional layer and a second upsampling layer to calculate the initial deformed image, the initial optical flow, and the second features of the adjacent light field sub-aperture images, to obtain a refined optical flow;

[0022] The initial optical flow, the refined optical flow and the sub-aperture image of the target position are warped and offset to form a light field sub-aperture image, wherein the sub-aperture offset is determined by the initial optical flow, the refined optical flow and the sub-aperture image of the target position.

[0023] Furthermore, the sub-pixel convolutional network is used to perform feature calculation on the offset-registered light field sub-aperture image to obtain the super-resolution light field image, including:

[0024] Performing pixel recombining on the offset-registered light field sub-aperture image to obtain a recombined super-resolution light field image;

[0025] The pixel reorganization of the offset-registered light field sub-aperture image comprises:

[0026] The pixels of the offset-registered light field sub-aperture image are divided into a plurality of sub-pixel points, and the feature map is filled into the plurality of sub-pixel points based on the pixel position, thereby completing the pixel reorganization of the offset-registered light field sub-aperture image.

[0027] Furthermore, the mapping relationship between parallax and depth in the scene is determined, including:

[0028] Use the preset calibration method to perform parameter calibration on the light field camera;

[0029] Determine the mapping relationship between parallax and depth in the scene after parameterized calibration: Among them, z c represents the scene depth, Δx represents the scene parallax, u represents the object distance, v represents the image distance, q represents the size of each pixel on the imaging sensor, and b represents the distance between the microlens array and the imaging sensor.

[0030] Furthermore, before collecting light field data, phase encoding and phase solving are performed on the target object, and light field data of the target object after phase encoding and phase solving are collected;

[0031] The calculating the disparity information of the target object based on the phase information of the light field image includes:

[0032] The disparity information of the target object is determined according to the straight line features presented by the phase information in the polar plane image.

[0033] In a second aspect, the present invention further provides a light field 3D reconstruction device, comprising:

[0034] an acquisition module, configured to acquire light field data of a target object based on a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image;

[0035] An output module, configured to input the light field image into a pre-trained optical flow network model and output a super-resolution light field image;

[0036] a calculation module, configured to calculate disparity information of the target object based on phase information of the light field image, and determine depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in a scene;

[0037] A reconstruction module is used to perform light field three-dimensional reconstruction of the target object according to the super-resolution light field image and the depth information.

[0038] In a third aspect, the present invention also provides an electronic device comprising a memory and a processor, wherein the memory is used to store a program; and the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the above-mentioned light field three-dimensional reconstruction method.

[0039] In a fourth aspect, the present invention further provides a computer storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps in the above-mentioned light field three-dimensional reconstruction method.

[0040] The beneficial effects of the above-described embodiment are as follows: the present invention collects light field data of a target object using a light field camera, so that the collected data more accurately reflects the current scene information; by inputting the light field image into a pre-trained optical flow network model, the light field image can be offset and registered based on the motion offset between sub-apertures described in the optical flow to achieve reconstruction from low resolution to high resolution, thereby ensuring that the resolution of each sub-aperture image is improved based on multiple perspectives. After obtaining the high-resolution light field image, the object's precise parallax information is calculated based on the phase information in the light field image, and the object's depth information is then determined based on the calibration results. Finally, the high-resolution light field image is assembled to achieve three-dimensional light field reconstruction of the current scene, thereby improving the efficiency and accuracy of light field three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A model reference diagram of a light field camera provided by one embodiment of the present invention;

[0042] Figure 2 A light field biplane model reference diagram provided by one embodiment of the present invention;

[0043] Figure 3 A polar plane diagram provided by one embodiment of the present invention;

[0044] Figure 4 A schematic flow chart of an embodiment of a light field 3D reconstruction method provided by the present invention;

[0045] Figure 5 A schematic diagram of the principle of sub-pixel offset registration of adjacent sub-aperture images based on an optical flow network according to an embodiment of the present invention;

[0046] Figure 6 A diagram illustrating the operating principle of pixel recombining according to an embodiment of the present invention;

[0047] Figure 7 A comparison diagram of phase calculation details before and after super-resolution provided in one embodiment of the present invention;

[0048] Figure 8 A comparison diagram of disparity calculation details before and after super-resolution provided in one embodiment of the present invention;

[0049] Figure 9 A comparison diagram of 3D reconstruction results before and after super-resolution provided in one embodiment of the present invention;

[0050] Figure 10 A schematic structural diagram of an embodiment of a light field 3D reconstruction device provided by the present invention;

[0051] Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0053] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, "multiple" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined. Reference to "embodiments" in this article means that the specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] The present invention provides a method, apparatus, electronic device, and computer storage medium for light field 3D reconstruction. Because light field cameras can simultaneously record the direction and intensity of light in a scene, they can acquire angular and spatial four-dimensional information with a single shot. Therefore, light field cameras can be used for 3D reconstruction. However, the resolution of a light field camera's imaging sensor is limited, and obtaining images from multiple perspectives reduces the resolution of each sub-aperture image. Optical flow, a simple and practical way to express image motion, can be used to describe the motion offset between adjacent sub-apertures. Based on this, motion compensation can be performed to achieve image alignment, ensuring that subsequent input into a super-resolution reconstruction network is sufficient.

[0055] Before describing the embodiments, the following definitions are given for the relevant terms:

[0056] Light field imaging: Light field imaging is an imaging technology based on computational imaging theory and image rendering theory. In addition to the two-dimensional spatial resolution recorded by traditional imaging, it can also simultaneously record the angular resolution of the scene. Through a single shot, images of the scene observed from different perspectives can be obtained, thereby realizing the acquisition of four-dimensional information of angle and space.

[0057] Light field camera: A light field camera is a new type of imaging acquisition device that can simultaneously record the direction and intensity of light in a scene. The most popular light field camera is based on a microlens array, and its model is as follows: Figure 1 As shown, Figure 1 This is a reference diagram of a light field camera model provided by one embodiment of the present invention. Based on a conventional camera, a microlens array is added between the main lens and the imaging sensor. Light passes through the main lens and then through the microlenses to form a secondary image. Photosensitive elements at different locations on the image sensor can collect light information from the same object point, which is aggregated into "macropixels" on the sensor. See [Note: The original text appears to be corrupted and should be omitted.] Figure 2 , Figure 2 A light field biplane model reference diagram is provided in accordance with an embodiment of the present invention, namely, L(s, t, x, y), where (s, t) corresponds to the angular resolution and (x, y) corresponds to the spatial resolution.

[0058] Polar plane graph: Polar plane graph theory can solve the problem of 3D reconstruction of scenes in motion. It can be combined with polar plane graph theory to obtain parallax clues of the scene to achieve the purpose of 3D reconstruction. For four-dimensional light fields, there is sub-pixel motion between multi-subaperture images, which also means parallax information exists. Therefore, polar plane theory is also applicable to four-dimensional light fields. In particular, when any two coordinates in the four-dimensional space are fixed and the light is confined to the two-dimensional space, the light field can be expressed as: in represents a subaperture image at an angle, Represents the set of an object point from all perspectives. Represents the polar plane diagram in the horizontal direction, A polar plane diagram representing the vertical direction. See Figure 3 , Figure 3 A polar plane diagram is provided in one embodiment of the present invention. Figure 3 The image shows a 7×7×512×512 light field image and its horizontal and vertical epipolar plane images. In the light field, an epipolar plane image can be understood as a two-dimensional horizontal or vertical slice of four-dimensional data. The image shows the presence of many lines with different slopes. These lines of varying inclination are called epipolar lines. Epipolar lines are the motion trajectories of the same object point due to different positions in multiple apertures, and their inclination reflects the disparity information of the corresponding pixel in the scene. The epipolar plane image-based 3D light field reconstruction method transforms the disparity calculation into the problem of solving the epipolar slope. This effectively reduces the computational effort while maintaining high robustness, thus playing an important role in light field 3D reconstruction.

[0059] Passive measurement: Passive measurement generally refers to using the surrounding environment, such as the reflection of natural light, to obtain images using a camera, and then calculating the three-dimensional spatial information of the object through specific algorithms. The main methods include texture recovery shape method, shadow recovery shape method and stereo vision method.

[0060] Active measurement: Active measurement refers to the use of light sources or energy sources such as lasers, sound waves, and electromagnetic waves to transmit to the target object, and obtaining the depth information of the object by receiving the returned light waves.

[0061] The specific embodiments are described in detail below:

[0062] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of a light field 3D reconstruction method provided by the present invention. A specific embodiment of the present invention discloses a light field 3D reconstruction method, comprising:

[0063] Step S401: collecting light field data of a target object using a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image;

[0064] Step S402: inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image;

[0065] Step S403: calculating disparity information of the target object based on the phase information of the light field image, and determining depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in the scene;

[0066] Step S404: performing light field 3D reconstruction of the target object according to the super-resolution light field image and the depth information.

[0067] First of all, it should be noted that before the light field camera collects the light field data required for three-dimensional reconstruction, specifically the light field data of the target object, it is necessary to perform parameter calibration on the light field camera. It can be understood that the light field data collected by the calibrated light field camera can more accurately reflect the current scene information. It should also be noted that before collecting light field data, the target object needs to be encoded. Specifically, it can be achieved by combining light field technology with structured light stripe projection technology and using a digital projector to phase encode the object. The light field data collected in this way includes the phase information of the light field image. Among them, the light field data is four-dimensional information data including angle and space, and the light field image includes multiple sub-aperture images, which are images of the target object at different angles.

[0068] In a pre-trained optical flow network model, since optical flow can describe the motion offset between adjacent sub-apertures, motion compensation is used to achieve image alignment. Feature calculations are then performed on the low-resolution sub-aperture images and aligned to a high-resolution space, achieving super-resolution reconstruction of the light field image. Therefore, inputting a light field image into the pre-trained optical flow network model produces a reconstructed super-resolution light field image.

[0069] It is understood that after parameterizing a light field camera, the mapping relationship between disparity and depth in a scene can be determined based on the calibration parameters. After phase encoding of a target object, the phase value of each point in the scene is unique, and the phase value of the same target point, modulated by the object's shape, does not change with changes in the observation angle. Disparity information of the target object can be calculated based on the phase information of the light field image. Therefore, the depth information of the target object can be determined based on the calculated disparity information and the aforementioned mapping relationship, and then light field 3D reconstruction of the target object can be performed based on the super-resolution image and depth information.

[0070] The beneficial effects of the above-described embodiment are as follows: the present invention collects light field data of a target object using a light field camera, enabling the collected data to more accurately reflect current scene information; by inputting the light field image into a pre-trained optical flow network model, the light field image can be offset and registered based on the motion offset between sub-apertures described in the optical flow to achieve reconstruction from low resolution to high resolution, thereby ensuring that the resolution of each sub-aperture image is improved based on multiple perspectives. After obtaining the high-resolution light field image, the object's precise parallax information is calculated based on the phase information in the light field image, and the object's depth information is then determined based on the calibration results. Finally, combined with the high-resolution light field image, light field three-dimensional reconstruction of the target object is achieved, thereby improving the efficiency and accuracy of light field three-dimensional reconstruction.

[0071] In one embodiment of the present invention, the pre-trained optical flow network model includes an optical flow network and a sub-pixel convolutional network;

[0072] In the process of training the optical flow network model, high-resolution light field images are used to form a training set to train the optical flow network, and the pixel mean square error of the reconstructed image is used as the objective function for training the optical flow network;

[0073] The training set can be composed of Represented by n = 1...N, where n = 1...N represents the number of training sets. The objective function of training the optical flow network is represented by the pixel mean square error of the reconstructed image, specifically:

[0074] The composite loss function of the optical flow network and the sub-pixel convolutional network is determined as the loss function of the optical flow network model.

[0075] It can be understood that since the optical flow network model consists of an optical flow network and a sub-pixel convolutional network, the loss function of the entire network model can be a composite loss function of the optical flow network and the sub-pixel convolutional network:

[0076] By combining the optical flow network with the sub-pixel convolutional network for joint optimization, the composite loss can be minimized.

[0077] In one embodiment of the present invention, the light field image includes a light field sub-aperture image, and the light field image is input into a pre-trained optical flow network model to output a super-resolution light field image, including:

[0078] The optical flow network is used to calculate the sub-pixel offset between adjacent light field sub-aperture images to obtain the offset-registered light field sub-aperture images.

[0079] The sub-pixel convolutional network is used to calculate the features of the offset-registered light field sub-aperture image to obtain a super-resolution light field image.

[0080] It is understandable that due to the narrow baseline characteristics of the light field camera, there is a high degree of information correlation between adjacent sub-aperture images. Based on this characteristic, the information of multiple adjacent sub-aperture images can be used to reconstruct the image of the intermediate aperture, thereby obtaining more image details. The optical flow method describes the motion offset between adjacent sub-apertures. Therefore, the optimal optical flow representation between adjacent sub-aperture images and the sub-aperture image at the target position can be found to realize the offset calculation between light field sub-aperture images. The light field sub-aperture image after offset registration is then passed into the sub-pixel convolutional network to map the feature map of the low-resolution image to the high-resolution grid, thereby realizing the improvement of the light field image from low resolution to high resolution.

[0081] In one embodiment of the present invention, calculating the sub-pixel offset between adjacent light field sub-aperture images using an optical flow network to obtain offset-registered light field sub-aperture images includes:

[0082] The first convolutional layer and the first upsampling layer are used to calculate the first features of the adjacent light field sub-aperture images to obtain the initial optical flow;

[0083] The initial optical flow and the light field sub-aperture image at the target position are warped to obtain an initial deformed image of the light field sub-aperture image at the target position;

[0084] The second convolutional layer and the second upsampling layer are used to calculate the second features of the initial deformed image, the initial optical flow, and the adjacent light field sub-aperture images to obtain the refined optical flow;

[0085] The initial optical flow, refined optical flow and sub-aperture image of the target position are warped and offset to obtain the registered light field sub-aperture image using the sub-aperture offset in the activation layer, wherein the sub-aperture offset is determined by the initial optical flow, refined optical flow and sub-aperture image of the target position.

[0086] First of all, it should be noted that when performing the offset calculation of the light field sub-aperture image, the input adjacent light field sub-aperture images can be defined as a spatiotemporal data block, that is, a stack of multiple adjacent sub-aperture images. This can be achieved by introducing the time depth D l To represent it. Use the input depth D0 to represent the input number of adjacent low-resolution sub-aperture images with an odd number. If the radius of an input spatiotemporal data block is defined as R = (D0-1) / 2, then a set of input images centered at position p can be expressed as: Where V represents a spatiotemporal data block, so the super-resolution process can be expressed as: Here, τ is a model parameter, and f(·) represents the mapping function from low resolution to high resolution. To calculate the offset between light field sub-aperture images, optical flow is used to describe the motion offset between adjacent sub-apertures. Based on this, motion compensation is performed to achieve image alignment, ensuring the subsequent input into the super-resolution reconstruction network, namely the sub-pixel convolutional network.

[0087] Taking two adjacent sub-aperture images as an example, we find the best optical flow representation between the adjacent sub-aperture images and the sub-aperture image at the target position, which is offset registration. Assuming that the optical flow is pixel-level dense, each pixel can be moved to a new position through transformation. In this case, the optical flow can be expressed as: Δ p+1 =(Δ p+1 x,Δ p+1 y), where Δ p+1 x and Δ p+1 y represents the components of the optical flow in the x and y directions respectively. Therefore, the image after optical flow compensation can be expressed as: I'p+1 (x,y)=K{I p+1 (x+Δ p+1 x,y+Δ p+1 y)}.

[0088] See also Figure 5 , Figure 5 This is a schematic diagram of the principle of sub-pixel offset registration of adjacent sub-aperture images based on an optical flow network provided by one embodiment of the present invention. Two adjacent sub-aperture images are stacked into a three-dimensional data block as the input of the initial optical flow calculation module network. In the initial optical flow calculation process, a convolution layer, namely the first convolution layer, is used to perform feature calculations. Finally, a ×4 upsampling layer, namely the first upsampling layer, is used to restore the scale of the calculated results, thereby obtaining the initial optical flow of the two adjacent sub-aperture images.

[0089] Then the initial optical flow and the sub-aperture image I at the target position p+1 Perform warping transformation to obtain the initial deformed image of the sub-aperture image at the target position Then use the fine optical flow calculation module to calculate the initial deformed image Initial optical flow and two original sub-aperture images I p , I p+1 For processing, a convolution layer with a step size of s=2 is used in the computing network, that is, the second convolution layer performs downscaling operation on the data and finally uses a ×2 upsampling layer, that is, the second upsampling layer performs scale recovery. The refined optical flow is calculated based on the initial optical flow.

[0090] Therefore, the final subaperture offset is In the output part, a tanh activation layer is used to represent the pixel displacement in the normalized space. Finally, the initial optical flow Fine optical flow and the sub-aperture image I of the target position p+1 Combined together to perform warping transformation to obtain the image I' after optical flow compensation alignment p+1 , that is, the offset-registered light field sub-aperture image.

[0091] It should be noted that in order to train the optical flow network for offset registration calculation, the parameter Δ p+1 Optimize so that the mean square error between the transformed coordinate system and the reference coordinate system is minimized, which is conducive to constraining the optical flow to present a stable state in space. Therefore, the Huber loss of the optical flow gradient is used for penalty constraint: In actual calculations, the Huber loss is usually approximated as follows:

[0092] Here, ε = 0.01. The Huber loss function makes the values ​​have smooth and stable L2 characteristics near the origin and good robustness in the area far from the origin, making the entire calculation process more reasonable.

[0093] In one embodiment of the present invention, a sub-pixel convolutional network is used to perform feature calculation on the offset-registered light field sub-aperture image to obtain a super-resolution light field image, including:

[0094] Perform pixel reassembly on the offset-registered light field sub-aperture image to obtain a reassembled super-resolution light field image;

[0095] Repixel-reconstruction of offset-registered light-field subaperture images, including:

[0096] The pixels of the offset-registered light field sub-aperture image are divided into multiple sub-pixel points, and the feature map is filled into the multiple sub-pixel points based on the pixel position to complete the pixel reconstruction of the offset-registered light field sub-aperture image.

[0097] In the optical flow network model, for a network model consisting of L layers, the network layer in the first L-1 layer structure can be expressed as:

[0098] f 1 (I LR ;W1,b1)=φ(W1*I LR +b1), f l (I LR ;W 1:l ,b 1:l )=φ(W l *f l-1 (I LR )+b l ),l∈(1,L-1), where W l and b l Represent the weight and bias in the network learning process, W l The convolution tensor size is n l-1 ×n l ×k l ×k l , n l is the number of feature images in layer l, n0=C,k l is the size of the convolution kernel in layer l; the deviation b l n l The length vector of φ is φ; φ is the activation function. In the last layer of the network, sub-pixel convolution is used to map the feature image of the low-resolution image to a high-resolution image.

[0099] In the sub-pixel convolution layer, the filter on the convolution layer is periodically activated according to the different sub-pixel positions to periodically map the low-resolution feature image to the high-resolution image space. The process is as follows: SR =f L (I LR )=PS(W L *f L-1 (I LR )+b L ), where PS(·) represents a periodic pixel re-arrangement operator that can transform a pixel with a size of H×W×C·r 2 The elements are reorganized into a structure of rH×rW×C. Figure 6 , Figure 6 This is a diagram illustrating the operating principle of pixel reorganization according to an embodiment of the present invention.

[0100] It is understandable that the role of pixel reorganization is to transform r 2 The feature images of the channels are recombined into a super-resolution result of rH×rW. By pixel reorganization, the pixels of the low-resolution image are divided into r 2 A smaller pixel point, that is, a sub-pixel; then r 2 The feature maps are arranged according to the pixel positions and filled into the sub-pixels, and so on through the entire data to achieve H×W×C·r 2 The upsampling process is from rH × rW × C. This method can effectively avoid the artificial trace problems in the interpolation and deconvolution processes.

[0101] The light field super-resolution reconstruction method proposed in the present invention stacks the aligned sub-aperture images into three-dimensional data blocks, performs feature calculations directly on the low-resolution image size, and then rearranges the obtained feature images to the high-resolution image size. The image upsampling process is implicitly included in the convolution calculation, so that the convolution operations in the network are all implemented on a smaller image size, ensuring the accuracy of the calculation, effectively improving the computational efficiency of the network, and obtaining better super-resolution results.

[0102] In one embodiment of the present invention, determining a mapping relationship between disparity and depth in a scene includes:

[0103] Use the preset calibration method to perform parameter calibration on the light field camera;

[0104] Determine the mapping relationship between parallax and depth in the scene after parameterized calibration: Among them, z c represents the scene depth, Δx represents the scene parallax, u represents the object distance, v represents the image distance, q represents the size of each pixel on the imaging sensor, and b represents the distance between the microlens array and the imaging sensor.

[0105] It can be understood that the mapping relationship between the parallax and depth in the scene is obtained by parametrically calibrating the light field camera. The preset calibration method can be the Zhang Zhengyou calibration method. The mapping relationship between the parallax and depth in the scene is derived based on the Gaussian imaging formula, similar triangles and other geometric knowledge and the light field imaging model. Based on the Zhang Zhengyou calibration method, the light field camera is used to collect the checkerboard, and the above mapping relationship is fitted through the checkerboard corner point information.

[0106] In one embodiment of the present invention, phase encoding and phase resolution are performed on the target object before light field data acquisition, and light field data of the target object after phase encoding and phase resolution is acquired;

[0107] Calculate the disparity information of the target object based on the phase information of the light field image, including:

[0108] The disparity information of the target object is determined based on the straight line features presented by the phase information in the polar plane image.

[0109] Among them, a digital projector can be used to project a stripe pattern to phase encode the target object. It can be understood that according to the principle of phase measurement profilometry, after the phase encoding and solution of the measured object by the phase shift method and the multi-frequency heterodyne method, the phase value of each point in the scene is unique, and the phase value of the same target point after being modulated by the object shape does not change with the change of the observation angle. Therefore, after projecting the sinusoidal stripes onto the surface of the object and using a light field camera to capture the phase-encoded object, the phase information of the same object point will show the characteristics of a straight line in the light field polar plane diagram, where the slope of the straight line corresponds to the parallax information of the point.

[0110] Taking the horizontal direction as an example, select the pixel point (s) in the middle row of the structured light field plane (s, x) * ,x * ) as the target point, search for the point with the highest phase similarity to the target point in other rows, record the position indexes of these points and fit them into a straight line. Since the phase information in the scene is unique and certain after phase solution, the straight line fitted according to the phase similarity principle corresponds to the position of the same object point in the scene, and the slope of the line reflects the disparity information of the target point. Repeat this method to traverse the entire light field data to achieve accurate disparity calculation based on phase information. The calculation process can be expressed as: Finally, the obtained disparity information is combined with the mapping relationship obtained in the above calibration step to obtain the depth information of the object. Finally, the light field of the target object is reconstructed based on the depth information and the super-resolution light field image.

[0111] The present invention inputs phase-encoded light field images into a trained optical flow network, obtains high-resolution light field sub-aperture images based on the high information redundancy between adjacent sub-aperture images, and then realizes high-precision light field 3D reconstruction based on the phase information.

[0112] In order to better verify the practicality of this invention, please refer to Figure 7 , Figure 7 A comparison diagram of phase calculation details before and after super resolution provided in one embodiment of the present invention. Figure 7 (a) and (c) show the phase principal value information after phase calculation of the image collected by the light field camera before super-resolution. It can be seen that the phase principal value will have discontinuous "stuttering" phenomenon at the position of periodic transformation. This is because the insufficient resolution leads to the introduction of environmental noise in the image acquisition. This noise interferes with the phase calculation process and causes discontinuity. Figure 7 (b) and (d) show the phase principal value information after super-resolution. After achieving light field super-resolution by utilizing the high correlation between the information of the sub-aperture images, the information contained in the light field sub-aperture images is enhanced, and the influence of the noise in the image on the phase calculation is effectively reduced. As can be seen from the figure, the quality of the phase principal value after super-resolution has been significantly improved. Figure 7 The discontinuity phenomenon caused by the phase period transformation in (a) and (c) is improved, laying a solid foundation for subsequent disparity calculation.

[0113] See also Figure 8 , Figure 8 A comparison diagram of the disparity calculation details before and after super-resolution provided in one embodiment of the present invention. Figure 8 (a) and (c) show the disparity maps before super-resolution. Due to the presence of discontinuous and sudden abnormal points in the phase information before super-resolution, the error inheritance phenomenon occurs in the disparity calculation process based on this. Figure 8 The detailed enlarged images in (a) and (c) show that there are abnormal points and mixed areas in the disparity graph. When using the disparity graph with noise for 3D reconstruction, the accuracy will be affected. Figure 8 (b) and (d) show the disparity maps after super-resolution. The disparity calculation is performed based on the phase calculation after super-resolution. It can be seen from the figure that the disparity map after super-resolution reconstruction has more complete and accurate disparity information than that before super-resolution. At the same time, it also better solves the confusion phenomenon caused by insufficient resolution before super-resolution, effectively improving the accuracy of light field 3D reconstruction.

[0114] See also Figure 9 , Figure 9 This is a comparison diagram of the 3D reconstruction results before and after super-resolution provided in one embodiment of the present invention. Figure 9(a) and (c) show the 3D reconstruction results of two groups of objects before super-resolution. Figure 9 (b) and (d) show the super-resolution 3D reconstruction results of two groups of objects. The accuracy of the light field 3D reconstruction results after super-resolution is better than that before super-resolution. This is because after super-resolution, the phase calculation and disparity calculation processes in 3D reconstruction are significantly optimized compared to before super-resolution. This solves the error accumulation problem that existed before super-resolution, further reduces the impact of factors such as noise on the reconstruction results, and effectively improves the accuracy of light field 3D reconstruction.

[0115] In order to better implement the light field 3D reconstruction method in the embodiment of the present invention, based on the light field 3D reconstruction method, correspondingly, please refer to Figure 10 , Figure 10 This is a schematic structural diagram of an embodiment of a light field 3D reconstruction device provided by the present invention. This embodiment of the present invention provides a light field 3D reconstruction device 1000, comprising:

[0116] An acquisition module 1001 is configured to acquire light field data of a target object based on a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image;

[0117] Output module 1002, used to input the light field image into a pre-trained optical flow network model and output a super-resolution light field image;

[0118] A calculation module 1003 is configured to calculate disparity information of the target object based on the phase information of the light field image, and determine depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in the scene;

[0119] The reconstruction module 1004 is configured to perform light field 3D reconstruction of the target object based on the super-resolution light field image and the depth information.

[0120] It should be noted here that the device 1000 provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0121] Based on the above-mentioned light field 3D reconstruction method, an embodiment of the present invention further provides an electronic device, including: a processor and a memory, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the light field 3D reconstruction method in the above-mentioned embodiments are implemented.

[0122] Figure 111 shows a schematic structural diagram of an electronic device 1100 suitable for implementing an embodiment of the present invention. The electronic devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0123] The electronic device includes: a memory and a processor, wherein the processor here may be referred to as the processing device 1101 below, and the memory may include at least one of the read-only memory (ROM) 1102, the random access memory (RAM) 1103, and the storage device 1108 below, as shown below:

[0124] like Figure 11 As shown, the electronic device 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the electronic device 1100 are also stored in the RAM 1103. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0125] Typically, the following devices may be connected to the I / O interface 1105: an input device 1106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1109. The communication device 1109 may allow the electronic device 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 11 The electronic device 1100 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 1109, or installed from the storage device 1108, or installed from the ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0127] Based on the above-mentioned light field 3D reconstruction method, an embodiment of the present invention further provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the light field 3D reconstruction method in the above-mentioned embodiments.

[0128] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0129] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A light field 3D reconstruction method, characterized in that: include: Collecting light field data of a target object based on a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image; The light field image is input into a pre-trained optical flow network model to output a super-resolution light field image, wherein the optical flow network model includes an optical flow network and a sub-pixel convolutional network; the optical flow network is trained using a training set composed of high-resolution light field images, and the pixel mean square error of the reconstructed image is used as the objective function for training the optical flow network; and a composite loss function of the optical flow network and the sub-pixel convolutional network is determined as the loss function of the optical flow network model; Calculating disparity information of the target object based on phase information of the light field image, and determining depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in the scene; Performing light field 3D reconstruction of the target object according to the super-resolution light field image and the depth information; The light field image includes a light field sub-aperture image; Inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image includes: The first convolutional layer and the first upsampling layer are used to calculate the first features of the adjacent light field sub-aperture images to obtain the initial optical flow; Performing a warping transformation on the initial optical flow and the light field sub-aperture image at the target position to obtain an initial deformed image of the light field sub-aperture image at the target position; Utilizing a second convolutional layer and a second upsampling layer to calculate the initial deformed image, the initial optical flow, and the second features of the adjacent light field sub-aperture images, to obtain a refined optical flow; warping the initial optical flow, the refined optical flow, and the sub-aperture image at the target position using a sub-aperture offset in the activation layer to obtain a light field sub-aperture image that is aligned, wherein the sub-aperture offset is determined by the initial optical flow, the refined optical flow, and the sub-aperture image at the target position; The sub-pixel convolutional network is used to perform feature calculation on the offset-registered light field sub-aperture image to obtain the super-resolution light field image.

2. The light field 3D reconstruction method according to claim 1, wherein: Using the sub-pixel convolutional network to perform feature calculation on the offset-registered light field sub-aperture image to obtain the super-resolution light field image, comprising: Performing pixel recombining on the offset-registered light field sub-aperture image to obtain a recombined super-resolution light field image; The pixel reorganization of the offset-registered light field sub-aperture image comprises: The pixels of the offset-registered light field sub-aperture image are divided into a plurality of sub-pixel points, and the feature map is filled into the plurality of sub-pixel points based on the pixel position, thereby completing the pixel reorganization of the offset-registered light field sub-aperture image.

3. The light field 3D reconstruction method according to claim 1, wherein: Determine the mapping relationship between parallax and depth in the scene, including: Use the preset calibration method to perform parameter calibration on the light field camera; Determine the mapping relationship between parallax and depth in the scene after parameterized calibration: ,in, Indicates the scene depth, represents the scene parallax, Indicates object distance, represents the image distance, represents the size of each pixel on the imaging sensor, Represents the distance between the microlens array and the imaging sensor.

4. The light field 3D reconstruction method according to claim 1, wherein: Performing phase encoding and phase resolution on the target object before light field data acquisition, and acquiring light field data of the target object after phase encoding and phase resolution; The calculating the disparity information of the target object based on the phase information of the light field image includes: The disparity information of the target object is determined according to the straight line features presented by the phase information in the polar plane image.

5. A light field 3D reconstruction device, characterized in that: include: an acquisition module, configured to acquire light field data of a target object based on a light field camera, wherein the light field data includes a light field image of the target object and phase information of the light field image; An output module is configured to input the light field image into a pre-trained optical flow network model and output a super-resolution light field image, wherein the optical flow network model includes an optical flow network and a sub-pixel convolutional network; the optical flow network is trained using a training set composed of high-resolution light field images, and the pixel mean square error of the reconstructed image is used as an objective function for training the optical flow network; and a composite loss function of the optical flow network and the sub-pixel convolutional network is determined as the loss function of the optical flow network model; a calculation module, configured to calculate disparity information of the target object based on phase information of the light field image, and determine depth information of the target object according to the disparity information and a predetermined mapping relationship between disparity and depth in a scene; A reconstruction module, configured to perform light field three-dimensional reconstruction of the target object based on the super-resolution light field image and the depth information; The light field image includes a light field sub-aperture image; Inputting the light field image into a pre-trained optical flow network model to output a super-resolution light field image includes: The first convolutional layer and the first upsampling layer are used to calculate the first features of the adjacent light field sub-aperture images to obtain the initial optical flow; Performing a warping transformation on the initial optical flow and the light field sub-aperture image at the target position to obtain an initial deformed image of the light field sub-aperture image at the target position; Utilizing a second convolutional layer and a second upsampling layer to calculate the initial deformed image, the initial optical flow, and the second features of the adjacent light field sub-aperture images, to obtain a refined optical flow; warping the initial optical flow, the refined optical flow, and the sub-aperture image at the target position using a sub-aperture offset in the activation layer to obtain a light field sub-aperture image that is aligned, wherein the sub-aperture offset is determined by the initial optical flow, the refined optical flow, and the sub-aperture image at the target position; The sub-pixel convolutional network is used to perform feature calculation on the offset-registered light field sub-aperture image to obtain the super-resolution light field image.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a program; and the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the light field three-dimensional reconstruction method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the light field three-dimensional reconstruction method according to any one of claims 1 to 4.

Images