Method and device for generating virtual viewpoint image

Abstract

The present invention provides a method and device for generating a virtual viewpoint image, the method comprising: extracting a three-dimensional model of a current scene based on camera parameters of an input viewpoint and a first image; obtaining a first target image of a target virtual viewpoint based on the three-dimensional model and camera parameters of a target virtual viewpoint; inputting a second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint. The method and device for generating a virtual viewpoint image provided by the present invention can improve the accuracy and efficiency of 3D light field information reconstruction by restoring geometric information of a corresponding three-dimensional scene through a first target image and restoring illumination and color information of a corresponding three-dimensional scene through a second target image, thereby obtaining a target virtual viewpoint image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating a virtual viewpoint image. Background Art

[0002] Traditional two-dimensional image acquisition and display technology is difficult to meet the growing viewing needs of viewers. With the innovation of display technology and the improvement of computing power, methods for accurately and efficiently reproducing three-dimensional light field information have received a lot of attention. In order to reproduce the light field information of real scenes in a true three-dimensional light field display, it is necessary to build a camera array to capture the real scene and restore the three-dimensional light field information through dense viewpoints. In this process, due to the physical limitations of the camera, it is difficult to perform dense viewpoint acquisition, and only sparse viewpoint acquisition can be satisfied. In order to generate dense viewpoint content, researchers have proposed many virtual viewpoint generation methods that use sparse viewpoints as input to generate dense viewpoints.

[0003] One is based on the Multi-Plane Image (MPI) method, which includes RGB-alpha images of multiple planes, each of which expresses the content of the scene at a certain depth. However, the depth of each plane is fixed and discrete, which works better when the input viewpoints are arranged horizontally. When the input viewpoint positions are rotated, the expression ability of the three-dimensional space is limited, resulting in poor accuracy of the reconstructed 3D light field information. The other is a method based on 3D Mesh Scaffold, which requires the pre-input of a large number of scene images (more than 50 images) and is extremely time-consuming (on the order of hours), resulting in low efficiency of the reconstructed 3D light field information. It can be seen that in the process of reconstructing 3D light field information in the prior art, it is impossible to take into account both accuracy and efficiency at the same time. Summary of the invention

[0004] The present invention provides a method and device for generating a virtual viewpoint image, so as to solve the defect that the accuracy and efficiency of generating a virtual viewpoint image cannot be taken into account at the same time in the prior art, and realize setting fewer input viewpoints to obtain images of free virtual viewpoints in a scene.

[0005] The present invention provides a method for generating a virtual viewpoint image, comprising:

[0006] Extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image;

[0007] Based on the three-dimensional model and the camera parameters of the target virtual viewpoint, acquiring a first target image of the target virtual viewpoint;

[0008] Inputting the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint;

[0009] In which, the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene collected under the input viewpoint, the second image is a color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

[0010] According to a method for generating a virtual viewpoint image provided by the present invention, the second image and the first target image are input into an image processing model to obtain a second target image of the target virtual viewpoint, comprising:

[0011] Inputting the second image into a depth coding layer of the image processing model to obtain a first coded image corresponding to the second image;

[0012] Inputting the first encoded image and the first target image into a feature fusion layer of the image processing model to obtain a second encoded image;

[0013] The second encoded image is input into the depth decoding layer of the image processing model to obtain a second target image of the target virtual viewpoint.

[0014] According to a method for generating a virtual viewpoint image provided by the present invention, the step of inputting the second image to a depth coding layer of the image processing model to obtain a first coded image corresponding to the second image includes:

[0015] Based on the second image, feature extraction is performed to obtain a target depth feature map;

[0016] Based on the target depth feature map, feature fusion is performed to obtain the first encoded image;

[0017] Wherein, the target depth feature map includes at least two depth feature maps with different resolutions.

[0018] According to a method for generating a virtual viewpoint image provided by the present invention, the first coded image and the first target image are input into a feature fusion layer of the image processing model to obtain a second coded image, comprising:

[0019] Based on the first coded image and the first target image, using the DIBR algorithm, the coordinates are projected to the coordinate system of the target virtual viewpoint to obtain a target projection image;

[0020] Image fusion is performed based on the target projection image to obtain the second encoded image.

[0021] According to a method for generating a virtual viewpoint image provided by the present invention, before the camera parameters based on the input viewpoint and the first image, the method further comprises: based on the input viewpoint, using a camera calibration method to obtain the camera parameters of the input viewpoint;

[0022] The camera parameters include camera internal parameters and camera external parameters.

[0023] According to a method for generating a virtual viewpoint image provided by the present invention, the three-dimensional model of the current scene is extracted based on the camera parameters of the input viewpoint and the first image, comprising:

[0024] Based on the camera parameters of the input viewpoint and the first image, a TSDF algorithm is used to obtain depth fusion information;

[0025] Based on the depth fusion information, a Marching Cube algorithm is used to obtain a three-dimensional model of the current scene.

[0026] The present invention also provides a device for generating a virtual viewpoint image, comprising:

[0027] A three-dimensional model acquisition module, used to extract a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image;

[0028] A first acquisition module, configured to acquire a first target image of the target virtual viewpoint based on the three-dimensional model and camera parameters of the target virtual viewpoint;

[0029] A second acquisition module, used for inputting a second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint;

[0030] In which, the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene collected under the input viewpoint, the second image is a color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for generating a virtual viewpoint image are implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a virtual viewpoint image are implemented.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a virtual viewpoint image are implemented.

[0034] The method and device for generating a virtual viewpoint image provided by the present invention obtains a three-dimensional model of a scene based on the camera parameters of an input viewpoint and a first image through the embodiment of the present invention, obtains a first target image through the camera parameters of a target virtual viewpoint, obtains a second target image through an image processing model inputted by a second image, restores geometric information of a corresponding three-dimensional scene through the first target image, and restores illumination and color information of a corresponding three-dimensional scene through the second target image, and obtains a target virtual viewpoint image. The accuracy and efficiency of 3D light field information reconstruction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flowchart of a method for generating a virtual viewpoint image provided by the present invention;

[0037] Figure 2 This is one of the sub-flow diagrams of the method for generating a virtual viewpoint image provided by the present invention;

[0038] Figure 3 This is the second sub-flow diagram of the method for generating a virtual viewpoint image provided by the present invention;

[0039] Figure 4 It is a structural schematic diagram of a device for generating a virtual viewpoint image provided by the present invention;

[0040] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0043] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0044] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0045] Figure 1 FIG. 1 is a flow chart of a method for generating a virtual viewpoint image provided by the present invention. Figure 1 As shown, the method for generating a virtual viewpoint image provided by an embodiment of the present invention includes: step 101, extracting a three-dimensional model of the current scene based on camera parameters of an input viewpoint and a first image.

[0046] The first image is a depth map of the current scene captured under an input viewpoint, and the number of the input viewpoints is one or more.

[0047] It should be noted that the execution subject of the method for generating a virtual viewpoint image provided by the embodiment of the present invention is a device for generating a virtual viewpoint image.

[0048] The input viewpoint refers to the location of the camera module's shooting point when shooting the current scene. The input viewpoints are small in number and irregularly distributed in the current scene.

[0049] The number of input viewpoints may be one or more, and the number of camera modules set in the current scene is equal to the number of input viewpoints.

[0050] The embodiment of the present invention does not specifically limit the camera module.

[0051] Optionally, the camera module may be an RGB-D camera.

[0052] The image captured by the RGB-D camera consists of a common RGB three-channel color image and a depth image, where the depth image is an image or image channel containing information about the distance of the surface of the scene object from the viewpoint. Each of its pixel values ​​is the actual distance from the sensor to the object. Usually the RGB image and the depth image are registered, and there is a one-to-one correspondence between the pixels.

[0053] Optionally, the camera module may also be a device having a color information sensor and a depth information sensor.

[0054] The camera parameters of the input viewpoint refer to the relevant parameters of the camera module at the input viewpoint. The camera parameters of the input viewpoint are used to uniquely distinguish the characteristics of the corresponding camera module in the current scene.

[0055] The embodiment of the present invention does not specifically limit the content of the camera parameters of the input viewpoint.

[0056] Exemplarily, the camera parameters of the input viewpoint may include parameters related to the characteristics of the camera module itself, such as the focal length and pixel size of the camera.

[0057] Exemplarily, the camera parameters of the input viewpoint may also include a horizontal position in the current scene.

[0058] Exemplarily, the camera parameters of the input viewpoint may also include a rotation position in the current scene.

[0059] The first image refers to a depth image among images obtained by photographing the current scene by the camera module at the position of the input viewpoint.

[0060] Exemplarily, the first image is a depth map in an image captured by an RGB-D camera at the position of the input viewpoint.

[0061] Specifically, in step 101, the device for generating a virtual viewpoint image obtains a first image taken by a corresponding camera module at an input viewpoint, fuses all the first images according to the posture information indicated by the camera parameters of the corresponding input viewpoints, and restores a three-dimensional model of the current scene.

[0062] The 3D model of the current scene refers to a data set including the 3D geometric information and structural information of the scene. Different types of 3D models can be obtained by using different processing methods. The embodiment of the present invention does not specifically limit this.

[0063] Exemplarily, the three-dimensional model of the current scene may be a voxel model.

[0064] A voxel or voxel, or volume pixel, is the smallest unit of digital data in a three-dimensional space. Voxels themselves do not contain data about their position in space (i.e. their coordinates), but their position relative to other voxels can be inferred.

[0065] Exemplarily, the three-dimensional model of the current scene may be a point cloud model.

[0066] Point cloud refers to the data obtained through a 3D scanner. The scanned data is recorded in the form of points, each of which contains three-dimensional coordinates, some of which may contain color information (R, G, B) or the intensity of the object's reflective surface.

[0067] Preferably, the three-dimensional model of the current scene may be a polygonal mesh model.

[0068] A polygon mesh is a collection of vertices and polygons that represent a polyhedral shape in three-dimensional computer graphics. These meshes are usually composed of triangles, quadrilaterals, or other simple convex polygons, which can simplify the rendering process.

[0069] Step 102: Acquire a first target image of the target virtual viewpoint based on the three-dimensional model and the camera parameters of the target virtual viewpoint.

[0070] It should be noted that the target virtual viewpoint refers to the free viewpoint in the current scene.

[0071] The embodiment of the present invention does not specifically limit the selection of free viewpoints.

[0072] Preferably, since the content observed by the target virtual viewpoint position should have a large overlap area with the image of the input viewpoint, the imaging quality of the viewpoint image can be improved to a certain extent. Therefore, the position of the target virtual viewpoint is closer to the input viewpoint position.

[0073] Correspondingly, the camera parameters of the target virtual viewpoint are close to the camera parameters of the adjacent input viewpoints, which is specifically manifested in that the parameters related to the characteristics of the camera module are consistent, and the position information of the target virtual viewpoint in the world coordinate system is close to the position information of the input viewpoint in the world coordinate system.

[0074] Specifically, in step 102, the device for generating a virtual viewpoint image sets a virtual camera module according to the camera parameters of the target virtual viewpoint, collects a depth map of the three-dimensional model of the current scene, and obtains a first target image of the target virtual viewpoint.

[0075] The first target image refers to a depth map collected by a correspondingly configured virtual camera module at a target virtual viewpoint.

[0076] Step 103: Input the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint.

[0077] The image processing model is obtained after training based on a real color image of a target virtual viewpoint as a sample, and the second image is a color image of the current scene collected under the input viewpoint.

[0078] It should be noted that the image processing model can be an artificial intelligence model, and the embodiment of the present invention does not specifically limit the model type.

[0079] Preferably, the image processing model is a neural network model, and the structure and parameters of the neural network include but are not limited to the number of input layers, hidden layers and output layers of the neural network, and the weight parameters of each layer, etc. The embodiment of the present invention does not specifically limit the type and structure of the neural network.

[0080] Preferably, the device for generating virtual viewpoint images utilizes an image processing model to project the color images of each input viewpoint to a target virtual viewpoint to generate a color image of the target virtual viewpoint, and the model is composed of an input layer, a hidden layer and an output layer.

[0081] The input layer is at the front end of the entire network and directly receives the second image and the first target image.

[0082] The hidden layer contains at least three layers, namely the depth encoding layer, feature fusion layer and depth decoding layer. Through its own neurons, the color image of each input viewpoint is used to obtain the corresponding depth map according to the spatial geometric relationship, and is projected to the coordinate system corresponding to the target virtual viewpoint according to the estimated depth map and the pose relationship between the input viewpoint and the target virtual viewpoint.

[0083] The output layer is the last layer and is used to output the second target image.

[0084] It should be noted that the sample data includes real color image data at each viewpoint position. The training set in the sample data at least includes real color image data at the target virtual viewpoint, while the test set does not include real color image data at the target virtual viewpoint. The embodiment of the present invention does not specifically limit the sample ratio of the training set and the test set.

[0085] The device for generating a virtual viewpoint image restores the image processing model in step 103 according to the weight coefficients of the input layer and the hidden layer, and inputs a group of test samples in the test set into the trained image processing model, so as to obtain the predicted value of the second target image corresponding to the test sample, which is used to verify the accuracy of the model.

[0086] The second image refers to a color image among images obtained by photographing the current scene by the camera module at the position of the input viewpoint.

[0087] Exemplarily, the second image is a color image in an image captured by an RGB-D camera at the position of the input viewpoint.

[0088] Specifically, in step 103, the device for generating the virtual viewpoint image initializes the weight coefficients between the layers of the constructed image processing model, and trains and tests the model in sequence according to the real color image of the target virtual viewpoint, so as to obtain the most accurate model parameters, namely the weight coefficients of the input layer and the hidden layer.

[0089] The device for generating virtual viewpoint images sets the image processing model according to the trained model parameters, and then processes each second image in turn through the model in combination with the first target image to obtain a second target image of the target virtual viewpoint.

[0090] The second target image refers to a color image captured by a correspondingly set virtual camera module at the target virtual viewpoint.

[0091] In the prior art, in the MPI method, the input viewpoints are set to be arranged horizontally to construct a three-dimensional scene. After the setting position of the input viewpoints is rotated, the accuracy of light field reconstruction is low. However, the embodiment of the present invention can break the limitation by setting the rotation position related parameters in the camera parameters of the input viewpoints, and improve the accuracy of light field reconstruction to a certain extent.

[0092] In the prior art, 3D Mesh Scaffold can effectively handle the problem that the input viewpoint position contains rotation, but this method needs to obtain scene images under many viewpoints for reconstruction, and the efficiency of light field reconstruction is low. However, by setting a small number of input viewpoints, the embodiment of the present invention can reconstruct the light field information of any viewpoint within the range formed by the small number of input viewpoints, thereby improving the efficiency of light field reconstruction to a certain extent.

[0093] The embodiment of the present invention obtains a three-dimensional model of a scene based on the camera parameters of an input viewpoint and a first image, obtains a first target image through the camera parameters of a target virtual viewpoint, obtains a second target image through an image processing model inputted by a second image, restores geometric information of a corresponding three-dimensional scene through the first target image, and restores illumination and color information of a corresponding three-dimensional scene through the second target image, and obtains a target virtual viewpoint image. The accuracy and efficiency of 3D light field information reconstruction can be improved.

[0094] Based on any of the above embodiments, the second image and the first target image are input into the image processing model to obtain the second target image of the target virtual viewpoint, including: inputting the second image into the depth coding layer of the image processing model to obtain a first encoded image corresponding to the second image.

[0095] It should be noted that the image processing model may include at least a depth coding layer, a feature fusion layer and a depth decoding layer.

[0096] Specifically, the depth coding layer of the device for generating the virtual viewpoint image receives the second images sent by the input layer, converts each second image into a first coded image, and uses the first coded image as the output of the depth coding layer.

[0097] The first encoded image refers to a feature map obtained by extracting features from the second image.

[0098] The embodiment of the present invention does not specifically limit the feature dimensions contained in the first encoded image.

[0099] Exemplarily, the first encoded image may include semantic features at a certain scale.

[0100] Exemplarily, the first encoded image may also include semantic features at certain scales.

[0101] The first encoded image and the first target image are input into the feature fusion layer of the image processing model to obtain a second encoded image.

[0102] Specifically, the feature fusion layer of the virtual viewpoint image generation device receives the first coded image sent by the depth coding layer and the first target image, projects each first coded image according to the coordinate system indicated by the first target image, performs image fusion on the projected images, generates a second coded image, and uses the second coded image as the output of the depth coding layer.

[0103] The second encoded image is input into the depth decoding layer of the image processing model to obtain a second target image of the target virtual viewpoint.

[0104] Specifically, the depth decoding layer of the device for generating the virtual viewpoint image receives the second encoded images sent by the feature fusion layer, converts each second encoded image into a second target image, and uses the second target image as the output of the depth decoding layer.

[0105] The embodiment of the present invention does not specifically limit the depth decoding layer.

[0106] Preferably, the second encoded image is decoded through multiple cascaded U-Net networks to generate a final 2D color image at the target viewpoint, namely, the second target image.

[0107] Correspondingly, in the training process of the image processing model, the Loss is calculated by comparing the generated second target image with the real 2D color image under the target viewpoint, the neural network is trained, and the weight parameters in the network are updated.

[0108] The embodiment of the present invention is based on allowing the second image and the first target image to enter the image processing model for optimization processing, by converting the second image into the first coded image, projecting and associating the first coded image according to the first target image, performing image fusion to obtain the second coded image, and then decoding the second coded image into the second target image. The logic and association between operations can be fully explored, the computational complexity and development difficulty are reduced, and the accuracy and efficiency of 3D light field information reconstruction are improved.

[0109] Based on any of the above embodiments, the second image is input into the depth coding layer of the image processing model to obtain a first coded image corresponding to the second image, including: performing feature extraction based on the second image to obtain a target depth feature map.

[0110] Among them, the target depth feature map includes at least two depth feature maps with different resolutions.

[0111] Specifically, after receiving the second image, the device for generating a virtual viewpoint image performs feature extraction processing in the depth coding layer of the image processing model according to different feature dimension requirements, and obtains a target depth feature map.

[0112] The embodiment of the present invention does not specifically limit the method and process of obtaining the target depth feature map.

[0113] Preferably, each second image is input into a VGG-16 pre-trained on ImageNet, and the deep feature maps output by the first three stages of the VGG-16 network are obtained as input for subsequent steps.

[0114] That is, the target depth feature map includes three depth feature maps with different resolutions outputted from the first three stages. The resolution of each depth feature map is equal to the resolution of the second image, half of the second image, and one quarter of the second image, respectively, compared with the input second image.

[0115] Based on the target depth feature map, feature fusion is performed to obtain a first encoded image.

[0116] Specifically, the device for generating a virtual viewpoint image upsamples the target depth feature map to the same resolution according to the corresponding resolution, splices and fuses them according to the feature dimension, and obtains a first encoded image.

[0117] The number of first encoded images is equal to the number of second images.

[0118] The embodiment of the present invention obtains target depth feature maps of different resolutions based on the second image, and obtains the first encoded image by fusing the target depth feature maps. It is possible to increase the semantic features contained in the image during the image encoding process, thereby improving the accuracy and efficiency of 3D light field information reconstruction.

[0119] Based on any of the above embodiments, the first encoded image and the first target image are input into the feature fusion layer of the image processing model to obtain a second encoded image, including: based on the first encoded image and the first target image, using the DIBR algorithm to project the coordinates into the coordinate system of the target virtual viewpoint to obtain a target projection image.

[0120] Specifically, after receiving the first encoded image and the first target image, the virtual viewpoint image generating device uses a projection algorithm in the feature fusion layer of the image processing model to convert the coordinate values ​​in the pixel coordinate system of the first encoded image into the coordinate values ​​in the camera coordinate system of the first target image to obtain the target projection image.

[0121] The embodiment of the present invention does not specifically limit the projection algorithm and the generated target projection image.

[0122] Preferably, the first target image Dtgt generated in step 102 and the camera parameters Ktgt, Rtgt, Ttgt of its target virtual viewpoint are obtained. i , projected to the target virtual viewpoint through the depth-image based rendering (DIBR) algorithm, and obtain the target projection image E i→tgt The specific transformation process is as follows:

[0123]

[0124] E i→tgt (p) = E i (p′ i )

[0125] Where p is the coordinate of any pixel in the first target image coordinate system, D tgt (p) is the depth value of the current pixel, K i [R i |T i ] is the camera parameter of input viewpoint i, p′ i is the coordinate value of the pixel p in the first target image coordinate system projected to the input viewpoint i coordinate system.

[0126] Target projection image E i→tgt The number and first coded image E i The quantities are equal.

[0127] Based on the target projection image, image fusion is performed to obtain a second encoded image.

[0128] Specifically, the device for generating the virtual viewpoint image fuses the data at the same pixel coordinate position in each target projection image into a feature vector to obtain a second encoded image.

[0129] The second encoded image is used to be input into the depth decoding layer for decoding to obtain a second target image.

[0130] Figure 2 FIG. 1 is one of the sub-flow diagrams of the method for generating a virtual viewpoint image provided by the present invention. Figure 2 As shown, a specific implementation process of obtaining the second target image is given:

[0131] (1) Use the VGG-16 network pre-trained on ImageNet to extract features from the input multi-view images. Output the feature maps of the first three stages of the VGG-16 network as input for subsequent steps.

[0132] (2) The feature maps of the first three stages of the VGG-16 network are equal to the resolution of the original image, half of the original image, and one quarter of the original image, respectively. The half and one quarter feature maps are upsampled to the same resolution as the original image through nearest neighbor interpolation. The feature maps of the three stages are concatenated together in the feature dimension to generate the first encoded image E of each input color image. i .

[0133] (3) Each first coded image E is converted into i Through DIBR transformation, projected to the target viewpoint, the target projection image E is obtained i→tgt .

[0134] (4) N input second images will generate N corresponding target projection images E i→tgt , through the designed self-attention-ray aggregation module, these N E i→tgt The images are fused into a second coded image according to the feature dimension.

[0135] (5) The second encoded image is input into multiple cascaded U-Net networks for decoding to generate a second target image.

[0136] The embodiment of the present invention projects the first coded image through the DIBR algorithm to obtain a target projection image, and obtains a second coded image through fusion of the target projection image. In the fusion process of the coded images, the accuracy and stability of the coordinate transformation can be improved, thereby improving the accuracy and efficiency of 3D light field information reconstruction.

[0137] Based on any of the above embodiments, before the camera parameters of the input viewpoint and the first image are obtained, the method further includes: based on the input viewpoint, using a camera calibration method to obtain the camera parameters of the input viewpoint.

[0138] The camera parameters include camera internal parameters and camera external parameters.

[0139] Specifically, a camera calibration algorithm is used for the camera module corresponding to the input viewpoint to map the world coordinates to the pixel coordinates, and obtain the camera parameters of the input viewpoint.

[0140] The camera parameters of the input viewpoint include the internal parameters and external parameters of the camera module corresponding to the input viewpoint.

[0141] The embodiment of the present invention does not specifically limit the camera calibration algorithm and its process.

[0142] For example, camera calibration can be performed by a checkerboard and an automatic algorithm to convert 3D points with [X, Y, Z] coordinates into 2D pixels with [X, Y] coordinates. The process can be divided into two stages, namely:

[0143] External calibration refers to the conversion process from world coordinates to camera coordinates. External calibration is used to obtain the rotation matrix R and translation matrix T in the camera's external parameters.

[0144] Internal calibration refers to the conversion process from camera coordinates to pixel coordinates. Internal calibration is used to obtain the camera's internal parameters, such as focal length, pixel size, optical center, etc., and express the internal parameters in a matrix K.

[0145] Preferably, the device for generating a virtual viewpoint image calibrates RGB-D cameras corresponding to multiple input viewpoints in the current scene, and obtains the intrinsic parameter Ki and extrinsic parameters Ri and Ti of each RGB-D camera, where i is the serial number of different RGB-D cameras.

[0146] The embodiment of the present invention performs camera calibration based on the input viewpoint to obtain the camera internal parameters and camera external parameters corresponding to the input viewpoint, thereby improving the accuracy of the input viewpoint image acquisition, and further improving the accuracy and efficiency of 3D light field information reconstruction.

[0147] Based on any of the above embodiments, a three-dimensional model of the current scene is extracted based on the camera parameters of the input viewpoint and the first image, including: obtaining depth fusion information using a TSDF algorithm based on the camera parameters of the input viewpoint and the first image.

[0148] Specifically, the device for generating the virtual viewpoint image uses a fusion algorithm to process the first image captured by the corresponding camera module according to the camera posture information indicated by the camera parameters of the input viewpoint to obtain depth fusion information.

[0149] The embodiment of the present invention does not specifically limit the fusion algorithm adopted and the corresponding depth fusion information generated.

[0150] Preferably, the device for generating the virtual viewpoint image constructs a voxel model based on a truncated signed distance function (TSDF) according to the intrinsic and extrinsic parameters of each RGB-D camera and the first image collected to obtain depth fusion information.

[0151] Depth fusion information refers to obtaining the TSDF value of the depth data corresponding to each image from all the depth data of the first images according to the TSDF algorithm, and fusing the TSDF value into the corresponding voxel by continuously updating the TSDF value to form a TSDF voxel model.

[0152] Based on the deep fusion information, the Marching Cube algorithm is used to obtain the 3D model of the current scene.

[0153] Specifically, the device for generating the virtual viewpoint image adopts an extraction algorithm to extract the surface information in the depth fusion information and obtain the three-dimensional model of the current scene.

[0154] The embodiment of the present invention does not specifically limit the extraction algorithm used and the corresponding three-dimensional model generated.

[0155] Preferably, the device for generating the virtual viewpoint image adopts the Marching Cube method to extract triangular faces from the depth fusion information (ie, the TSDF voxel model), and uses a mesh model composed of multiple triangular faces as the three-dimensional model of the current scene.

[0156] The 3D model of the current scene refers to a mesh model containing the 3D information of the current scene. The 3D model of the current scene is used to capture the first target image (i.e., the depth map of the virtual camera position) in combination with the camera parameters of any given target virtual viewpoint (i.e., the virtual camera pose).

[0157] For example, Figure 3 FIG. 2 is a second sub-flow diagram of the method for generating a virtual viewpoint image provided by the present invention. Figure 3 As shown, a specific implementation process of obtaining the first target image is given:

[0158] (1) Calibrate multiple RGB-D cameras placed in the scene and obtain the intrinsic parameters and spatial position parameters Ki, Ri, and Ti of each depth camera.

[0159] (2) Based on the camera parameters of each camera and the collected depth data, all depth data are fused by constructing a TSDF Volume.

[0160] (3) Through the Marching Cube method, the mesh model is extracted from the TSDF Volume to obtain the three-dimensional model of the scene.

[0161] (4) Determine the position of the target virtual viewpoint and its internal parameters and spatial position parameters Ktgt, Rtgt, Ttgt.

[0162] (5) The virtual camera is set according to the parameters Ktgt, Rtgt, and Ttgt of the target virtual viewpoint, and a depth map of the three-dimensional mesh model of the scene is collected to obtain the depth map Dtgt of the target viewpoint.

[0163] The embodiment of the present invention constructs the depth fusion information of the voxel space through the TSDF algorithm based on the camera parameters of the input viewpoint and the first image, and calculates the three-dimensional model of the current scene through the Marching Cube algorithm for the depth fusion information. It can improve the accuracy of the three-dimensional model of the current scene, and further improve the accuracy and efficiency of 3D light field information reconstruction.

[0164] Figure 4 FIG. 1 is a schematic diagram of the structure of a device for generating a virtual viewpoint image provided by the present invention. Figure 4 As shown, the device includes: a three-dimensional model acquisition module 410, a first acquisition module 420 and a second acquisition module 430, wherein:

[0165] The three-dimensional model acquisition module 410 is used to extract the three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image.

[0166] The first acquisition module 420 is used to acquire a first target image of the target virtual viewpoint based on the three-dimensional model and the camera parameters of the target virtual viewpoint.

[0167] The second acquisition module 430 is used to input the second image and the first target image into the image processing model to obtain a second target image of the target virtual viewpoint.

[0168] Among them, the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is the depth map of the current scene collected under the input viewpoint, the second image is the color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

[0169] Specifically, the three-dimensional model acquisition module 410 , the first acquisition module 420 , and the second acquisition module 430 are electrically connected in sequence.

[0170] The 3D model acquisition module 410 acquires the first image taken by the corresponding camera module at the input viewpoint, fuses all the first images according to the posture information indicated by the camera parameters of the corresponding input viewpoints, and restores the 3D model of the current scene.

[0171] The three-dimensional model of the current scene refers to a data set including the three-dimensional geometric information and structural information of the scene.

[0172] The first acquisition module 420 sets the virtual camera module through the camera parameters of the target virtual viewpoint, collects the depth map of the three-dimensional model of the current scene, and obtains the first target image.

[0173] The first target image refers to a depth map collected by a correspondingly configured virtual camera module at a target virtual viewpoint.

[0174] The second acquisition module 430 initializes the weight coefficients between the layers of the image processing model constructed, and trains and tests the model in sequence according to the real color image of the target virtual viewpoint, so as to obtain the most accurate model parameters, namely the weight coefficients of the input layer and the hidden layer.

[0175] The device for generating virtual viewpoint images sets the image processing model according to the trained model parameters, and then processes each second image in turn through the model in combination with the first target image to obtain the second target image.

[0176] The second target image refers to a color image captured by a correspondingly set virtual camera module at the target virtual viewpoint.

[0177] Optionally, the second acquisition module 430 includes an encoding unit, a fusion unit and a decoding unit, wherein:

[0178] The encoding unit is used to input the second image into the depth encoding layer of the image processing model to obtain a first encoded image corresponding to the second image.

[0179] The fusion unit is used to input the first encoded image and the first target image into the feature fusion layer of the image processing model to obtain a second encoded image.

[0180] The decoding unit is used to input the second encoded image into the depth decoding layer of the image processing model to obtain a second target image of the target virtual viewpoint.

[0181] Optionally, the encoding unit includes a feature extraction subunit and a first fusion subunit, wherein:

[0182] The feature extraction subunit is used to perform feature extraction based on the second image to obtain a target depth feature map.

[0183] The first fusion subunit is used to perform feature fusion based on the target depth feature map to obtain a first encoded image.

[0184] Among them, the target depth feature map includes at least two depth feature maps with different resolutions.

[0185] Optionally, the fusion unit includes a projection subunit and a second fusion subunit, wherein:

[0186] The projection subunit is used to project the coordinates to the coordinate system of the target virtual viewpoint based on the first coded image and the first target image by adopting the DIBR algorithm to obtain the target projection image.

[0187] The second fusion subunit is used to perform image fusion based on the target projection image to obtain a second encoded image.

[0188] Optionally, the device further comprises a calibration module, wherein:

[0189] The calibration module is used to obtain the camera parameters of the input viewpoint by adopting a camera calibration method based on the input viewpoint.

[0190] The camera parameters include camera internal parameters and camera external parameters.

[0191] Optionally, the 3D model acquisition module 410 includes a depth fusion unit and a model acquisition unit, wherein:

[0192] The depth fusion unit is used to obtain depth fusion information by using a TSDF algorithm based on the camera parameters of the input viewpoint and the first image.

[0193] The model acquisition unit is used to acquire the three-dimensional model of the current scene based on the depth fusion information and using the Marching Cube algorithm.

[0194] The virtual viewpoint image generation device provided in an embodiment of the present invention is used to execute the above-mentioned virtual viewpoint image generation method of the present invention. Its implementation method is consistent with the implementation method of the virtual viewpoint image generation method provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0195] The embodiment of the present invention obtains a three-dimensional model of a scene based on the camera parameters of an input viewpoint and a first image, obtains a first target image through the camera parameters of a target virtual viewpoint, obtains a second target image through an image processing model inputted by a second image, restores geometric information of a corresponding three-dimensional scene through the first target image, and restores illumination and color information of a corresponding three-dimensional scene through the second target image, and obtains a target virtual viewpoint image. The accuracy and efficiency of 3D light field information reconstruction can be improved.

[0196] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the method for generating a virtual viewpoint image, the method comprising: extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image; acquiring a first target image of the target virtual viewpoint based on the three-dimensional model and the camera parameters of the target virtual viewpoint; inputting the second image and the first target image into the image processing model to obtain a second target image of the target virtual viewpoint; wherein the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is a depth image of the current scene collected under the input viewpoint, the second image is a color image of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

[0197] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0198] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual viewpoint image generation method provided by the above-mentioned methods, which includes: extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image; acquiring a first target image of the target virtual viewpoint based on the three-dimensional model and the camera parameters of the target virtual viewpoint; inputting the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint; wherein the image processing model is obtained after training based on a real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene collected under the input viewpoint, the second image is a color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

[0199] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for generating a virtual viewpoint image provided by the above-mentioned methods, the method comprising: extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image; acquiring a first target image of the target virtual viewpoint based on the three-dimensional model and the camera parameters of the target virtual viewpoint; inputting the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint; wherein the image processing model is obtained after training based on a real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene captured under the input viewpoint, the second image is a color image of the current scene captured under the input viewpoint, and the number of input viewpoints is one or more.

[0200] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a virtual viewpoint image, characterized in that: include: Extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image; Based on the three-dimensional model and camera parameters of the target virtual viewpoint, acquiring a first target image of the target virtual viewpoint; Inputting the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint; In which, the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene collected under the input viewpoint, the second image is a color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

2. The method for generating a virtual viewpoint image according to claim 1, characterized in that: The step of inputting the second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint includes: Inputting the second image into a depth coding layer of the image processing model to obtain a first coded image corresponding to the second image; Inputting the first encoded image and the first target image into a feature fusion layer of the image processing model to obtain a second encoded image; The second encoded image is input into the depth decoding layer of the image processing model to obtain a second target image of the target virtual viewpoint.

3. The method for generating a virtual viewpoint image according to claim 2, characterized in that: The step of inputting the second image into the depth coding layer of the image processing model to obtain a first coded image corresponding to the second image includes: Based on the second image, feature extraction is performed to obtain a target depth feature map; Based on the target depth feature map, feature fusion is performed to obtain the first encoded image; Wherein, the target depth feature map includes at least two depth feature maps with different resolutions.

4. The method for generating a virtual viewpoint image according to claim 2, characterized in that: The step of inputting the first encoded image and the first target image into a feature fusion layer of the image processing model to obtain a second encoded image includes: Based on the first coded image and the first target image, using the DIBR algorithm, the coordinates of the first coded image are projected onto the coordinate system of the target virtual viewpoint to obtain a target projection image; Image fusion is performed based on the target projection image to obtain the second encoded image.

5. The method for generating a virtual viewpoint image according to claim 1, characterized in that: Before the camera parameters based on the input viewpoint and the first image, it also includes: based on the input viewpoint, using a camera calibration method to obtain the camera parameters of the input viewpoint; The camera parameters include camera internal parameters and camera external parameters.

6. The method for generating a virtual viewpoint image according to claim 5, characterized in that: The step of extracting a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image includes: Based on the camera parameters of the input viewpoint and the first image, a TSDF algorithm is used to obtain depth fusion information; Based on the depth fusion information, a Marching Cube algorithm is used to obtain a three-dimensional model of the current scene.

7. A device for generating a virtual viewpoint image, characterized in that: include: A three-dimensional model acquisition module, used to extract a three-dimensional model of the current scene based on the camera parameters of the input viewpoint and the first image; A first acquisition module, configured to acquire a first target image of the target virtual viewpoint based on the three-dimensional model and camera parameters of the target virtual viewpoint; A second acquisition module, used for inputting a second image and the first target image into an image processing model to obtain a second target image of the target virtual viewpoint; In which, the image processing model is obtained after training based on the real color image of the target virtual viewpoint as a sample, the first image is a depth map of the current scene collected under the input viewpoint, the second image is a color map of the current scene collected under the input viewpoint, and the number of input viewpoints is one or more.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for generating a virtual viewpoint image according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a virtual viewpoint image according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a virtual viewpoint image according to any one of claims 1 to 6 are implemented.

Images