Virtual viewpoint image generation method and device, terminal equipment and storage medium

Abstract

The embodiment of the invention provides a virtual viewpoint image generation method and device, terminal equipment and a storage medium, and the method comprises the steps: obtaining the image features of a material image and a pixel depth map corresponding to the material image; obtaining an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features; obtaining an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generating a first candidate frame according to the material image and the convolution kernel parameter, and generating a second candidate frame according to the multi-size feature; and fusing the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image. The depth information of the pixel depth map is fused in the generated virtual viewpoint image, so that the image quality problems of cavities, overlapping and artifacts of the output virtual viewpoint image can be effectively reduced or eliminated, and the image quality of the output virtual viewpoint image is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional display technology, and in particular to a method, device, terminal equipment and storage medium for generating a virtual viewpoint image. Background Art

[0002] At present, with the continuous advancement of three-dimensional display technology, ordinary 2D videos are gradually developing into 3D videos. As a key technology for three-dimensional display, the depth image based virtual viewpoint image rendering technology (Depth Image Based Rendering, DIBR) has the advantages of fast rendering speed, small amount of transmission data, and less storage space. In addition, due to the quality of the depth map and the occlusion between objects, the rendered image still has problems such as holes, overlaps, and artifacts, resulting in an overall decrease in the image quality of the virtual viewpoint image.

[0003] With the development of hardware performance and deep learning technology, deep learning technology has shown strong understanding and perception of high-level semantics of images. Many deep learning-based methods have also been applied to virtual view synthesis. However, directly using existing deep learning for hole filling will cause local color aberration and blur in the repaired image.

[0004] Therefore, how to improve the image quality of virtual viewpoint images is a hot topic being studied by those skilled in the art. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a method, an apparatus, a terminal device and a storage medium for generating a virtual viewpoint image, aiming to improve the image quality of the virtual viewpoint image.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a virtual viewpoint image, the method comprising:

[0007] Acquire image features of a material image and a pixel depth map corresponding to the material image;

[0008] Acquire an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image according to the image features;

[0009] Acquire an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features;

[0010] The first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0011] In a second aspect, an embodiment of the present application further provides an upgrading device for a network element device, including:

[0012] A device for generating a virtual viewpoint image, characterized in that it comprises:

[0013] An image processing module, used to obtain image features of a material image and a pixel depth map corresponding to the material image;

[0014] A feature processing module, used to obtain an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features;

[0015] A candidate image module, configured to obtain an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features;

[0016] An image generation module is used to fuse the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0017] In a third aspect, an embodiment of the present application further provides a terminal device, comprising: a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing a connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the aforementioned method for generating a virtual viewpoint image are implemented.

[0018] In a fourth aspect, an embodiment of the present application further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of a method for generating a virtual viewpoint image.

[0019] The embodiment of the present application provides a method, apparatus, terminal device and storage medium for generating a virtual viewpoint image, wherein the method for generating a virtual viewpoint image obtains the image features of a material image and a pixel depth map corresponding to the material image; obtains an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features; obtains an image fusion mask matrix according to the pixel depth map and the initial mask matrix, and generates a first candidate frame according to the material image and the convolution kernel parameters, and generates a second candidate frame according to the multi-size features; and fuses the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image. Based on the depth information of the pixel depth map fused in the generated virtual viewpoint image, the hole, overlap and artifact image quality problems of the output virtual viewpoint image can be effectively reduced or eliminated, thereby improving the image quality of the output virtual viewpoint image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flowchart of a method for generating a virtual viewpoint image provided by an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of a scene of obtaining a pixel depth map by using a material image in a method for generating a virtual viewpoint image provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a scene in which a virtual viewpoint image is generated by a method for generating a virtual viewpoint image according to an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of a scene in which a method for generating a virtual viewpoint image provided in an embodiment of the present application uses a pixel depth map to optimize a mask matrix;

[0025] Figure 5 It is a schematic diagram of the module structure of a device for generating a virtual viewpoint image provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the framework structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] At present, with the continuous advancement of three-dimensional display technology, ordinary 2D videos are gradually developing into 3D videos. As a key technology for three-dimensional display, the depth image based virtual viewpoint image rendering technology (Depth Image Based Rendering, DIBR) has the advantages of fast rendering speed, small amount of transmission data, and less storage space. In addition, due to the quality of the depth map and the occlusion between objects, the rendered image still has problems such as holes, overlaps, and artifacts, resulting in an overall decrease in the image quality of the virtual viewpoint image.

[0032] With the development of hardware performance and deep learning technology, deep learning technology has shown strong understanding and perception of high-level semantics of images. Many deep learning-based methods have also been applied to virtual view synthesis. However, directly using existing deep learning for hole filling will cause local color aberration and blur in the repaired image.

[0033] Therefore, how to improve the image quality of virtual viewpoint images is a hot topic being studied by those skilled in the art.

[0034] Based on this, the embodiment of the present application provides a method, apparatus, terminal device and storage medium for generating a virtual viewpoint image, wherein the method for generating a virtual viewpoint image obtains the image features of a material image and a pixel depth map corresponding to the material image; obtains an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features; obtains an image fusion mask matrix according to the pixel depth map and the initial mask matrix, and generates a first candidate frame according to the material image and the convolution kernel parameters, and generates a second candidate frame according to the multi-size features; fuses the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image. The virtual viewpoint image generated based on this is fused with the depth information of the pixel depth map, so that the image quality problems of holes, overlaps and artifacts of the output virtual viewpoint image are reduced or eliminated, and the image quality of the output virtual viewpoint image is improved.

[0035] See also Figure 1 , Figure 1 A flowchart of the steps of a method for generating a virtual viewpoint image provided in an embodiment of the present application.

[0036] The method for generating a virtual viewpoint image is applied to a terminal device, wherein the terminal device may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, a wearable device, etc.; the server may be an independent server or a server cluster.

[0037] like Figure 1 As shown, the method for generating a virtual viewpoint image includes steps S101 to S104.

[0038] Step S101: Acquire image features of a material image and a pixel depth map corresponding to the material image;

[0039] Exemplarily, the acquired image features are used to characterize at least the features of each image element in the pixel image and the two-dimensional spatial relationship between each image element. The pixel depth map is used to characterize the three-dimensional spatial relationship between each image element of the material image, and the foreground and background parts in the pixel depth map are distinguished by acquiring the depth information of each pixel point in the pixel depth map.

[0040] After determining the material image for generating the virtual viewpoint image, the corresponding model or neural network is used to extract features of the pixel image to obtain image features corresponding to the material image and a pixel depth map corresponding to the material image.

[0041] In some implementations, obtaining the image features of the material image and the pixel depth map corresponding to the material image includes:

[0042] Acquire image features of the material image through a feature extraction network;

[0043] A pixel depth map corresponding to the material image is obtained through a depth extraction network.

[0044] Exemplarily, the feature extraction network includes but is not limited to a U-Net network, and the feature extraction network extracts features of the material image to obtain image features of the material image. The image features include at least an initial mask matrix, convolution kernel parameters, and a feature pyramid, and the image features are used to describe the color features, texture features, shape features, and spatial relationship features of the image.

[0045] like Figure 2 As shown, a depth extraction network is used to extract depth information of a material image to obtain a pixel depth map corresponding to the material image, wherein the depth extraction network is a monocular depth estimation network, such as a MegaDepth network.

[0046] In the present application, the material image is described by taking the example that the material image includes the first material image I0 and the second material image I1, but the material image is not limited to including the first material image I0 and the second material image I1.

[0047] like Figure 2As shown, after the first material image I0 and the second material image I1 are subjected to depth information extraction by a depth extraction network, a first pixel depth map D0 and a second pixel depth map D1 are obtained, wherein the first pixel depth map D0 is extracted from the first material image I0, and the second pixel depth map D1 is extracted from the second material image I1.

[0048] like Figure 3 As shown, in the process of extracting the image features of the first material image I0 and the second material image I1 using the feature extraction network, the high-dimensional features are first extracted by the feature extraction network, and then the high-dimensional features are sampled into low-dimensional features of the original image size. In the process of gradually extracting high-dimensional features from the original first material image I0 and the second material image I1, multi-scale features of different dimensions can be extracted layer by layer, and these multi-scale features constitute a feature pyramid. At the same time, and in the process of upsampling the high-dimensional features into low-dimensional features of the original image size, the convolution kernel parameter matrix W1, W2, the pixel displacement vector matrix α1, β1, α2, β2, and the initial mask matrix V1, V2 of the same size as the original first material image I0 and the second material image I1 can also be obtained. Among them, the convolution kernel parameter matrix W1, W2, the pixel displacement vector matrix α1, β1, α2, β2 constitute the convolution kernel parameters.

[0049] Step S102: obtaining an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features.

[0050] After obtaining the image features of the pixel image through the feature extraction network, the relevant information required for synthesizing the virtual viewpoint image is extracted from the image features, such as the initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image.

[0051] In some implementations, obtaining an initial mask matrix and convolution kernel parameters according to the image features includes:

[0052] Obtaining an initial mask matrix using the image features based on a first extraction network;

[0053] The convolution kernel parameters are obtained using the image features based on the second extraction network, wherein the first extraction network and the second extraction network are both convolutional neural networks, and the convolutional neural network parameters corresponding to the first extraction network and the second extraction network are different.

[0054] Exemplarily, an initial mask matrix is ​​obtained from image features through a first extraction network, and then convolution kernel parameters are obtained using image features through a second extraction network, wherein the first extraction network and the second extraction network are both convolutional neural networks, and the convolutional neural network parameters corresponding to the first extraction network and the second extraction network are different.

[0055] Optionally, the first extraction network and the second extraction network both include a convolution layer, a pooling layer and an activation function layer, wherein at least one network layer parameter among the convolution layer, the pooling layer and the activation function layer of the first extraction network and the second extraction network is different.

[0056] For example, the image features are input into the first extraction network to obtain the initial mask matrix (V1, V2), and the image features are input into the second extraction network to obtain the convolution kernel parameters (W1, W2, α1, β1, α2, β2). The convolution kernel parameters include the convolution kernel parameter matrix (W1, W2) and the pixel displacement vector matrix (α1, β1, α2, β2).

[0057] In some implementations, obtaining multi-size features corresponding to the material image according to the image features includes:

[0058] Extracting a feature image pyramid from the image features, wherein the feature image pyramid includes size features of multiple dimensions of the material image;

[0059] The multi-size features corresponding to the material image are obtained according to the feature image pyramid.

[0060] Step S103: obtaining an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generating a first candidate frame according to the material image and the convolution kernel parameters, and generating a second candidate frame according to the multi-size features.

[0061] Exemplarily, a first candidate frame is generated by using an initial material image and convolution kernel parameters so that the image corresponding to the first candidate frame has corresponding image features of the material image, and image fusion is performed using the first candidate frame so that the generated image has the image features of the material image.

[0062] The second candidate frame is generated by using multi-size features, so that the second candidate frame image has the morphological, texture, structural and other characteristics of the material image at different sizes. Then, the second candidate frame is used for image fusion so that the generated image can obtain more accurate and reliable spatial information, thereby improving the image quality of the output image.

[0063] The obtained pixel depth map is used to optimize the obtained initial mask matrix to obtain an image fusion mask matrix. Based on the three-dimensional information of the image attached to the pixel depth map, the image fusion mask matrix obtained by optimizing the initial mask matrix with the pixel depth map can better optimize the foreground and background of the synthesized image during the image fusion process, thereby further improving the image quality of the output image.

[0064] like Figure 3 and 4As shown, for the purpose of illustration, the material image includes a first material image I0 and a second material image I1. After the initial mask matrix (V1, V2) and the pixel depth map are extracted according to the image features corresponding to the original material image, the depth adaptive fusion module of the terminal device is used to optimize the initial mask matrix V1 and the optimized mask matrix V1`. Finally, the image fusion mask matrix (V1`, V2) is obtained, that is, the image fusion mask matrix includes the first fusion mask matrix V1` and the second fusion mask matrix V2.

[0065] Specifically, firstly, the pixel depth maps (D0, D1) corresponding to the first material image I0 and the second material image I1 are extracted through the depth extraction network, and then the optimized mask matrix V1' is obtained through the mask matrix optimization module. The specific calculation formula is as follows:

[0066]

[0067] Where V1`(i, j) is the value of the optimized mask matrix at position (i, j), V1(i, j) is the value of the initial mask matrix at position (i, j), D0(i, j) and D1(i, j) are the values ​​of the pixel depth maps D0 and D1 at position (i, j), respectively.

[0068] That is, the material images (I0, I1) include a first material image I0 and a second material image I1, the pixel depth map (D0, D1) includes a first pixel depth map D0 corresponding to the first material image I0 and a second pixel depth map D1 corresponding to the second material image I1; the initial mask matrix (V1, V2) includes a first initial mask matrix V1 and a second initial mask matrix V2.

[0069] Optionally, the first fused mask matrix V1' is obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map. For example, the first fused mask matrix V1' is obtained by optimizing a first initial mask matrix V1 of the initial mask matrix (V1, V2) according to the first pixel depth map.

[0070] It can be understood that the fused mask matrix can also be obtained by optimizing the corresponding initial mask matrices according to the pixel depth map, that is, the fused mask matrix can be a first initial mask matrix V1 obtained by optimizing the initial mask matrix (V1, V2) according to the first pixel depth map, and a second initial mask matrix V2 obtained by optimizing the initial mask matrix (V1, V2) according to the second pixel depth map.

[0071] and the second pixel depth map to optimize the initial mask matrix. For example, the first fusion mask matrix V1′ is obtained by optimizing the initial mask matrix (V1, V2) according to the second pixel depth map.

[0072] In some implementations, based on the material images including the first material image I0 and the second material image I1 , the first candidate frame generated according to the material images and the convolution kernel parameters includes at least two images.

[0073] For example, two source images (I0, I1) and convolution kernel parameters (W1, W2, α1, β1, α2, β2) are used to generate a first candidate frame (I t0 ,I t1 ), the specific calculation formula is as follows:

[0074]

[0075] The formula is used to calculate the first image I of the first candidate frame using the first source image I0 and the first set of convolution kernel parameters (W1, α1, β1). t0 , using the second material image I1 and the second set of convolution kernel parameters (W2, α2, β2), to calculate the second image I of the first candidate frame t1 .

[0076] Among them, the formula is the pixel value of the mapped image at position (i, j); F is the convolution kernel size; k, l are the convolution kernel position indexes; W k,l (i, j) is the convolution kernel parameter used for the pixel value of the mapped image at position (i, j) when the convolution kernel index is k, l; I(i+k+α k,l ,j+l+β k,l ) is the pixel value of the image at position (i, j) after mapping, and the convolution kernel parameters used are when the convolution kernel index is k, l, the pixel point is the displacement vector is α k,l , β k,l is the pixel value of the original material image I.

[0077] In some implementations, generating a second candidate frame according to the multi-size feature includes:

[0078] A plurality of mapping images are generated according to the multi-scale feature maps, and the mapping images are synthesized into a second candidate frame through a synthesis network.

[0079] Exemplarily, multi-scale features are extracted from the image features extracted from the feature extraction network, image mapping is performed on the multi-scale features, and then the multi-scale candidate second frame It2 is synthesized through a synthesis network (eg, a GridNet network).

[0080] Step S104: fusing the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0081] Exemplarily, after obtaining the first candidate frame, the second candidate frame and the image fusion mask matrix, the image fusion mask matrix is ​​used to fuse the first candidate frame and the second candidate frame to output a virtual viewpoint image. Based on the fact that the first candidate frame has image features of the material image, the second candidate frame has relatively accurate and reliable spatial information, and the image fusion mask matrix has three-dimensional spatial information, the virtual viewpoint image generated by fusing the first candidate frame and the second candidate frame using the image fusion mask matrix fuses the depth information of the pixel depth map, reliable spatial information and the initial material image information, thereby effectively reducing or eliminating the image quality problems of holes, overlaps, and artifacts in the output virtual viewpoint image, thereby improving the image quality of the output virtual viewpoint image.

[0082] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including:

[0083] fusing the first candidate frames according to the first fusion mask matrix to obtain a third candidate frame;

[0084] The second candidate frame and the third candidate frame are fused according to the second fusion mask matrix to obtain a virtual viewpoint image.

[0085] like Figure 3 As shown, the first candidate frame (I t0 ,I t1 ), the second candidate frame It2 is fused to obtain the final synthesized virtual viewpoint image I out , where the fusion formula is as follows:

[0086] I out =V⊙I n +(JV)⊙I n+1

[0087] Among them, In and In+1 are candidate frames to be fused, V is the image fusion mask matrix (the values ​​in the matrix are all greater than 0 and less than 1), J is an all-1 matrix, and Iout is the fused image.

[0088] Figure 3The image reconstruction module in the image reconstruction module uses the convolution parameters W, α, β, and the original image I to reconstruct the image of the target viewpoint. In the process of image fusion, the first fusion mask matrix V1` of the image fusion mask matrix optimized by the proposed deep adaptive fusion module and the first candidate frames It0 and It1 are first fused to generate the third candidate frame It3. Then, the second fusion mask matrix V2 of the image fusion mask matrix and the candidate frames It2 and It3 are fused to generate the final virtual viewpoint image I out .

[0089] See also Figure 5 , Figure 5 It is a schematic diagram of the module structure of a device for generating a virtual viewpoint image provided in an embodiment of the present application.

[0090] like Figure 5 As shown, the device 200 for generating a virtual viewpoint image includes an image processing module 201, a feature processing module 202, a candidate image module 203 and an image generation module 204. The image processing module 201 is used to obtain the image features of the material image and the pixel depth map corresponding to the material image. The feature processing module 202 is used to obtain the initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features. The candidate image module 203 is used to obtain the image fusion mask matrix according to the pixel depth map and the initial mask matrix, and to generate a first candidate frame according to the material image and the convolution kernel parameters, and to generate a second candidate frame according to the multi-size features. The image generation module 204 is used to fuse the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0091] In some implementations, obtaining the image features of the material image and the pixel depth map corresponding to the material image includes:

[0092] Acquire image features of the material image through a feature extraction network;

[0093] A pixel depth map corresponding to the material image is obtained through a depth extraction network.

[0094] In some implementations, obtaining an initial mask matrix and convolution kernel parameters according to the image features includes:

[0095] Obtaining an initial mask matrix using the image features based on a first extraction network;

[0096] The convolution kernel parameters are obtained using the image features based on the second extraction network, wherein the first extraction network and the second extraction network are both convolutional neural networks, and the convolutional neural network parameters corresponding to the first extraction network and the second extraction network are different.

[0097] In some implementations, obtaining multi-size features corresponding to the material image according to the image features includes:

[0098] Extracting a feature image pyramid from the image features, wherein the feature image pyramid includes size features of multiple dimensions of the material image;

[0099] The multi-size features corresponding to the material image are obtained according to the feature image pyramid.

[0100] In some implementations, generating a second candidate frame according to the multi-size feature includes:

[0101] A plurality of mapping images are generated according to the multi-scale feature maps, and the mapping images are synthesized into a second candidate frame through a synthesis network.

[0102] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including:

[0103] fusing the first candidate frames according to the first fusion mask matrix to obtain a third candidate frame;

[0104] The second candidate frame and the third candidate frame are fused according to the second fusion mask matrix to obtain a virtual viewpoint image.

[0105] In some embodiments, the material image includes a first material image and a second material image, the pixel depth map includes a first pixel depth map corresponding to the first material image and a second pixel depth map corresponding to the second material image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix;

[0106] The first fusion mask matrix is ​​obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map.

[0107] It can be understood that in some embodiments, the virtual viewpoint image generation device 200 can be applied to the terminal device 10, for example, the virtual viewpoint image generation device 200 is integrated in the terminal device 10, or is independently set relative to the terminal device 10 and communicated with the terminal device 10.

[0108] See also Figure 6 , Figure 6 A schematic diagram of the framework structure of a terminal device provided in an embodiment of the present application.

[0109] like Figure 6 As shown, the terminal device 10 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, and the bus 103 is, for example, an I2C (Inter-integrated Circuit) bus.

[0110] Specifically, the processor 101 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0111] The memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0112] The processor 101 is used to run the computer program stored in the memory 102, and implements the following steps when executing the computer program:

[0113] Acquire image features of a material image and a pixel depth map corresponding to the material image;

[0114] Acquire an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image according to the image features;

[0115] Acquire an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features;

[0116] The first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0117] In some implementations, obtaining the image features of the material image and the pixel depth map corresponding to the material image includes:

[0118] Acquire image features of the material image through a feature extraction network;

[0119] A pixel depth map corresponding to the material image is obtained through a depth extraction network.

[0120] In some implementations, obtaining an initial mask matrix and convolution kernel parameters according to the image features includes:

[0121] Obtaining an initial mask matrix using the image features based on a first extraction network;

[0122] The convolution kernel parameters are obtained using the image features based on the second extraction network, wherein the first extraction network and the second extraction network are both convolutional neural networks, and the convolutional neural network parameters corresponding to the first extraction network and the second extraction network are different.

[0123] In some implementations, obtaining multi-size features corresponding to the material image according to the image features includes:

[0124] Extracting a feature image pyramid from the image features, wherein the feature image pyramid includes size features of multiple dimensions of the material image;

[0125] The multi-size features corresponding to the material image are obtained according to the feature image pyramid.

[0126] In some implementations, generating a second candidate frame according to the multi-size feature includes:

[0127] A plurality of mapping images are generated according to the multi-scale feature maps, and the mapping images are synthesized into a second candidate frame through a synthesis network.

[0128] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including:

[0129] fusing the first candidate frames according to the first fusion mask matrix to obtain a third candidate frame;

[0130] The second candidate frame and the third candidate frame are fused according to the second fusion mask matrix to obtain a virtual viewpoint image.

[0131] In some embodiments, the material image includes a first material image and a second material image, the pixel depth map includes a first pixel depth map corresponding to the first material image and a second pixel depth map corresponding to the second material image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix;

[0132] The first fusion mask matrix is ​​obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map.

[0133] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned corresponding virtual viewpoint image generation method embodiment, and will not be repeated here.

[0134] An embodiment of the present application also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of generating a virtual viewpoint image of any embodiment provided in the specification of the embodiment of the present application.

[0135] The storage medium may be an internal storage unit of the aforementioned terminal device 10, such as a hard disk or memory of the terminal device 10. The storage medium may also be an external storage device of the terminal device 10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device 10.

[0136] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0137] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0138] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a virtual viewpoint image, characterized in that: The method comprises: Acquire image features of a material image and a pixel depth map corresponding to the material image; Acquire an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image according to the image features; Acquire an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features; The first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image.

2. The method according to claim 1, characterized in that The obtaining of the image features of the material image and the pixel depth map corresponding to the material image includes: Acquire image features of the material image through a feature extraction network; A pixel depth map corresponding to the material image is obtained through a depth extraction network.

3. The method according to claim 1, characterized in that The obtaining of the initial mask matrix and convolution kernel parameters according to the image features includes: Obtaining an initial mask matrix using the image features based on the first extraction network; The convolution kernel parameters are obtained using the image features based on the second extraction network, wherein the first extraction network and the second extraction network are both convolutional neural networks, and the convolutional neural network parameters corresponding to the first extraction network and the second extraction network are different.

4. The method according to claim 1, characterized in that: Acquiring multi-size features corresponding to the material image according to the image features includes: Extracting a feature image pyramid from the image features, wherein the feature image pyramid includes size features of multiple dimensions of the material image; The multi-size features corresponding to the material image are obtained according to the feature image pyramid.

5. The method according to claim 1, characterized in that: The generating a second candidate frame according to the multi-size feature comprises: A plurality of mapping images are generated according to the multi-scale feature maps, and the mapping images are synthesized into a second candidate frame through a synthesis network.

6. The method according to any one of claims 1 to 5, characterized in that: The image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including: fusing the first candidate frames according to the first fusion mask matrix to obtain a third candidate frame; The second candidate frame and the third candidate frame are fused according to the second fusion mask matrix to obtain a virtual viewpoint image.

7. The method according to claim 6, characterized in that The material image includes a first material image and a second material image, the pixel depth map includes a first pixel depth map corresponding to the first material image and a second pixel depth map corresponding to the second material image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix; The first fusion mask matrix is ​​obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map.

8. A device for generating a virtual viewpoint image, characterized in that: include: An image processing module, used to obtain image features of a material image and a pixel depth map corresponding to the material image; A feature processing module, used to obtain an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features; A candidate image module, configured to obtain an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features; An image generation module is used to fuse the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image.

9. A terminal device, characterized in that: The terminal device includes: a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the method for generating a virtual viewpoint image as described in any one of claims 1-7 are realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the method for generating a virtual viewpoint image according to any one of claims 1 to 7.

Images