Method and apparatus for generating virtual viewpoint image, terminal device, and storage medium

Abstract

Embodiments of the present application provide a method and apparatus for generating a virtual viewpoint image, a terminal device, and a storage medium. The method for generating a virtual viewpoint image comprises: acquiring an image feature of a materials image and a pixel depth map corresponding to the materials image; on the basis of the image feature, acquiring an initial mask matrix, a convolution kernel parameter, and a multi-scale feature corresponding to the materials image; acquiring an image fusion mask matrix on the basis of the pixel depth map and the initial mask matrix, generating a first candidate frame on the basis of the materials image and the convolution kernel parameter, and generating a second candidate frame on the basis of the multi-scale feature; and fusing the first candidate frame and the second candidate frame on the basis of the image fusion mask matrix to obtain a virtual viewpoint image.

Description

Method and device for generating virtual viewpoint image, terminal device and storage medium

[0001] This application claims priority to Chinese patent application No. 202311134294.2, filed on September 4, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of three-dimensional display technology, and in particular to a method and apparatus for generating a virtual viewpoint image, a terminal device, and a storage medium. Background Art

[0003] With the continuous advancement of three-dimensional display technology, conventional two-dimensional (2D) video is gradually evolving into three-dimensional (3D) video. Depth Image Based Rendering (DIBR), a key technology for 3D display, offers advantages such as fast rendering speed, small data transmission volume, and minimal storage space.

[0004] Summary of the Invention

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a virtual viewpoint image, which includes: obtaining image features of a material image and a pixel depth map corresponding to the material image; obtaining an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image based on the image features; obtaining an image fusion mask matrix based on the pixel depth map and the initial mask matrix, and generating a first candidate frame based on the material image and the convolution kernel parameters, and generating a second candidate frame based on the multi-size features; fusing the first candidate frame and the second candidate frame based on the image fusion mask matrix to obtain a virtual viewpoint image.

[0006] In a second aspect, an embodiment of the present disclosure also provides a device for generating a virtual viewpoint image, which includes: an image processing module for obtaining image features of a material image and a pixel depth map corresponding to the material image; a feature processing module for obtaining an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image based on the image features; a candidate image module for obtaining an image fusion mask matrix based on the pixel depth map and the initial mask matrix, and generating a first candidate frame based on the material image and the convolution kernel parameters, and generating a second candidate frame based on the multi-size features; an image generation module for fusing the first candidate frame and the second candidate frame based on the image fusion mask matrix to obtain a virtual viewpoint image.

[0007] In a third aspect, an embodiment of the present disclosure further provides a terminal device, which includes: a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory. When the computer program is executed by the processor, the aforementioned method for generating a virtual viewpoint image is realized.

[0008] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium for computer-readable storage, wherein the storage medium stores at least one program, and the at least one program can be executed by at least one processor to implement a method for generating a virtual viewpoint image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0010] FIG1 is a flowchart of a method for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0011] FIG2 is a schematic diagram of a scenario of obtaining a pixel depth map by using a material image in a method for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0012] FIG3 is a schematic diagram of a scene in which a virtual viewpoint image is generated using a method for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0013] FIG4 is a schematic diagram of a scenario in which a mask matrix is ​​optimized using a pixel depth map in a method for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0014] FIG5 is a schematic diagram of the module structure of a device for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0015] FIG6 is a schematic diagram of a framework structure of a terminal device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0017] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0018] It should be understood that the terms used in this disclosure are only for the purpose of describing particular embodiments and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0019] The following embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.

[0020] With the continuous advancement of 3D display technology, conventional 2D videos are gradually evolving into 3D videos. Depth Image Based Rendering (DIBR), a key technology for 3D display, offers advantages such as fast rendering speed, minimal data transmission, and reduced storage space. However, due to the quality of depth maps and occlusion between objects, the rendered images can still suffer from issues such as holes, overlaps, and artifacts, resulting in a decrease in the overall quality of virtual viewpoint images.

[0021] With the advancement of hardware performance and deep learning technology, deep learning has demonstrated a strong understanding and perception of high-level image semantics. Many deep learning-based methods have also been applied to virtual view synthesis. However, directly using relevant deep learning methods for hole filling can lead to localized color differences and blurring in the restored image.

[0022] Therefore, how to improve the image quality of virtual viewpoint images is a hot topic being studied by those skilled in the art.

[0023] Based on this, the embodiments of the present disclosure provide a method and apparatus for generating a virtual viewpoint image, a terminal device, and a storage medium. In the method for generating a virtual viewpoint image, the image features of the material image and the pixel depth map corresponding to the material image are obtained; the initial mask matrix, convolution kernel parameters, and multi-scale features corresponding to the material image are obtained according to the image features; the image fusion mask matrix is ​​obtained according to the pixel depth map and the initial mask matrix, and a first candidate frame is generated according to the material image and the convolution kernel parameters, and a second candidate frame is generated according to the multi-scale features; the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image. The virtual viewpoint image generated based on this is fused with the depth information of the pixel depth map, so that the image quality problems of holes, overlaps, and artifacts in the output virtual viewpoint image can be reduced or eliminated, thereby improving the image quality of the output virtual viewpoint image.

[0024] Please refer to FIG1 , which is a flowchart of a method for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0025] The method for generating a virtual viewpoint image is applied to a terminal device, which may be a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, wearable device, etc.

[0026] As shown in FIG. 1 , the method for generating a virtual viewpoint image includes steps S101 to S104 .

[0027] In S101 , image features of a material image and a pixel depth map corresponding to the material image are obtained.

[0028] Exemplarily, the acquired image features are used to characterize at least the characteristics of each image element in the source image and the two-dimensional spatial relationships between the image elements. The pixel depth map is used to characterize the three-dimensional spatial relationships between the image elements in the source image. By acquiring the depth information of each pixel in the pixel depth map, the foreground and background portions of the pixel depth map can be distinguished.

[0029] After determining the material image for generating the virtual viewpoint image, feature extraction is performed on the material image using a corresponding model or neural network to obtain image features of the material image and a pixel depth map corresponding to the material image.

[0030] In some embodiments, obtaining image features of a material image and a pixel depth map corresponding to the material image includes: obtaining image features of the material image through a feature extraction network; and obtaining a pixel depth map corresponding to the material image through a depth extraction network.

[0031] Exemplarily, the feature extraction network includes, but is not limited to, a U-Net network. The feature extraction network extracts features from the source image to obtain image features. The image features include at least an initial mask matrix, convolution kernel parameters, and a feature pyramid. The image features are used to describe the color, texture, shape, and spatial relationship features of the image.

[0032] As shown in Figure 2, a depth extraction network is used to extract depth information from a source image to obtain a pixel depth map corresponding to the source image. The depth extraction network is a monocular depth estimation network, such as a MegaDepth network.

[0033] In the present disclosure, description is given by taking an example where the material image includes the first material image I0 and the second material image I1 , but the invention is not limited to the material image including only the first material image I0 and the second material image I1 .

[0034] As shown in Figure 2, after the first material image I0 and the second material image I1 are subjected to depth information extraction by the depth extraction network, a first pixel depth map D0 and a second pixel depth map D1 are obtained. The first pixel depth map D0 is extracted from the first material image I0, and the second pixel depth map D1 is extracted from the second material image I1.

[0035] As shown in Figure 3, when extracting image features from the first and second material images I0 and I1 using a feature extraction network, high-dimensional features are first extracted through the feature extraction network, and then low-dimensional features of the original image size are sampled from the high-dimensional features. As high-dimensional features are gradually extracted from the original first and second material images I0 and I1, multi-scale features of different dimensions can be extracted layer by layer. These multi-scale features constitute a feature pyramid. Simultaneously, while sampling low-dimensional features of the original image size from the high-dimensional features, convolution kernel parameter matrices W1 and W2, pixel displacement vector matrices α1, β1, α2, β2, and initial mask matrices V1 and V2 of the same size as the original first and second material images I0 and I1 are also obtained. The convolution kernel parameter matrices W1 and W2 and the pixel displacement vector matrices α1, β1, α2, β2 constitute the convolution kernel parameters.

[0036] In S102, an initial mask matrix, convolution kernel parameters, and multi-scale features corresponding to the material image are obtained according to the image features.

[0037] After obtaining the image features of the source image through the feature extraction network, the relevant information required to synthesize the virtual viewpoint image is extracted from the image features, such as the initial mask matrix, convolution kernel parameters, and multi-scale features corresponding to the source image.

[0038] In some embodiments, obtaining an initial mask matrix and convolution kernel parameters based on image features includes: obtaining the initial mask matrix using the image features based on a first extraction network; and obtaining convolution kernel parameters using the image features based on a second extraction network. Both the first extraction network and the second extraction network are convolutional neural networks, and the parameters of the convolutional neural networks corresponding to the first extraction network and the second extraction network are different.

[0039] For example, a first extraction network is used to obtain an initial mask matrix from image features, and a second extraction network is used to obtain convolution kernel parameters using the image features. Both the first extraction network and the second extraction network are convolutional neural networks, and the parameters of the convolutional neural networks corresponding to the first extraction network and the second extraction network are different.

[0040] In some embodiments, the first extraction network and the second extraction network each include a convolution layer, a pooling layer, and an activation function layer, and parameters of at least one of the convolution layer, the pooling layer, and the activation function layer of the first extraction network and the second extraction network are different.

[0041] For example, the image features are input into the first extraction network to obtain the initial mask matrix (V1, V2), and the image features are input into the second extraction network to obtain the convolution kernel parameters (W1, W2, α1, β1, α2, β2). The convolution kernel parameters include the convolution kernel parameter matrix (W1, W2) and the pixel displacement vector matrix (α1, β1, α2, β2).

[0042] In some embodiments, obtaining multi-scale features corresponding to a material image based on image features includes: extracting a feature image pyramid from the image features, the feature image pyramid including dimensional features of multiple dimensions of the material image; and obtaining multi-scale features corresponding to the material image based on the feature image pyramid.

[0043] In S103 , an image fusion mask matrix is ​​obtained according to the pixel depth map and the initial mask matrix, and a first candidate frame is generated according to the source image and the convolution kernel parameters, and a second candidate frame is generated according to the multi-size features.

[0044] Exemplarily, the initial source image and convolution kernel parameters are used to generate a first candidate frame, so that the image corresponding to the first candidate frame has image features of the source image. The first candidate frame is used to perform image fusion, so that the generated image has image features of the source image.

[0045] The multi-scale features are used to generate a second candidate frame, so that the second candidate frame image has the morphological, texture, and structural characteristics of the original image at different scales. Furthermore, the second candidate frame is used for image fusion, so that the generated image can obtain more accurate and reliable spatial information, resulting in a higher-quality output image.

[0046] The obtained pixel depth map is used to optimize the obtained initial mask matrix to obtain an image fusion mask matrix. Based on the three-dimensional information of the accompanying image in the pixel depth map, the image fusion mask matrix obtained by optimizing the initial mask matrix using the pixel depth map can better optimize the foreground and background of the synthesized image during the image fusion process, further improving the image quality of the output image.

[0047] As shown in Figures 3 and 4, the material image includes a first material image I0 and a second material image I1. After extracting the initial mask matrix (V1, V2) and the pixel depth map based on the image features corresponding to the original material image, the depth adaptive fusion module of the terminal device is used to optimize the initial mask matrix V1 to obtain the optimized mask matrix V1', and finally obtain the image fusion mask matrix (V1', V2), that is, the image fusion mask matrix includes the first fusion mask matrix V1' and the second fusion mask matrix V2.

[0048] For example, first, the pixel depth maps (D0, D1) corresponding to the first material image I0 and the second material image I1 are extracted through the depth extraction network, and then the optimized mask matrix V1' is obtained through the mask matrix optimization module. The calculation formula is as follows:

[0049] Among them, V1'(i, j) is the value of the optimized mask matrix at position (i, j), V1(i, j) is the value of the initial mask matrix at position (i, j), D0(i, j) and D1(i, j) are the values ​​of the pixel depth maps D0 and D1 at position (i, j), respectively, and i and j are both positive integers.

[0050] That is, the material image (I0, I1) includes a first material image I0 and a second material image I1, the pixel depth map (D0, D1) includes a first pixel depth map D0 corresponding to the first material image I0 and a second pixel depth map D1 corresponding to the second material image I1; the initial mask matrix (V1, V2) includes a first initial mask matrix V1 and a second initial mask matrix V2.

[0051] In some embodiments, the first fused mask matrix V1′ is obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map. For example, the first fused mask matrix V1′ is obtained by optimizing the first initial mask matrix V1 of the initial mask matrix (V1, V2) according to the first pixel depth map.

[0052] It can be understood that the fused mask matrix can also be obtained by optimizing the corresponding initial mask matrices according to the pixel depth map, that is, the fused mask matrix can be the first initial mask matrix V1 of the initial mask matrix (V1, V2) optimized according to the first pixel depth map, and the second initial mask matrix V2 of the initial mask matrix (V1, V2) optimized according to the second pixel depth map.

[0053] The fused mask matrix can also be obtained by optimizing a second initial mask matrix of the initial mask matrix according to at least one of the second pixel depth maps. For example, the first fused mask matrix V1' is obtained by optimizing a second initial mask matrix V2 of the initial mask matrix (V1, V2) according to the second pixel depth map.

[0054] In some embodiments, based on the material images including the first material image I0 and the second material image I1 , the first candidate frame generated according to the material images and convolution kernel parameters includes at least two images.

[0055] For example, using two source images (I0, I1) and convolution kernel parameters (W1, W2, α1, β1, α2, β2), the image mapping module generates the first candidate frame (I t0 , I t1 ), the calculation formula is as follows:

[0056] Using this formula, the first image I of the first candidate frame is calculated using the first source image I0 and the first set of convolution kernel parameters (W1, α1, β1). t0 , using the second material image I1 and the second set of convolution kernel parameters (W2, α2, β2), to calculate the second image I of the first candidate frame t1 .

[0057] In the formula is the pixel value of the mapped image at position (i, j); F is the size of the convolution kernel; k and l are the convolution kernel position indexes, both of which are positive integers; W k,l (i, j) is the convolution weight of the convolution kernel parameter used for the pixel value of the mapped image at the position (i, j) when the convolution kernel index is k, l; I(i+k+α k,l ,j+l+β k,l ) is the pixel value of the image at position (i, j) after mapping, and the convolution kernel parameters used are when the convolution kernel index is k, l, the pixel point is the displacement vector is α k,l , β k,l The pixel value of the original material image I, i, j, k, l are all positive integers.

[0058] In some embodiments, generating the second candidate frame based on the multi-scale features includes: generating multiple mapping images based on the multi-scale feature mapping, and synthesizing the mapping images into the second candidate frame through a synthesis network.

[0059] For example, multi-scale features are extracted from the image features extracted by the feature extraction network, image mapping is performed on the multi-scale features, and then the multi-scale second selected frame I is synthesized by a synthesis network (e.g., a GridNet network). t2 .

[0060] In S104 , the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image.

[0061] Exemplarily, after obtaining a first candidate frame, a second candidate frame, and an image fusion mask matrix, the first candidate frame and the second candidate frame are fused using the image fusion mask matrix to output a virtual viewpoint image. Because the first candidate frame has image features of the source image, the second candidate frame has relatively accurate and reliable spatial information, and the image fusion mask matrix has three-dimensional spatial information, the virtual viewpoint image generated by fusing the first and second candidate frames using the image fusion mask matrix incorporates depth information from a pixel depth map, reliable spatial information, and information from the initial source image. This effectively reduces or eliminates image quality issues such as holes, overlaps, and artifacts in the output virtual viewpoint image, thereby improving the image quality of the output virtual viewpoint image.

[0062] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix. The first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map. The first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain the virtual viewpoint image, including: fusing the first candidate frame according to the first fusion mask matrix to obtain the third candidate frame; and fusing the second candidate frame and the third candidate frame according to the second fusion mask matrix to obtain the virtual viewpoint image.

[0063] As shown in FIG3 , the first candidate frame (I t0 , I t1 ), the second candidate frame I t2 Perform fusion to obtain the final synthesized virtual viewpoint image I out , the fusion formula is as follows: I out =V⊙I n +(JV)⊙I n+1

[0064] Among them, I n , I n+1is the candidate frame to be fused, V is the image fusion mask matrix (the values ​​in the matrix are all greater than 0 and less than 1), J is the all-1 matrix, I out is the fused image, and n is a positive integer.

[0065] The image reconstruction module in Figure 3 uses the convolution kernel parameters W1, W2, α1, β1, α2, β2 and the original image I to reconstruct the image of the target viewpoint. In the image fusion process, the first fusion mask matrix V1' of the image fusion mask matrix optimized by the proposed depth adaptive fusion module and the first candidate frame (I t0 , I t1 ), to fuse and generate the third candidate frame I t3 Then, the second fusion mask matrix V2 of the image fusion mask matrix and the second candidate frame I t2 and the third candidate frame I t3 , to fuse and generate the final virtual viewpoint image I out .

[0066] Please refer to FIG5 , which is a schematic diagram of the module structure of a device for generating a virtual viewpoint image according to an embodiment of the present disclosure.

[0067] As shown in Figure 5, the device 200 for generating a virtual viewpoint image includes an image processing module 201, a feature processing module 202, a candidate image module 203, and an image generation module 204. The image processing module 201 is used to obtain the image features of the material image and the pixel depth map corresponding to the material image. The feature processing module 202 is used to obtain the initial mask matrix, convolution kernel parameters, and multi-scale features corresponding to the material image based on the image features. The candidate image module 203 is used to obtain the image fusion mask matrix based on the pixel depth map and the initial mask matrix, and to generate a first candidate frame based on the material image and the convolution kernel parameters, and to generate a second candidate frame based on the multi-scale features. The image generation module 204 is used to fuse the first candidate frame and the second candidate frame based on the image fusion mask matrix to obtain a virtual viewpoint image.

[0068] In some embodiments, obtaining image features of a material image and a pixel depth map corresponding to the material image includes: obtaining image features of the material image through a feature extraction network; and obtaining a pixel depth map corresponding to the material image through a depth extraction network.

[0069] In some embodiments, obtaining an initial mask matrix and convolution kernel parameters based on image features includes: obtaining an initial mask matrix using image features based on a first extraction network; obtaining convolution kernel parameters using image features based on a second extraction network, wherein the first extraction network and the second extraction network are both convolution neural networks, and the parameters of the convolution neural networks corresponding to the first extraction network and the second extraction network are different.

[0070] In some embodiments, obtaining multi-scale features corresponding to the material image based on image features includes: extracting a feature image pyramid from the image features, the feature image pyramid including size features of multiple dimensions of the material image; and obtaining multi-scale features corresponding to the material image based on the feature image pyramid.

[0071] In some embodiments, generating the second candidate frame based on the multi-scale features includes: generating multiple mapping images based on the multi-scale feature mapping, and synthesizing the mapping images into the second candidate frame through a synthesis network.

[0072] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and fusing the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain the virtual viewpoint image includes: fusing the first candidate frame according to the first fusion mask matrix to obtain the third candidate frame; and fusing the second candidate frame and the third candidate frame according to the second fusion mask matrix to obtain the virtual viewpoint image. In some embodiments, the source images include a first source image and a second source image, and the pixel depth map includes a first pixel depth map corresponding to the first source image and a second pixel depth map corresponding to the second source image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix; the first fusion mask matrix is ​​obtained by optimizing the first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map.

[0073] It can be understood that in some embodiments, the virtual viewpoint image generation device 200 can be applied to the terminal device 10, for example, the virtual viewpoint image generation device 200 is integrated into the terminal device 10, or is independently set relative to the terminal device 10 and communicated with the terminal device 10.

[0074] Please refer to FIG6 , which is a schematic diagram of a framework structure of a terminal device according to an embodiment of the present disclosure.

[0075] As shown in FIG6 , the terminal device 10 includes a processor 101 and a memory 102 . The processor 101 and the memory 102 are connected via a bus 103 , which is, for example, an I 2 C (Inter-Integrated Circuit) bus.

[0076] For example, the processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0077] The memory 102 may be a Flash chip, a Read-Only Memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0078] The processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, implements the following: obtaining image features of a material image and a pixel depth map corresponding to the material image; obtaining an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image based on the image features; obtaining an image fusion mask matrix based on the pixel depth map and the initial mask matrix, and generating a first candidate frame based on the material image and the convolution kernel parameters, and generating a second candidate frame based on the multi-size features; fusing the first candidate frame and the second candidate frame based on the image fusion mask matrix to obtain a virtual viewpoint image.

[0079] In some embodiments, obtaining image features of a material image and a pixel depth map corresponding to the material image includes: obtaining image features of the material image through a feature extraction network; and obtaining a pixel depth map corresponding to the material image through a depth extraction network.

[0080] In some embodiments, obtaining an initial mask matrix and convolution kernel parameters based on image features includes: obtaining an initial mask matrix based on a first extraction network using image features; obtaining convolution kernel parameters based on a second extraction network using image features, the first extraction network and the second extraction network are both convolution neural networks, and the parameters of the convolution neural network pairs corresponding to the first extraction network and the second extraction network are different.

[0081] In some embodiments, obtaining multi-scale features corresponding to the material image based on image features includes: extracting a feature image pyramid from the image features, the feature image pyramid including size features of multiple dimensions of the material image; and obtaining multi-scale features corresponding to the material image based on the feature image pyramid.

[0082] In some embodiments, generating the second candidate frame based on the multi-scale features includes: generating multiple mapping images based on the multi-scale feature mapping, and synthesizing the mapping images into the second candidate frame through a synthesis network.

[0083] In some embodiments, the image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including: fusing the first candidate frame according to the first fusion mask matrix to obtain a third candidate frame; fusing the second candidate frame and the third candidate frame according to the second fusion mask matrix to obtain a virtual viewpoint image.

[0084] In some embodiments, the material image includes a first material image and a second material image, the pixel depth map includes a first pixel depth map corresponding to the first material image and a second pixel depth map corresponding to the second material image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix; the first fusion mask matrix is ​​obtained by optimizing the first initial mask matrix of the initial mask matrix based on at least one of the first pixel depth map and the second pixel depth map.

[0085] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned corresponding virtual viewpoint image generation method embodiment, and will not be repeated here.

[0086] The embodiments of the present disclosure also provide a storage medium (e.g., a non-transitory computer storage medium) for computer-readable storage, wherein the storage medium stores at least one program, and the at least one program can be executed by at least one processor to implement a method for generating a virtual viewpoint image as provided in any embodiment of the specification of the embodiments of the present disclosure.

[0087] The storage medium may be an internal storage unit of the aforementioned terminal device 10, such as a hard disk or memory of the terminal device 10. The storage medium may also be an external storage device of the terminal device 10, such as a plug-in hard disk equipped on the terminal device 10, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0088] The embodiments of the present disclosure provide a method and apparatus for generating a virtual viewpoint image, a terminal device, and a storage medium. In the method for generating a virtual viewpoint image, the image features of a material image and a pixel depth map corresponding to the material image are obtained; an initial mask matrix, convolution kernel parameters, and multi-scale features corresponding to the material image are obtained based on the image features; an image fusion mask matrix is ​​obtained based on the pixel depth map and the initial mask matrix, and a first candidate frame is generated based on the material image and the convolution kernel parameters, and a second candidate frame is generated based on the multi-scale features; the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image. Based on the depth information of the pixel depth map fused in the virtual viewpoint image generated in this way, the image quality problems of holes, overlaps, and artifacts in the output virtual viewpoint image can be effectively reduced or eliminated, thereby improving the image quality of the output virtual viewpoint image.

[0089] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM (Random Access Memory), ROM, EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0090] It should be understood that the term "and / or" as used in this disclosure and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0091] The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for generating a virtual viewpoint image, comprising: Acquire image features of a material image and a pixel depth map corresponding to the material image; Acquire an initial mask matrix, convolution kernel parameters, and multi-size features corresponding to the material image according to the image features; Acquire an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features; The first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image.

2. The method according to claim 1, wherein: The acquiring the image features of the material image and the pixel depth map corresponding to the material image includes: Acquire image features of the material image through a feature extraction network; A pixel depth map corresponding to the material image is obtained through a depth extraction network.

3. The method according to claim 1, wherein: The obtaining the initial mask matrix and the convolution kernel parameters according to the image features includes: Acquire the initial mask matrix using the image features based on a first extraction network; The convolution kernel parameters are obtained using the image features based on a second extraction network, wherein both the first extraction network and the second extraction network are convolutional neural networks, and the parameters of the convolutional neural networks corresponding to the first extraction network and the second extraction network are different.

4. The method according to claim 1, wherein: Acquiring multi-size features corresponding to the material image according to the image features includes: Extracting a feature image pyramid from the image features, wherein the feature image pyramid includes size features of multiple dimensions of the material image; Multi-scale features corresponding to the material image are acquired according to the feature image pyramid.

5. The method according to claim 1, wherein: The generating the second candidate frame according to the multi-size feature comprises: A plurality of mapping images are generated according to the multi-scale feature maps, and the plurality of mapping images are synthesized into a second candidate frame through a synthesis network.

6. The method according to any one of claims 1 to 5, wherein: The image fusion mask matrix includes a first fusion mask matrix and a second fusion mask matrix; the first fusion mask matrix is ​​obtained by adjusting the initial mask matrix according to the pixel depth map, and the first candidate frame and the second candidate frame are fused according to the image fusion mask matrix to obtain a virtual viewpoint image, including: fusing the first candidate frames according to the first fusion mask matrix to obtain a third candidate frame; The second candidate frame and the third candidate frame are fused according to the second fusion mask matrix to obtain a virtual viewpoint image.

7. The method according to claim 6, wherein: The material image includes a first material image and a second material image, the pixel depth map includes a first pixel depth map corresponding to the first material image and a second pixel depth map corresponding to the second material image; the initial mask matrix includes a first initial mask matrix and a second initial mask matrix; The first fusion mask matrix is ​​obtained by optimizing a first initial mask matrix of the initial mask matrix according to at least one of the first pixel depth map and the second pixel depth map.

8. A device for generating a virtual viewpoint image, wherein: include: An image processing module, used to obtain image features of a material image and a pixel depth map corresponding to the material image; A feature processing module, used to obtain an initial mask matrix, convolution kernel parameters and multi-size features corresponding to the material image according to the image features; A candidate image module, configured to obtain an image fusion mask matrix according to the pixel depth map and the initial mask matrix, generate a first candidate frame according to the material image and the convolution kernel parameters, and generate a second candidate frame according to the multi-size features; An image generation module is used to fuse the first candidate frame and the second candidate frame according to the image fusion mask matrix to obtain a virtual viewpoint image.

9. A terminal device, wherein: The terminal device includes: a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the method for generating a virtual viewpoint image according to any one of claims 1-7 is realized.

10. A storage medium for computer-readable storage, wherein: The storage medium stores at least one program, and the at least one program can be executed by at least one processor to implement the method for generating a virtual viewpoint image according to any one of claims 1 to 7.

Images