Multi-viewpoint image generation method, training data acquisition method, and neural network training method

Abstract

Embodiments of the present application at least provide a method for generating a multi-viewpoint image. The method comprises: acquiring a two-dimensional image and acquiring a corresponding disparity map on the basis of the two-dimensional image; inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation, wherein the neural network comprises a compression module and an expansion module, the compression module comprises a plurality of down-sampling structures, and the expansion module comprises a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and on the basis of the three-dimensional image implicit representation, performing rendering to generate two-dimensional images and disparity maps for different viewpoints. According to the method, the neural network is used to generate the implicit representation of a three-dimensional image on the basis of the two-dimensional image and the disparity map corresponding to the two-dimensional image, and on this basis, the two-dimensional images and the disparity maps for different viewpoints are generated by means of rendering, so that a three-dimensional display device can output three-dimensional content better conforming to a real depth-of-field relationship, and thus, the method has the characteristic of flexibility and can be suitable for three-dimensional display devices with different numbers of viewpoints.

Description

Multi-view image generation method, training data acquisition method, and neural network training method TECHNICAL FIELD

[0001] The present application relates to the technical field of display, more particularly, to a method for generating multi-view images, a method for acquiring training data for a neural network, a method for training a neural network using the training data, a computing device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] The free stereoscopic display technology, also known as naked-eye stereoscopic display technology, aims to enable viewers to watch stereoscopic images without using any special glasses. The principle of the free stereoscopic display technology involves using a display screen with a specific structure designed to display multiple view images simultaneously. Each view image corresponds to the image seen by a viewer at a different position. By dividing the screen into multiple regions, each displaying a different image, the left and right eyes of the viewer can simultaneously see different images, thereby creating a stereoscopic effect. Depending on the optical design of the free stereoscopic display device, different free stereoscopic display devices can have different numbers of views, and accordingly, different numbers of virtual cameras need to be set up. For example, for a 49-view free stereoscopic display device, 49 virtual cameras need to be set up. After obtaining the images taken by the multiple virtual cameras, these images can be combined into one image. After copying or distributing the combined image to the free stereoscopic display device, the viewers can perceive the stereoscopic display effect. However, for conventional techniques for generating stereoscopic images based on two-dimensional images, there are problems such as high cost and poor applicability.

[0003] SUMMARY

[0004] The embodiments of the present application propose a method for generating multi-view images, which includes: obtaining a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image; inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation, the neural network including a compression module and an expansion module, the compression module including multiple down-sampling structures, and the expansion module including multiple up-sampling structures corresponding to the multiple down-sampling structures; and rendering generating two-dimensional images and disparity maps for different views based on the three-dimensional image implicit representation.

[0005] According to some embodiments of the present application, the compression module further comprises a plurality of two-dimensional convolution structures, the expansion module further comprises a plurality of three-dimensional convolution structures, each down-sampling structure is located between two-dimensional convolution structures of adjacent layers in the plurality of two-dimensional convolution structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures and a first three-dimensional convolution structure in the plurality of three-dimensional convolution structures, the second up-sampling structure is located between three-dimensional convolution structures of adjacent layers in the plurality of three-dimensional convolution structures, and wherein inputting the two-dimensional image and the disparity map into the neural network to obtain the three-dimensional image implicit representation comprises: providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolution structure in the plurality of two-dimensional convolution structures; fusing the first feature map and an up-sampled feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure has a same size as the first feature map; providing the fused feature map to a three-dimensional convolution structure of a corresponding layer in the plurality of three-dimensional convolution structures to output a corresponding second feature map; and obtaining the implicit representation of the three-dimensional image based on a second feature map output by a last three-dimensional convolution structure in the plurality of three-dimensional convolution structures.

[0006] According to some embodiments of the present application, providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolution structure in the plurality of two-dimensional convolution structures comprises: down-sampling a first feature map output by a front-stage two-dimensional convolution structure in two adjacent two-dimensional convolution structures in the plurality of two-dimensional convolution structures by using each down-sampling structure in the plurality of down-sampling structures to obtain a down-sampled feature map; and inputting the down-sampled feature map into a rear-stage two-dimensional convolution structure in the two adjacent two-dimensional convolution structures to output the corresponding first feature map.

[0007] According to some embodiments of the present application, the up-sampled feature map comprises a first up-sampled feature map output by the first up-sampling structure and a second up-sampled feature map output by the second up-sampling structure, and wherein the method further comprises: up-sampling a first feature map output by a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures by using a first up-sampling structure in the plurality of up-sampling structures to obtain the first up-sampled feature map; and up-sampling a second feature map output by a front-stage three-dimensional convolution structure in two adjacent three-dimensional convolution structures by using the second up-sampling structure to obtain the second up-sampled feature map.

[0008] According to some embodiments of the present application, the fusing the first feature map and the up-sampling feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map comprises: fusing the first feature map output by a preceding two-dimensional convolution structure of the last layer of two-dimensional convolution structures and the first up-sampling feature map to obtain a fused feature map provided to a first layer of three-dimensional convolution structures in the plurality of layers of three-dimensional convolution structures; and fusing the second up-sampling feature map and a first feature map output by a preceding two-dimensional convolution structure of a first down-sampling structure corresponding to the second up-sampling structure to obtain a fused feature map provided to a subsequent three-dimensional convolution structure of the second up-sampling structure.

[0009] According to some embodiments of the present application, the fusing the first feature map and the up-sampling feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map comprises: fusing the first feature map and the up-sampling feature map in a direction along a depth dimension.

[0010] According to some embodiments of the present application, the implicit representation of the three-dimensional image comprises a multi-plane image.

[0011] According to some embodiments of the present application, the different viewpoints comprise a reference viewpoint corresponding to the two-dimensional image and a target viewpoint different from the reference viewpoint, and the rendering generating two-dimensional images and disparity maps for different viewpoints based on the implicit representation of the three-dimensional image comprises: rendering generating two-dimensional images and disparity maps for the reference viewpoint and the target viewpoint based on the implicit representation of the three-dimensional image.

[0012] According to some embodiments of the present application, the obtaining the two-dimensional image and the corresponding disparity map based on the two-dimensional image comprises: generating the disparity map corresponding to the two-dimensional image based on the two-dimensional image by using a monocular depth estimation network.

[0013] Another embodiment of the present application provides a method for obtaining training data for a neural network, which can be the neural network described in any of the foregoing embodiments, and the method comprises: obtaining a plurality of reference viewpoint two-dimensional images corresponding to a reference viewpoint and a reference viewpoint disparity map corresponding to each reference viewpoint two-dimensional image in the plurality of reference viewpoint two-dimensional images; determining a viewpoint conversion matrix and a target viewpoint for each reference viewpoint two-dimensional image, wherein the viewpoint conversion matrix represents a spatial position offset of the target viewpoint relative to the reference viewpoint; determining a target viewpoint two-dimensional image corresponding to the target viewpoint based on the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image and camera intrinsic parameters; obtaining a target viewpoint disparity map corresponding to the target viewpoint two-dimensional image; and forming training data based on the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the target viewpoint two-dimensional image and the target viewpoint disparity map.

[0014] According to some embodiments of the present application, the forming training data based on the reference-view two-dimensional image, the reference-view disparity map, the target-view two-dimensional image and the target-view disparity map comprises: repairing holes in the target-view two-dimensional image and the target-view disparity map to obtain a repaired target-view two-dimensional image and a repaired target-view disparity map, and taking the reference-view two-dimensional image, the reference-view disparity map, the repaired target-view two-dimensional image and the repaired target-view disparity map as the training data.

[0015] According to some embodiments of the present application, the determining the target-view two-dimensional image corresponding to the target view according to the view conversion matrix, the reference-view disparity map, the reference-view two-dimensional image and the camera intrinsic parameters comprises: respectively determining pixel values of each pixel of the target-view two-dimensional image according to the view conversion matrix, the reference-view disparity map, pixel values of each pixel of the reference-view two-dimensional image and the camera intrinsic parameters.

[0016] According to some embodiments of the present application, the determining the view conversion matrix and the target view for each reference-view two-dimensional image comprises: randomly determining a rotation vector and a translation vector for a reference view corresponding to the reference-view two-dimensional image, the rotation vector representing an Euler angle offset of the target view relative to the reference view, and the translation vector representing a coordinate offset of the target view relative to the reference view.

[0017] Another embodiment of the present application provides a method for training a neural network using the training data obtained by the method for obtaining training data for a neural network as described in the foregoing method embodiments, which can be the neural network described in the foregoing method embodiments for generating multi-view images, the method for training the neural network comprising: providing the reference-view two-dimensional image and the reference-view disparity map to the neural network to generate an implicit representation of a three-dimensional image; rendering a reference-view target two-dimensional image and a target-view target two-dimensional image at the reference view and the target view respectively based on the implicit representation of the three-dimensional image; obtaining a reference-view target disparity map and a target-view target disparity map respectively based on the reference-view target two-dimensional image and the target-view target two-dimensional image; determining a loss function based on differences between the reference-view two-dimensional image and the reference-view target two-dimensional image, between the target-view two-dimensional image and the target-view target two-dimensional image, between the reference-view disparity map and the reference-view target disparity map, and between the target-view disparity map and the target-view target disparity map; and training the neural network using the training data until the value of the loss function is less than a threshold value.

[0018] According to some embodiments of the present application, determining the loss function based on the difference between the reference-view two-dimensional image and the reference-view target two-dimensional image, the difference between the target-view two-dimensional image and the target-view target two-dimensional image, the difference between the reference-view disparity map and the reference-view target disparity map, and the difference between the target-view disparity map and the target-view target disparity map comprises: determining a first loss sub-function, a second loss sub-function, a third loss sub-function and a fourth loss sub-function according to the difference between the reference-view two-dimensional image and the reference-view target two-dimensional image, the difference between the target-view two-dimensional image and the target-view target two-dimensional image, the difference between the reference-view disparity map and the reference-view target disparity map, and the difference between the target-view disparity map and the target-view target disparity map, respectively; and determining the sum of the first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function as the loss function.

[0019] Yet another embodiment of the present application provides an apparatus for generating multi-view images, the apparatus comprising: an image acquisition module configured to acquire a two-dimensional image and acquire a corresponding disparity map based on the two-dimensional image; a three-dimensional image implicit representation generation module configured to generate a three-dimensional image implicit representation based on the two-dimensional image and the disparity map, the three-dimensional image implicit representation generation module comprising a compression module and an expansion module, the compression module comprising a plurality of down-sampling structures, and the expansion module comprising a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and a rendering module configured to render two-dimensional images and disparity maps for different views based on the three-dimensional image implicit representation.

[0020] According to some embodiments of the present application, the compression module further comprises a plurality of two-dimensional convolution structures, the expansion module further comprises a plurality of three-dimensional convolution structures, each down-sampling structure is located between two-dimensional convolution structures of adjacent layers in the plurality of two-dimensional convolution structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures and a first three-dimensional convolution structure in the plurality of three-dimensional convolution structures, the second up-sampling structure is located between three-dimensional convolution structures of adjacent layers in the plurality of three-dimensional convolution structures, wherein the compression module is configured to receive the two-dimensional image and the disparity map to output a corresponding first feature map from each two-dimensional convolution structure in the plurality of two-dimensional convolution structures, the three-dimensional image implicit representation generation module further comprises a feature map fusion module, the feature map fusion module is configured to fuse the first feature map with an up-sampled feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure has a same size as the first feature map, wherein a three-dimensional convolution structure of a corresponding layer in the plurality of three-dimensional convolution structures receives the fused feature map to output a corresponding second feature map, the expansion module obtains the implicit representation of the three-dimensional image based on a second feature map output by a last three-dimensional convolution structure in the plurality of three-dimensional convolution structures.

[0021] Yet another embodiment of the present application provides a computing device comprising: a memory configured to store computer-executable instructions; and a processor configured to perform a method as recited in any of the method embodiments of the preceding when the computer-executable instructions are executed by the processor.

[0022] Yet another embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions that, when executed, implement a method as recited in any of the method embodiments of the preceding.

[0023] Yet another embodiment of the present application provides a computer program product comprising a computer program that, when executed by a processor, implements a method as recited in any of the method embodiments of the preceding.

[0024] Compared with conventional techniques of converting two-dimensional images into three-dimensional images, the method and apparatus for generating multi-viewpoint images provided by embodiments of the present application utilize a neural network to generate an implicit representation of a three-dimensional image based on a two-dimensional image and a disparity map corresponding to the two-dimensional image, and on this basis generate two-dimensional images and disparity maps for different viewpoints through rendering, so that a three-dimensional display device can output three-dimensional content that is more consistent with real depth-of-field relationships, and promote consistency of generation results for different viewpoints, while also having the characteristics of flexibility, and can be applied to three-dimensional display devices with different numbers of viewpoints. These and other advantages of the application will become apparent from the embodiments described below, and the embodiments described below are used to clarify these and other advantages of the application. BRIEF DESCRIPTION OF DRAWINGS

[0025] Embodiments of the present application will now be described in more detail and with reference to the drawings, in which:

[0026] Figure 1 illustrates some steps in a method for generating multi-viewpoint images according to embodiments of the present application;

[0027] Figure 2 is used to illustrate the calculation of disparity;

[0028] Figure 3 is used to illustrate zero disparity, positive disparity and negative disparity;

[0029] Figure 4 schematically illustrates an example of a reference viewpoint and a target viewpoint different from the reference viewpoint;

[0030] Figure 5 schematically illustrates some steps included in inputting a two- dimensional image and a disparity map into a neural network to obtain an implicit representation of a three-dimensional image according to embodiments of the present application;

[0031] Figure 6 illustrates some steps in a method for generating multi-viewpoint images according to another embodiment of the present application;

[0032] Figure 7 schematically illustrates an example of a model of a neural network including a compression module and an expansion module;

[0033] Figure 8 is used to illustrate a multi-plane image;

[0034] Figure 9 illustrates some steps included in an embodiment of a method for obtaining training data for a neural network,

[0035] Figure 10a schematically illustrates an example of a reference viewpoint two- dimensional image, a reference viewpoint disparity map, a target viewpoint two-dimensional image and a target viewpoint disparity map;

[0036] Figure 10b schematically illustrates a comparison between a target viewpoint two- dimensional image and a target viewpoint disparity map before and after repair;

[0037] FIG. 11 illustrates some steps included in a method of training a neural network provided according to an embodiment of the present application;

[0038] FIG. 12 is used to schematically illustrate training a model of the neural network shown in FIG. 7;

[0039] FIG. 13 illustrates an apparatus for generating multi-view images;

[0040] FIG. 14 illustrates an example system including an example computing device representative of one or more systems and / or devices that can implement various methods or apparatuses described herein. DETAILED DESCRIPTION

[0041] The following description provides specific details for a thorough understanding of, and enabling disclosure of, various embodiments of the application. In some instances, well-known structures or functions can not be described in detail to avoid obscuring the description of the various embodiments of the application. The techniques of the present application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0042] In the following description, numerous specific details are given to provide a thorough understanding of the application. However, it will be apparent that the application can be practiced without one or more of these specific details. In some instances, well-known structures and functions have not been described in detail to avoid obscuring the description of the application. The detailed description is set forth with specific reference to the drawings. A neural network is a computational model that simulates the human nervous system, which learns and trains on a large amount of data to achieve prediction and classification of unknown data. The basic unit of a neural network is a neuron, each neuron receives input signals from other neurons and processes and outputs according to certain rules. These neurons are connected to each other through connection weights, forming a complex network structure. In the training phase, the neural network receives a large amount of labeled data and optimizes the performance of the network by continuously adjusting the connection weights of the neurons. This learning and prediction process is the core function of the neural network. When the neural network receives new data, it will make predictions and classifications based on the knowledge it has learned.

[0043] Embodiments of the present application provide a method for generating multi-view images. FIG. 1 illustrates some steps included in an embodiment of the method for generating multi-view images. As shown in FIG. 1, the method for generating multi-view images can include the following steps: S101, acquiring a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image; S102, inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation, the neural network including a compression module and an expansion module, the compression module including a plurality of down-sampling structures, and the expansion module including a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; S103, rendering two-dimensional images and disparity maps for different viewpoints based on the three-dimensional image implicit representation.

[0044] Three-dimensional representation is a key technology in machine vision, which can intuitively reflect the shape of an object. Compared with two-dimensional representation, three-dimensional representation has depth information, so effective three-dimensional representation is an important key to realize machine vision tasks such as three-dimensional model reconstruction, three-dimensional target detection, scene semantic segmentation, and has wide application prospects in the fields of robots, AR / VR, human-computer interaction, remote sensing mapping, etc. The representation form of three-dimensional space can include display representation and implicit representation. Common display representations include voxels, point clouds, triangular patches (meshes), etc. The display representation is characterized in that the description of a scene is displayed, and the three-dimensional representation of the scene can be directly seen, for example, the corresponding scene can be directly visualized based on triangular patches. Common implicit representations include signed distance function, occupancy field, neural radiance field (NeRF), multi-plane image (MPI), etc.

[0045] In a spatial scene, the image points formed by the same object point in left and right viewpoint images are called homologous image points, and the position difference of homologous image points is defined as disparity. The difference of homologous image points in the vertical direction is called vertical disparity, and the difference in the horizontal direction is called horizontal disparity. Horizontal disparity is the main factor for realizing three-dimensional display, while vertical disparity has no actual effect on three-dimensional display and is easy to cause fatigue of the viewer. The disparity referred to in the embodiments of the present application refers to horizontal disparity. The disparity is further described below taking two viewpoints as an example. As shown in FIG. 2, P represents an object point in a three-dimensional scene, the object point P forms homologous image points p u , p' u in left and right viewpoint images, O1 and O2 represent the eyes of a person, z represents the vertical distance from the eyes of the person to the object point, f represents the vertical distance from the eyes of the person to the viewpoint images u and u', and B represents the distance between the eyes of the person, then the disparity d is

[0046] It can be seen that the closer the object point p is to the human eye, the greater the parallax d is, and the distance of the object point to the human eye can be determined by the size of the parallax. The horizontal parallax can be divided into zero parallax, positive parallax and negative parallax. The following will schematically illustrate the zero parallax, positive parallax and negative parallax by means of FIG. 3. As shown in FIG. 3, point O L represents the position of the imaged point of the object point in the left view point image (i.e. the display plane), point O R represents the position of the imaged point of the object point in the right view point image (i.e. the display plane). When point O L coincides with point O R , it is zero parallax, in the stereoscopic display scene, the viewer will perceive that the reproduced object point is located on the display plane; when point O R is located to the right of point O L , it is positive parallax, in the stereoscopic display scene, the viewer will perceive that the reproduced object point is located behind the display plane, i.e. "into the screen"; when point O R is located to the left of point O L , it is negative parallax, in the stereoscopic display scene, the viewer will perceive that the reproduced object point is located in front of the display plane, i.e. "out of the screen".

[0047] As mentioned above, the number of view points can be different according to the optical design of the stereoscopic display device, each view point can be understood as corresponding to a virtual camera, FIG. 4 schematically shows view point V0, view point V1, view point V2, view point V3 and view point V4, each view point can be respectively regarded as corresponding to a different virtual camera, the shooting directions of the virtual cameras can converge at point Q, by adjusting the convergence position of the shooting directions of the virtual cameras, it can be determined which part of the three-dimensional display is out of the screen and which part is into the screen.

[0048] In the foregoing step S101 of obtaining a two-dimensional image and obtaining a corresponding parallax map based on the two-dimensional image, a monocular depth estimation network can be used to estimate the parallax map of the two-dimensional image. Depth estimation is a basic problem in the field of computer vision, which is to obtain the distance information of each point in the scene in the image to the camera. The image composed of such distance information can be referred to as a depth map, which can be applied in the fields of robot navigation, augmented reality, three-dimensional reconstruction, autonomous driving, etc. Monocular depth estimation is to estimate the distance of each pixel in the image relative to the source of shooting by using one or only one RGB image under a certain perspective. For the human eye, a large amount of depth information can be extracted from the image information obtained by one eye due to a large amount of prior knowledge. The specific algorithm of the monocular depth estimation network that can be used is not limited in the present application, and monocular depth estimation based on clues, monocular depth estimation based on machine learning, monocular depth estimation based on deep learning, etc. can be used. For example, the parallax data generation method disclosed in patent CN112991419A can be used to generate a parallax map. A monocular depth estimation network can be pre-trained, and then the trained monocular depth estimation network is used to generate parallax data corresponding to the parallax map based on the two-dimensional image. For example, a depth network can be used as a monocular depth estimation network, and the trained depth network generates a corresponding parallax map based on a two-dimensional image.

[0049] In step S102, the two-dimensional image and the parallax map corresponding to the two-dimensional image are input into a neural network, the neural network includes a compression module and an expansion module, the compression module includes a plurality of down-sampling structures, the expansion module includes a plurality of up-sampling structures corresponding to the plurality of down-sampling structures, that is, the number of layers of the up-sampling structure is the same as the number of the up-sampling structure, so that the implicit representation of the three-dimensional image can be generated based on the combination of the output information of the compression module and the output information of the expansion module. On the basis of obtaining the implicit representation of the three-dimensional image, in step S103, two-dimensional images and parallax maps for different viewpoints are generated by rendering.

[0050] Compared with conventional technologies of converting two-dimensional images into three-dimensional images, the method for generating multi-viewpoint images provided in the embodiments of the present application generates an implicit representation of a three-dimensional image based on a two-dimensional image and a parallax map corresponding to the two-dimensional image by using a neural network, and generates two-dimensional images and parallax maps for different viewpoints by rendering on this basis. Therefore, the three-dimensional display device can output three-dimensional content that is more consistent with the real depth relationship, promotes the consistency of the generation results of different viewpoints, and has the characteristics of flexibility, and can be applied to three-dimensional display devices with different numbers of viewpoints.

[0051] According to some embodiments of the present application, the compression module of the neural network further comprises a plurality of two-dimensional convolution structures, the expansion module further comprises a plurality of three-dimensional convolution structures, each down-sampling structure is located between two-dimensional convolution structures of adjacent layers in the plurality of two-dimensional convolution structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures and a first three-dimensional convolution structure in the plurality of three-dimensional convolution structures, and the second up-sampling structure is located between three-dimensional convolution structures of adjacent layers in the plurality of three-dimensional convolution structures. That is, the two-dimensional convolution structures form a cascaded plurality of two-dimensional convolution structures via the plurality of down-sampling structures, the three-dimensional convolution structures form a cascaded plurality of three-dimensional convolution structures via the plurality of second up-sampling structures, and the last two-dimensional convolution structure in the plurality of two-dimensional convolution structures and the first three-dimensional convolution structure in the plurality of three-dimensional convolution structures are connected via the first up-sampling structure. The difference between convolutions of different dimensions mainly lies in different degrees of freedom of convolution kernels. For example, two-dimensional convolution can perform sliding window operation in the width and height directions, and three-dimensional convolution can perform sliding window operation in the width, height and channel directions. That is, two-dimensional convolution is to perform sliding window operation on a single-channel image, and the input data is a two-dimensional matrix. The input of three-dimensional convolution is a three-dimensional matrix, which increases the channel dimension (depth) compared with two-dimensional convolution. In three-dimensional convolution, the filter can move in three directions (width, height and channel of the image). Because the filter slides through a three-dimensional space, the output value is also a three-dimensional data. Three-dimensional convolution is widely used in video classification, image segmentation and other scenarios. After receiving the two-dimensional image and the corresponding disparity map, the compression module can output corresponding first feature maps from the two-dimensional convolution structures, and the down-sampling structure can comprise a pooling layer to reduce the height and width of the first feature maps output by the two-dimensional convolution structures. The first up-sampling structure or the second up-sampling structure can comprise a deconvolution layer, a bilinear interpolation layer or a de-pooling layer, which can realize the enlargement of the second feature maps output by the three-dimensional convolution structure and the first feature maps output by the last two-dimensional convolution structure in the plurality of two-dimensional convolution structures.As shown in FIG. 5, in some embodiments, the step S102 of inputting the two-dimensional image and the disparity map into the neural network to obtain the three-dimensional image implicit representation can include the following steps: S501, providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolution structure in the multi-layer two-dimensional convolution structure; S502, fusing the first feature map and an up-sampling feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampling feature map output by the corresponding up-sampling structure having the same size as the first feature map; S503, providing the fused feature map to a corresponding layer of the three-dimensional convolution structure to output a corresponding second feature map; and S504, obtaining the implicit representation of the three-dimensional image based on the second feature map output by the last layer of the multi-layer three-dimensional convolution structure.

[0052] Each two-dimensional convolution structure in the compression module forms a cascaded multi-layer two-dimensional convolution structure via a plurality of down-sampling structures, each three-dimensional convolution structure in the expansion module forms a cascaded multi-layer three-dimensional convolution structure via a plurality of second up-sampling structures, the first feature map output by each two-dimensional convolution structure is fused with the up-sampling feature map output by the corresponding up-sampling structure between each two-dimensional convolution structure to generate a fused feature map, the three-dimensional convolution structure of the corresponding layer in the expansion module outputs a corresponding second feature map based on the fused feature map, and the second feature map output by the last layer of the multi-layer three-dimensional convolution structure can be used as the implicit representation of the three-dimensional image. The fused feature map can be formed by stacking the first feature map and the up-sampling feature map in a depth direction different from the width and height directions of the feature map, so that each three-dimensional convolution structure can receive a corresponding different fused feature map. Based on the fused feature map and the processing of the three-dimensional convolution structure on the fused feature map, the quality of the generated implicit representation of the three-dimensional image can be improved, and it is more conducive for the three-dimensional display device to output three-dimensional content that is more consistent with the real depth relationship.

[0053] In some embodiments, the step S501 of providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolution structure in the multi-layer two-dimensional convolution structure includes: down-sampling the first feature map output by a front-stage two-dimensional convolution structure in two adjacent two-dimensional convolution structures using each down-sampling structure in the plurality of down-sampling structures to obtain a down-sampled feature map; and inputting the down-sampled feature map into a rear-stage two-dimensional convolution structure in the two adjacent two-dimensional convolution structures to output the corresponding first feature map.

[0054] In some embodiments, the up-sampled feature maps include a first up-sampled feature map output by the first up-sampling structure and a second up-sampled feature map output by the second up-sampling structure, and the step of inputting the two-dimensional image and the disparity map into the neural network to obtain the implicit representation of the three-dimensional image can further include: up-sampling, by the first up-sampling structure of the plurality of up-sampling structures, a first feature map output by a last two-dimensional convolutional structure of the plurality of two-dimensional convolutional structures to obtain the first up-sampled feature map; and up-sampling, by the second up-sampling structure, a second feature map output by a front-stage three-dimensional convolutional structure of adjacent two three-dimensional convolutional structures to obtain the second up-sampled feature map.

[0055] Therefore, in some embodiments, the step of inputting the two-dimensional image and the disparity map into the neural network to obtain the implicit representation of the three-dimensional image can include the following steps, as shown in FIG. 6: S601, down-sampling, by each down-sampling structure of the plurality of down-sampling structures, a first feature map output by a front-stage two-dimensional convolutional structure of adjacent two two-dimensional convolutional structures to obtain a down-sampled feature map; S602, inputting the down-sampled feature map into a rear-stage two-dimensional convolutional structure of the adjacent two two-dimensional convolutional structures to output the corresponding first feature map; S603, up-sampling, by a first up-sampling structure of the plurality of up-sampling structures, a first feature map output by a last two-dimensional convolutional structure of the plurality of two-dimensional convolutional structures to obtain a first up-sampled feature map; S604, up-sampling, by a second up-sampling structure, a second feature map output by a front-stage three-dimensional convolutional structure of adjacent two three-dimensional convolutional structures to obtain a second up-sampled feature map; S605, fusing the first feature map and an up-sampled feature map output by a corresponding up-sampling structure of the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure having the same size as the first feature map; S606, providing the fused feature map to a corresponding three-dimensional convolutional structure to output a corresponding second feature map; and S607, obtaining the implicit representation of the three-dimensional image based on a second feature map output by a last three-dimensional convolutional structure of the plurality of three-dimensional convolutional structures. In some embodiments, the last three-dimensional convolutional structure outputs a plurality of discrete planar images, i.e., a multi-planar image MPI, as the implicit representation of the three-dimensional image.

[0056] In some embodiments, the fusing the first feature map with the up-sampled feature map output by the corresponding up-sampling structure of the plurality of up-sampling structures to obtain a fused feature map can include the following cases: fusing the first feature map output by the preceding two-dimensional convolution structure of the last two-dimensional convolution structure with the first up-sampled feature map to obtain a fused feature map provided to the first three-dimensional convolution structure of the plurality of three-dimensional convolution structures; and fusing the second up-sampled feature map with the first feature map output by the preceding two-dimensional convolution structure of the first down-sampling structure corresponding to the second up-sampling structure to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second up-sampling structure. That is, the form of the feature map provided to the first up-sampling structure of the plurality of up-sampling structures can be different from the form of the feature map provided to the second up-sampling structure. The first up-sampling structure outputs the first up-sampled feature map based on the first feature map output by the last two-dimensional convolution structure of the plurality of two-dimensional convolution structures, while each second up-sampling structure outputs the corresponding second up-sampled feature map based on the second feature map output by the preceding three-dimensional convolution structure adjacent to the up-sampling structure.

[0057] In some embodiments, the fusing the first feature map with the up-sampled feature map output by the corresponding up-sampling structure of the plurality of up-sampling structures to obtain a fused feature map includes: superimposing the first feature map and the up-sampled feature map in a direction along a depth dimension. The direction along the depth dimension mentioned here refers to another dimension direction different from the width and height. For example, assuming that the first feature map and the up-sampled feature map are both two-dimensional matrices of a x b, the fused feature map obtained based on the first feature map and the up-sampled feature map can be two two-dimensional matrices superimposed on each other, thereby obtaining a three-dimensional matrix of a x b x 2, that is, the width and height of the fused feature map are still a and b respectively, but the depth dimension becomes 2. Similarly, in the case where the first feature map and the up-sampled feature map are both three-dimensional matrices, the fused feature map obtained based on the first feature map and the up-sampled feature map can be two three-dimensional matrices superimposed on each other in another depth direction different from the length, width and height of the three-dimensional matrix. The fused feature map increases one dimension compared to the first feature map or the up-sampled feature map.

[0058] Next, referring to FIG. 7, a model of the neural network in the embodiment of the present application is schematically illustrated. As shown in FIG. 7, the neural network including the multi-layer two-dimensional convolution structure, the multi-layer three-dimensional convolution structure, the multiple up-sampling structures and the multiple down-sampling structures can form a convolutional neural network approximately in a U shape. The input of the convolutional neural network is a two-dimensional image (for example, an RGB image) and a disparity map corresponding to the two-dimensional image, and the output of the convolutional neural network is a three-dimensional image implicit representation, for example, a multi-plane image (MPI). The resolution of the two-dimensional image provided to the convolutional neural network is h x w, and the output is a set of discrete planes. In the example of FIG. 7, the blue arrow represents the two-dimensional convolution structure, the yellow arrow represents the three-dimensional convolution structure, the red arrow represents the down-sampling structure, the green arrow represents the up-sampling structure, and the gray arrow represents the fusion operation. Four layers of two-dimensional convolution structures, three layers of three-dimensional convolution structures, three up-sampling structures and three down-sampling structures are schematically shown in FIG. 7. As shown in FIG. 7, each layer of two-dimensional convolution structure can include two two-dimensional convolutions, each layer of three-dimensional convolution structure can include two three-dimensional convolutions, the initial data provided to the convolutional neural network includes 4 channels, the first two-dimensional convolution in each layer of two-dimensional convolution structure increases the data of the channel, and the second two-dimensional convolution does not change the size of the feature map output by the first two-dimensional convolution. In the example of FIG. 7, the compression module of the neural network includes 3 down-sampling structures, and correspondingly, the expansion module includes 3 up-sampling structures, so that the resolution of the feature map output by the expansion module is the same as the resolution of the input image provided to the compression module. The compression module further includes 4 layers of two-dimensional convolution structures, and each down-sampling structure is located between the two-dimensional convolution structures of adjacent layers in the 4 layers of two-dimensional convolution structures. Each down-sampling structure down-samples a first feature map output by a previous two-dimensional convolution structure in the two-dimensional convolution structures of the adjacent two layers to obtain a down-sampled feature map, and the down-sampled feature map is input to a subsequent two-dimensional convolution structure in the two-dimensional convolution structures of the adjacent two layers to output a corresponding first feature map. The expansion module further includes 3 layers of three-dimensional convolution structures, the 3 up-sampling structures described above include 1 first up-sampling structure and 2 second up-sampling structures, the first up-sampling structure is located between the last two-dimensional convolution structure in the 4 layers of two-dimensional convolution structures and the first three-dimensional convolution structure in the 3 layers of three-dimensional convolution structures, and each second up-sampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure. The first up-sampling structure up-samples a first feature map output by the last two-dimensional convolution structure in the 4 layers of two-dimensional convolution structures to obtain a first up-sampled feature map, and each second up-sampling structure up-samples a second feature map output by a previous three-dimensional convolution structure in the three-dimensional convolution structures of the adjacent two layers to obtain a second up-sampled feature map.Referring to FIG. 7, the first feature map output by the preceding two-dimensional convolution structure of the last two-dimensional convolution structure in the four-layer two-dimensional convolution structure (i.e., the third two-dimensional convolution structure in FIG. 7) is fused with the first up-sampling feature map output by the first up-sampling structure (as shown by the lowermost green arrow in FIG. 7) to obtain a fused feature map provided to the first three-dimensional convolution structure in the three-layer three-dimensional convolution structure. The second up-sampling feature map output by the second up-sampling structure (e.g., the middle green arrow in FIG. 7) is fused with the first feature map output by the preceding two-dimensional convolution structure of the first down-sampling structure (as shown by the middle red arrow in FIG. 7) corresponding to the second up-sampling structure (as shown by the middle gray arrow in FIG. 7) to obtain a fused feature map provided to the following three-dimensional convolution structure (i.e., the second three-dimensional convolution structure in FIG. 7) of the second up-sampling structure. Referring to FIG. 7, each down-sampling structure can reduce the size of the first feature map output by the preceding two-dimensional convolution structure by half, and thus the size or resolution of the down-sampled feature map provided to the last two-dimensional convolution structure (i.e., the bottommost two-dimensional convolution structure in FIG. 7) is one-eighth of the input two-dimensional image. In the expansion module, the fused feature map received by the three-dimensional convolution structure has an additional dimension in the depth direction compared to the first feature map output by the two-dimensional convolution structure, and thus the feature maps in the expansion module shown in the right side of FIG. 7 are illustrated as three-dimensional structures. For the last two-dimensional convolution structure, the first feature map output therefrom is not fused with other feature maps, and thus the first feature map still does not have a dimension in the depth direction, or the first feature map can be considered to have a dimension of 1 in the depth direction. As shown in FIG. 7, the first feature map output by the first two-dimensional convolution structure includes 64 channels and has a resolution of h x w. After down-sampling by the first down-sampling structure, the resolution of the first feature map is halved, i.e.,. Accordingly, the resolution of the first feature map output by the second two-dimensional convolution structure is The number of channels is increased to 128. After down-sampling by the three down-sampling structures, the resolution of the first feature map output by the last two-dimensional convolution structure is However, after two-dimensional convolution operations performed by the multiple two-dimensional convolution structures, the first feature map output by the last two-dimensional convolution structure includes 512 channels. The first feature map output by each two-dimensional convolution structure does not have a dimension in the depth direction, or the first feature map output by each two-dimensional convolution structure has a dimension of 1 in the depth direction. To more clearly illustrate this point, as shown in FIG. 7, the first feature map output by the last two-dimensional convolution structure is identified as “512 x 1”. The first up-sampling structure in the expansion module receives the first feature map output by the last two-dimensional convolution structure, and expands the size of the first feature map by a factor of two and reduces the number of channels by half through up-sampling (e.g., deconvolution) to obtain a feature map having a channel of 256 and a size or resolution of The first upsampled feature map, which has the same size as the first feature map output by the preceding two-dimensional convolutional structure of the last two-dimensional convolutional structure, is fused to obtain a fused feature map with 256 channels and a depth dimension of 2, which is labeled "256×2" in Figure 7. Similarly, the second upsampled structure can double the size of the second feature map output by the first three-dimensional convolutional structure through deconvolution, reducing the number of channels by half, thereby obtaining a feature map with 128 channels and a size or resolution of [missing information]. The upsampled feature map, which has the same size as the first feature map output by the second 2D convolutional structure, is fused to obtain a fused feature map with 128 channels and a depth dimension of 3, as labeled "128×3" in Figure 7. For the last 3D convolutional structure, the fused feature map it receives has the same size as the 2D image and corresponding disparity map provided to the neural network, with a depth dimension of 4, and is labeled "64×4" in Figure 7. The number of channels in the feature map output by the last 3D convolutional structure can be set to the number of planar images N in the multi-plane image (MPI) output by the convolutional neural network. In other words, the number of planar images in the multi-plane image (MPI) output by the convolutional neural network can be set as needed, and this embodiment does not limit this. Furthermore, Figure 7 only provides an example of a neural network model. The number of layers in the two-dimensional convolutional structure, the number of layers in the three-dimensional convolutional structure, the number of two-dimensional convolutions within each two-dimensional convolutional structure, and the number of three-dimensional convolutions within each three-dimensional convolutional structure are merely illustrative and do not constitute a limitation on the scope of protection of this application. Upsampling structures (e.g., deconvolution) can expand the feature map, but cannot completely restore the feature map. Through the above-described fusion operation, the features processed by the downsampling structure corresponding to the upsampling structure can be superimposed, reducing data loss. In addition, this fusion operation can combine deep and shallow information. Deep information includes low-resolution information after multiple downsampling of the input image by multiple downsampling structures, while shallow information includes high-resolution information passed from the compression module to the expansion module through the fusion operation, providing more refined features.

[0059] In some embodiments, the implicit representation of the three-dimensional image comprises a multi-plane image (MPI). A multi-plane image represents a scene using multiple RGBA planes in a camera frustum, supporting differentiable rendering. FIG. 8 is used to schematically illustrate a multi-plane image. A multi-plane image comprises multiple RGBA images respectively located at multiple parallel planes, which are at different layers, also referred to as multi-layered RGBA images, and each layer of the RGBA images is at a different disparity, which can be uniformly spaced in space. For each layer of the RGBA images, each layer of the planes can encode two parameters: a color C of an RGB image and an opacity a. That is, each layer of the RGBA images can be represented as (Ci, ai), where Ci and ai represent the color channel and the a channel of the i-th layer of the RGBA images, respectively. A multi-plane image is a collection of RGBA images. In computer graphics, a true color graphic of an RGB color model is composed of three color information channels of red, green and blue, and to realize the transparency effect of an image, another channel information is added in the processing and storage of the image. This additional channel information representing the transparency of each pixel point in the image is called an a channel. For example, using an 8-bit binary number for the a channel, 256 levels of gray, i.e., 256 levels of transparency, can be represented. Each layer of the RGBA images in the MPI is fixed at a certain disparity, and the a channel of each layer of the RGBA images represents its transparency or visibility. In FIG. 8, examples of a reference viewpoint two-dimensional image at a reference viewpoint and a target viewpoint two-dimensional image at a target viewpoint are also schematically shown. Based on the provision of the multi-plane image (MPI), the reference viewpoint and the target viewpoint, the reference viewpoint two-dimensional image and the target viewpoint two-dimensional image described above can be generated by rendering.

[0060] As mentioned before, each viewpoint can be understood as corresponding to a virtual camera, and accordingly, the two-dimensional image provided to the neural network can be considered as being taken by a virtual camera corresponding to the reference viewpoint, i.e., the two-dimensional image provided to the neural network can correspond to the reference viewpoint, and the different viewpoints described before can comprise the reference viewpoint corresponding to the two-dimensional image and other target viewpoints different from the reference viewpoint. Therefore, the reference viewpoint two-dimensional image (RGB image) and the corresponding disparity map and the target viewpoint two-dimensional image (RGB image) and the corresponding disparity map can be generated by rendering based on the obtained implicit representation of the three-dimensional image.

[0061] Different algorithms in the art can be employed to render a two-dimensional image of a corresponding viewpoint based on a multi-plane image (MPI). The main procedure of rendering is actually to calculate the RGB value of each pixel on the imaging plane of a virtual camera corresponding to the target viewpoint. For example, for any pixel X on the imaging plane of the virtual camera, the calculation method of its RGB value can be as follows: a straight line is formed by connecting the optical center of the virtual camera and the pixel X, the straight line passes through each layer of RGBA image in the multi-plane image, and the intersection of the straight line and each layer of RGBA image is determined, thus each intersection will correspond to a color channel Ci of a RGBA image, i.e. the RGB value. Assuming that the multi-plane image includes N layers of RGBA image, the above straight line will form N intersections with the multi-plane image, and the RGB value of the pixel X is the weighted sum of the RGB values of the N intersections, and different weights can be set for the RGB values of different intersections. For example, assuming that the RGB value of each intersection is Y i , then the RGB value of the pixel X is , where i represents the weight coefficient of the RGB value of each intersection. On the basis of obtaining the two-dimensional image of the corresponding viewpoint, a monocular depth estimation network can be used to estimate the disparity map of the two-dimensional image.

[0062] The neural network mentioned in the above embodiments of the present application can be obtained through training of training data. Another embodiment of the present application provides a method for obtaining training data for a neural network, which includes but is not limited to the convolutional neural network as described in the above embodiments. Figure 9 illustrates some steps included in an embodiment of the method for obtaining training data for a neural network. As shown in Figure 9, the method for obtaining training data for a neural network can include the following steps: step S901, obtaining a plurality of reference-view two-dimensional images corresponding to a reference view and a reference-view disparity map corresponding to each of the plurality of reference-view two-dimensional images. The plurality of reference-view two-dimensional images can be regarded as a plurality of two-dimensional images taken by a virtual camera at a reference view. Similarly, a monocular depth estimation network can be employed to estimate a reference-view disparity map corresponding to each of the plurality of reference-view two-dimensional images, respectively. The disparity map is a single-channel image, and the resolution of the disparity map can be the same as that of the corresponding two-dimensional image. The value of each pixel point on the disparity map represents the disparity at the pixel. Step S902, determining a view conversion matrix and a target view for each reference-view two-dimensional image, the view conversion matrix representing the spatial positional offset of the target view relative to the reference view; step S903, determining a target-view two-dimensional image corresponding to the target view according to the view conversion matrix, the reference-view disparity map, the reference-view two-dimensional image, and the camera intrinsic parameter; step S904, obtaining a target-view disparity map corresponding to the target-view two-dimensional image; similarly, a monocular depth estimation network can be employed to estimate the target-view disparity map corresponding to the target-view two-dimensional image. And step S905, forming training data based on the reference-view two-dimensional image, the reference-view disparity map, the target-view two-dimensional image, and the target-view disparity map. That is, for each reference-view two-dimensional image, a corresponding set of training data can be obtained through the above steps, which includes the reference-view two-dimensional image, the reference-view disparity map corresponding to the reference-view two-dimensional image, the target-view two-dimensional image, and the target-view disparity map corresponding to the target-view two-dimensional image. Based on the plurality of reference-view two-dimensional images, a plurality of similar sets of training data can be obtained, thereby enabling effective training of the neural network.

[0063] In some cases, the target view 2D image can have contents that are not visible in the reference view 2D image, i.e., the target view 2D image can have holes, and correspondingly, the target view disparity map can also have holes. Therefore, in some embodiments, the step S905 described above can include repairing the holes in the target view 2D image and the target view disparity map to obtain a repaired target view 2D image and a repaired target view disparity map, and using the reference view 2D image, the reference view disparity map, the repaired target view 2D image and the repaired target view disparity map as training data. In this way, the sample quality of the training data can be improved. The specific method of hole repair is not limited herein, and a hole repair method based on deep learning can be used, or a non-deep learning method such as the PatchMatch algorithm can be used. The PatchMatch algorithm is proposed for image editing, which fills the boundary area by finding the most matching other part to the edge part, so as to achieve a natural effect. The core purpose of the algorithm is to quickly find the corresponding small area between two pictures. In application, the patchmatch algorithm can be combined with image reorganization and other technologies to realize functions such as image repair, picture fusion, watermark removal, etc.

[0064] Similar to the previous embodiments, in the step S901 described above, the reference view disparity map can be generated based on the reference view 2D image using a monocular depth estimation network. In some embodiments, the step S902 described above - determining a view conversion matrix and a target view for each reference view 2D image can include randomly determining a rotation vector and a translation vector for the reference view corresponding to the reference view 2D image, the rotation vector representing the Euler angle offset of the target view relative to the reference view, and the translation vector representing the coordinate offset of the target view relative to the reference view. The step S903 described above - determining a target view 2D image corresponding to the target view according to the view conversion matrix, the reference view disparity map, the reference view 2D image and the camera intrinsic parameter can include determining the pixel value of each pixel of the target view 2D image according to the view conversion matrix, the reference view disparity map, the pixel value of each pixel of the reference view 2D image and the camera intrinsic parameter, respectively.

[0065] It can be understood that for each reference view two-dimensional image in the plurality of reference view two-dimensional images, different view conversion matrices and corresponding plurality of target views can be determined respectively, at this time, the method for obtaining training data for the neural network can comprise: determining a plurality of target view two-dimensional images corresponding to the plurality of target views respectively according to each reference view two-dimensional image, the camera internal parameter, the view conversion matrix corresponding to each reference view two-dimensional image, and the reference view disparity map; obtaining a plurality of target view disparity maps corresponding to the plurality of target view two-dimensional images; and forming a plurality of groups of training data based on the plurality of reference view two-dimensional images, the plurality of reference view disparity maps, the plurality of target view two-dimensional images and the plurality of target view disparity maps, each group of training data comprising one reference view two-dimensional image in the plurality of reference view two-dimensional images, and the corresponding reference view disparity map, target view two-dimensional image and target view disparity map.

[0066] The steps S902 and S903 described above are further described below by way of example. The view conversion matrix and the target view can be explained again by means of the different views shown in FIG. 4. Referring back to FIG. 4, if the view V0 is regarded as a reference view, and the views V1, V2, V3 and V4 are regarded as target views, the view conversion matrix here represents the spatial position offset of the target views V1, V2, V3 or V4 relative to the reference view V0. Since each view corresponds to a virtual camera, the spatial position offset here is actually also the spatial position offset of the virtual camera at the target view relative to the virtual camera at the reference view. The spatial position offset can comprise a rotation vector and a translation vector of the virtual camera. In some embodiments, the rotation vector R can be generated by sampling a random Euler angle. Euler angle is a way to describe the rotation of an object in three-dimensional space, which uses three angles to represent the rotation, usually denoted as yaw angle, pitch angle and roll angle. The yaw angle represents the rotation of the object around the vertical axis, the pitch angle represents the rotation of the object around the horizontal axis, and the roll angle represents the rotation of the object around the front-back axis. In an aircraft, the yaw angle represents the angle of the aircraft turning left and right, the pitch angle represents the angle of the aircraft rolling up and down, and the roll angle represents the angle of the aircraft rolling sideways. Meanwhile, the translation vector t is generated by sampling a random three-dimensional vector, so as to obtain the view conversion matrix (R|t). Assuming that the Euler angle triplet sampled at a certain time is (a, b, g), i.e. the rotation angles of the virtual camera around the X, Y and Z axes are a, b and g respectively, the rotation R vector can be generated according to the following formula: R = R z ( g ) * R y ( b ) * R x ( a ) (Formula 2)

[0067] wherein:

[0068] The translation vector t mentioned above can be represented as t = (x, y, z), representing the translation of the virtual camera relative to the reference viewpoint. x, y, and z represent the translation of the virtual camera on the X, Y, and Z axes, respectively, and the values ​​of x, y, and z can be randomly sampled within a preset range. The viewpoint transformation matrix T = (R|t) is obtained by concatenating the rotation vector R and the translation vector t. For example, the rotation vector R can be a 3×3 matrix, t can be a 3×1 matrix, and the viewpoint transformation matrix T obtained by concatenating the rotation vector R and the translation vector t is a 3×4 matrix. Therefore, the viewpoint transformation matrix T can characterize the spatial position offset of the target viewpoint relative to the reference viewpoint. In step S903, the target viewpoint two-dimensional image corresponding to the target viewpoint is determined based on the viewpoint transformation matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image, and the camera intrinsic parameters. The process of determining the target viewpoint two-dimensional image is actually the process of determining the RGB values ​​of each pixel in the target viewpoint two-dimensional image. In some embodiments, the target viewpoint two-dimensional image corresponding to the target viewpoint can be determined according to the following formula. tgt ~KT src→tgt D src (p src )K -1 p src (Formula 4)

[0069] p tgt p represents the pixel RGB values ​​of the two-dimensional image at the target viewpoint. src D represents the pixel RGB values ​​of the 2D image from the reference viewpoint. src (psrc) represents the disparity of the corresponding pixel in the reference viewpoint disparity map, and K represents the camera intrinsic parameter. The camera intrinsic parameter K can be the actual intrinsic parameter of a conventional camera in this field. Intrinsic parameters are parameters that describe the internal properties of a camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc., which characterize the camera's imaging behavior. Once the camera intrinsic parameters are determined, they generally remain unchanged during the use of the camera. The camera intrinsic parameter K can be expressed by the following formula 5.

[0070] Assume the camera's focal length is f, the pixels on the imaging plane are rectangles, and the pixel size (i.e., the physical size of one pixel) in the x-direction (horizontal direction) is d. x The pixel size in the y-direction (vertical direction) is d. y Then the parameter f in the above formula x f y They can be respectively Parameter f x f y This indicates that the focal length f, which is in physical dimensions, is converted to the focal length in pixels. x,c y These represent the positions of the camera's principal point in the horizontal and vertical directions, respectively. In some embodiments, the camera intrinsic parameter K is set as follows:

[0071] It is understandable that different viewpoint transformation matrices T can be set for different target viewpoints, but the same camera intrinsics can be used.

[0072] Figure 10a schematically illustrates examples of a reference viewpoint 2D image, a reference viewpoint disparity map, a target viewpoint 2D image, and a target viewpoint disparity map. As shown in Figure 10a, the two images on the right represent the reference viewpoint 2D image (elephant) captured by the virtual camera at the reference viewpoint, and the corresponding reference viewpoint disparity map obtained based on the reference viewpoint 2D image. Using the viewpoint transformation matrix T, the virtual camera is moved to the target viewpoint, and a target viewpoint 2D image and a target viewpoint disparity map are captured at the target viewpoint. The two images on the left in Figure 10a represent the target viewpoint 2D image and the target viewpoint disparity map, respectively. Figure 10b schematically shows a comparison of the target viewpoint 2D image and target viewpoint disparity map before and after restoration. The right image in Figure 10b shows the restored target viewpoint 2D image and the restored target viewpoint disparity map, while the left image in Figure 10b shows the target viewpoint 2D image and the target viewpoint disparity map before restoration.

[0073] Another embodiment of the present application provides a method of training a neural network using the training data obtained in the method of obtaining training data as described above, which can be the convolutional neural network described in the foregoing embodiments. The training data can be provided in the form of training data sets, each set of training data can include a reference-view two-dimensional image, a reference-view disparity map corresponding to the reference-view two-dimensional image, a target-view two-dimensional image, and a target-view disparity map corresponding to the target-view two-dimensional image, and can further include a view transformation matrix corresponding to the target view. As shown in FIG. 11, the method of training the neural network can include the following steps: S1101, providing the reference-view two-dimensional image and the reference-view disparity map to the neural network to generate an implicit representation of a three-dimensional image. The implicit representation of the three-dimensional image can include a multi-plane image MPI, i.e., a set of N layers of RGBA images, each plane image having a color channel Ci and an alpha channel. S1102, rendering a reference-view target two-dimensional image and a target-view target two-dimensional image based on the implicit representation of the three-dimensional image at the reference view and the target view, respectively. FIG. 12 schematically shows the training of the convolutional neural network shown in FIG. 7. By providing the reference-view two-dimensional image and the reference-view disparity map to the convolutional neural network, a multi-plane image MPI is generated at the output end of the neural network. S1103, obtaining a reference-view target disparity map and a target-view target disparity map based on the reference-view target two-dimensional image and the target-view target two-dimensional image, respectively; S1104, determining a loss function based on the difference between the reference-view two-dimensional image and the reference-view target two-dimensional image, the difference between the target-view two-dimensional image and the target-view target two-dimensional image, the difference between the reference-view disparity map and the reference-view target disparity map, and the difference between the target-view disparity map and the target-view target disparity map; and S1105, training the neural network using the training data until the value of the loss function is less than a threshold. In the above step S1102, the position of the target view, i.e., the orientation of the virtual camera corresponding to the target view, can be determined according to the reference view and the view transformation matrix, so that a two-dimensional image taken by a virtual camera at the target view (hereinafter referred to as a target-view target two-dimensional image) can be rendered based on the implicit representation of the three-dimensional image, and at the same time, a two-dimensional image taken by a virtual camera at the reference view (hereinafter referred to as a reference-view target two-dimensional image) can be rendered based on the implicit representation of the three-dimensional image. The rendering method used here can be similar to the rendering process described above, which will not be described here again. Similarly, the reference-view target disparity map and the target-view target disparity map can be obtained based on the reference-view target two-dimensional image and the target-view target two-dimensional image, respectively, using the method of generating a disparity map mentioned in the foregoing embodiments.In step S1104, the differences between the reference-view two-dimensional image and the reference-view target two-dimensional image, the differences between the target-view two-dimensional image and the target-view target two-dimensional image, the differences between the reference-view disparity map and the reference-view target disparity map, and the differences between the target-view disparity map and the target-view target disparity map can be determined first, and then the overall loss function for the neural network can be determined based on the above differences. The differences can also be measured by using appropriate loss functions in the art, for example, the above differences can be calculated by using an absolute value loss function (L1 loss function) or a mean square error loss function (L2 loss function). In step S1105, a threshold value can be set for the loss function, and the neural network can be trained by using a plurality of sets of training data, and when the value of the loss function is less than the above threshold value, the training process is stopped, thereby obtaining the trained neural network.

[0074] In some embodiments, the step S1104 of determining the loss function based on the differences between the reference-view two-dimensional image and the reference-view target two-dimensional image, the differences between the target-view two-dimensional image and the target-view target two-dimensional image, the differences between the reference-view disparity map and the reference-view target disparity map, and the differences between the target-view disparity map and the target-view target disparity map can include: determining a first loss sub-function Ls1, a second loss sub-function Ls2, a third loss sub-function Ls3 and a fourth loss sub-function Ls4 according to the differences between the reference-view two-dimensional image and the reference-view target two-dimensional image, the differences between the target-view two-dimensional image and the target-view target two-dimensional image, the differences between the reference-view disparity map and the reference-view target disparity map, and the differences between the target-view disparity map and the target-view target disparity map, respectively, i.e., the first loss sub-function Ls1 represents the differences between the reference-view two-dimensional image and the reference-view target two-dimensional image, the second loss sub-function Ls2 represents the differences between the target-view two-dimensional image and the target-view target two-dimensional image, the third loss sub-function Ls3 represents the differences between the reference-view disparity map and the reference-view target disparity map, and the fourth loss sub-function Ls4 represents the differences between the target-view disparity map and the target-view target disparity map; and determining the sum of the first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function as the loss function. That is, the loss function Ls for the neural network is Ls = Ls1 + Ls2 + Ls3 + Ls4. The first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function can use an absolute value loss function (L1 loss function) or a mean square error loss function (L2 loss function).

[0075] Another embodiment of the present application provides a device for generating multi-view images, as shown in FIG. 13, the device 1300 comprises: an image acquisition module 1310 configured to acquire a two-dimensional image and acquire a corresponding parallax map based on the two-dimensional image; a three-dimensional image implicit representation generation module 1320 configured to generate a three-dimensional image implicit representation based on the two-dimensional image and the parallax map, the three-dimensional image implicit representation generation module comprising a compression module and an expansion module, the compression module comprising a plurality of down-sampling structures, and the expansion module comprising a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and a rendering module 1330 configured to render two-dimensional images and parallax maps for different viewpoints based on the three-dimensional image implicit representation. The various modules in the device for generating multi-view images correspond to the various steps in the method for generating multi-view images described above, and these modules can be implemented in the form of software programming, or in the form of hardware circuit, or in the form of a combination of software programming and hardware circuit. The device uses a neural network to generate an implicit representation of a three-dimensional image based on a two-dimensional image and a parallax map corresponding to the two-dimensional image, and then generates two-dimensional images and parallax maps for different viewpoints through rendering, so that a three-dimensional display device can output three-dimensional content that is more consistent with the real depth relationship, while having the flexibility to be applicable to three-dimensional display devices with different numbers of viewpoints.

[0076] In some embodiments, the compression module in the apparatus further comprises a plurality of two-dimensional convolution structures, the expansion module further comprises a plurality of three-dimensional convolution structures, each down-sampling structure is located between two-dimensional convolution structures of adjacent layers in the plurality of two-dimensional convolution structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures and a first three-dimensional convolution structure in the plurality of three-dimensional convolution structures, the second up-sampling structure is located between three-dimensional convolution structures of adjacent layers in the plurality of three-dimensional convolution structures, wherein the compression module is configured to receive the two-dimensional image and the disparity map to output a corresponding first feature map from each two-dimensional convolution structure in the plurality of two-dimensional convolution structures, wherein the three-dimensional image implicit representation generation module further comprises a feature map fusion module configured to fuse the first feature map with an up-sampled feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure has a same size as the first feature map. A three-dimensional convolution structure of a corresponding layer in the plurality of three-dimensional convolution structures receives the fused feature map to output a corresponding second feature map, and the expansion module obtains the implicit representation of the three-dimensional image based on the second feature map output by a last three-dimensional convolution structure in the plurality of three-dimensional convolution structures. The use of the fused feature map generated by the feature map fusion module and the processing of the fused feature map by the three-dimensional convolution structure can further improve the quality of the generated implicit representation of the three-dimensional image, and is more conducive to the three-dimensional display apparatus to output three-dimensional content that is more consistent with the real depth relationship.

[0077] In some embodiments, the compression module providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolution structure in the plurality of two-dimensional convolution structures can specifically comprise: down-sampling a first feature map output by a previous two-dimensional convolution structure in two adjacent two-dimensional convolution structures in the plurality of two-dimensional convolution structures by using each down-sampling structure in the plurality of down-sampling structures to obtain a down-sampled feature map; and inputting the down-sampled feature map into a subsequent two-dimensional convolution structure in the two adjacent two-dimensional convolution structures to output the corresponding first feature map. In some embodiments, the up-sampled feature map output by the up-sampling structure in the expansion module comprises a first up-sampled feature map output by the first up-sampling structure and a second up-sampled feature map output by the second up-sampling structure, the expansion module is further configured to up-sample the first feature map output by a last two-dimensional convolution structure in the plurality of two-dimensional convolution structures by using the first up-sampling structure in the plurality of up-sampling structures to obtain the first up-sampled feature map; and up-sample the second feature map output by a previous three-dimensional convolution structure in two adjacent three-dimensional convolution structures by using the second up-sampling structure to obtain the second up-sampled feature map.

[0078] In some embodiments, the feature map fusion module fuses the first feature map and the up-sampled feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map includes: fusing the first feature map output by the preceding two-dimensional convolution structure of the last layer of two-dimensional convolution structures and the first up-sampled feature map to obtain a fused feature map provided to the first layer of three-dimensional convolution structures in the plurality of layers of three-dimensional convolution structures; and fusing the second up-sampled feature map and the first feature map output by the preceding two-dimensional convolution structure of the first down-sampling structure corresponding to the second up-sampling structure to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second up-sampling structure.

[0079] In some embodiments, the feature map fusion module fuses the first feature map and the up-sampled feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map includes: superimposing the first feature map and the up-sampled feature map in a direction along the depth dimension. The implicit representation of the three-dimensional image described above includes a multi-plane image.

[0080] Further embodiments of the present application provide a computing device comprising: a memory configured to store computer-executable instructions; a processor configured to perform the steps of any of the methods of the embodiments of the method of generating a multi-view image, the method of obtaining training data for a neural network, the method of training a neural network with training data as previously described when the computer-executable instructions are executed by the processor.

[0081] In particular, the methods in the methods described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of the present application provide a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing at least one step of the method of generating a multi-view image, the method of obtaining training data for a neural network, or the method of training a neural network with training data as described in the above embodiments.

[0082] Another embodiment of the present application provides one or more computer readable storage media having computer readable instructions stored thereon that, when executed by a computer, implement a method of generating multi-view images, a method of obtaining training data for a neural network, or a method of training a neural network with the training data according to some embodiments of the present application. Each step of the method of generating multi-view images, the method of obtaining training data for a neural network, or the method of training a neural network with the training data can be converted into computer readable instructions through programming, so as to be stored in the computer readable storage medium. When such computer readable storage medium is read or accessed by a computing device or computer, the computer readable instructions therein are executed by a processor on the computing device or computer to implement the method of generating multi-view images, the method of obtaining training data for a neural network, or the method of training a neural network with the training data.

[0083] FIG. 14 illustrates an example system including a computing device 1410, which is an example computing device of one or more systems and / or devices that can implement the techniques described in various embodiments herein. The computing device 1410 can be, for example, a server of a service provider, a device associated with the server, a system on a chip, and / or any other suitable computing device or computing system. The apparatus for generating multi-view images described above with reference to FIG. 13 can take the form of the computing device 1410. Alternatively, the apparatus for generating multi-view images can be implemented as a computer program in the form of the application 1416.

[0084] The example computing device 1410 as illustrated in FIG. 14 includes a processing system 1411, one or more computer readable media 1412, and one or more I / O interfaces 1413 that are communicatively coupled with one another. Although not shown, the computing device 1410 can also include a system bus or other data and command transfer system that couples the various components with one another in communication. The system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a serial bus, a parallel bus, and / or a local bus using any of a variety of bus architectures by which processors and other digital electronic components can communicate.

[0085] Processing system 1411 represents the functionality of performing one or more operations using hardware. Therefore, processing system 1411 is illustrated as including hardware elements 1414 that can be configured as processors, function blocks, etc. This may include other logic devices implemented in hardware as application-specific integrated circuits (ASICs) or formed using one or more semiconductors. Hardware elements 1414 are not limited by the materials in which they are formed or the processing mechanism employed therein. For example, a processor may consist of semiconductors and / or transistors (e.g., integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically executable instructions. Computer-readable medium 1412 is illustrated as including memory / storage device 1415. Memory / storage device 1415 represents a memory / storage capacity associated with one or more computer-readable media. Memory / storage device 1415 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage device 1415 may include fixed media (e.g., RAM, ROM, fixed hard disk drive, etc.) and removable media (e.g., flash memory, removable hard disk drive, optical disk, etc.). Computer-readable medium 1412 may be configured in various other ways as further described below. One or more I / O interfaces 1413 represent functions that allow users to input commands and information to computing device 1410 using various input devices and optionally also allow information to be presented to the user and / or other components or devices using various output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones (e.g., for voice input), scanners, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., capable of detecting non-touch-related movements as gestures using visible or invisible wavelengths (such as infrared frequencies), etc. Examples of output devices include display devices (e.g., monitors or projectors), speakers, printers, network interface cards, haptic-responsive devices, etc. Therefore, computing device 1410 may be configured in various ways as further described below to support user interaction.

[0086] The computing device 1410 also includes an application 1416. The application 1416 may be, for example, a software example of the apparatus 1300 for generating multi-view images described with reference to FIG13, and implements the techniques described herein in combination with other elements in the computing device 1410.

[0087] Various techniques can be described in the general context of software hardware elements or program modules being executed on a computer. Generally, these modules include routines, programs, objects, elements, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The terms "module", "functionality", and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that they can be implemented on a variety of computing platforms having a variety of processors.

[0088] Implementations of the described modules and techniques can be stored or transmitted across some form of computer-readable media. Computer-readable media can include various media that can be accessed by the computing device 1410. By way of example, and not limitation, computer-readable media can include "computer-readable storage media" and "computer-readable signal media".

[0089] In contrast to signal bearing media, "computer-readable storage media" refers to media or means configured to hold information for a period of time. Accordingly, computer-readable storage media does not include signals per se. Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, program modules, logical elements / circuits, or other data. Examples of computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture that are appropriate for storage of desired information and that can be accessed by a computer.

[0090] "Computer-readable signal media" refers to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device 1410, such as via a network. Signal media typically can take the form of modulated data signals, e.g., carrier waves, data signals, or other transport mechanisms. Signal media include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0091] As previously described, hardware elements 1414 and computer-readable media 1412 are representative of instructions, modules, programmable device logic and / or fixed device logic implemented in a hardware form that can be employed in some embodiments to implement at least portions of the techniques described herein. Hardware elements can include components of an integrated circuit or

[0092] The foregoing combination of software and / or hardware modules can also be employed to implement various techniques and modules described herein. Accordingly, software, hardware, or program modules and other program modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage media and / or by one or more hardware elements 1414. The computing device 1410 can be configured to implement particular instructions and / or functions corresponding to the software and / or hardware modules. Accordingly, implementation of a module that is executable by the computing device 1410 as software can be achieved at least partially in hardware, e.g., through the use of computer-readable storage media and / or hardware elements 1414 of the processing system. The instructions and / or functions can be executable / operable by one or more articles of manufacture (for example, one or more computing devices 1410 and / or processing systems 1411) to implement techniques, modules, and examples described herein.

[0093] In various implementations, the computing device 1410 can assume a variety of different configurations. For example, the computing device 1410 can be implemented as a computer-class device comprising a personal computer, desktop computer, multi-screen computer, laptop computer, netbook, etc. The computing device 1410 can also be implemented as a mobile device-class device comprising a mobile phone, portable music player, portable gaming device, tablet computer, multi-screen computer, etc. The computing device 1410 can also be implemented as a television-class device comprising a device having or connected to a generally larger screen in a casual viewing environment. These devices include televisions, set-top boxes, gaming consoles, etc.

[0094] The techniques described herein can be supported by these various configurations of the computing device 1410 and are not limited to the specific examples of the techniques described herein. Functionality can also be implemented all or in part within the platform 1422, e.g., as a platform abstraction layer (PAL). The cloud 1420 includes and / or is representative of the platform 1422 for resources 1424. The platform 1422 abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud 1420. The resources 1424 can include additional applications and / or data that can be utilized by a user of the computing device 1410, depending on the capacity of the computing device 1410 and the particular needs of the user. Resources 1424 can also include services provided over the Internet and / or through a subscriber network, such as a cellular or Wi-Fi network.

[0095] The platform 1422 can abstract resources and functions to connect the computing device 1410 with other computing devices. The platform 1422 can also serve to abstract scaling of resources to provide a corresponding level of scale to encountered demand for the resources 1424 that are implemented via the platform 1422. Accordingly, in an interconnected device embodiment, implementation of functionality described herein can be distributed throughout the system 1400. For example, the functionality can be implemented in part on the computing device 1410 as well as via the platform 1422 that abstracts the functionality of the cloud 1420.

[0096] With the method, device, and computer device capable of executing the foregoing method for generating multi-view images, computer readable medium, and computer program product provided by the embodiments of the present application, a three-dimensional image implicit representation (e.g., a multi-plane image MPI) can be generated according to the provided two-dimensional images, and based on the three-dimensional image implicit representation, two-dimensional images for different viewpoints and a disparity map can be rendered. The two-dimensional images for different viewpoints can be regarded as images taken by virtual cameras at different positions and orientations. After obtaining the two-dimensional images for different viewpoints, they can be combined into a single image, and finally the combined image can be copied or delivered to a three-dimensional display terminal, so that a stereoscopic display effect can be seen. The total number of viewpoints including the reference viewpoint and the target viewpoint can be different for different stereoscopic display devices, and the technical solutions provided by the embodiments of the present application are not limited by the number of viewpoints required by the stereoscopic display device, and can be applied to stereoscopic display devices requiring different numbers of viewpoints.

[0097] It will be appreciated that, for clarity, embodiments of the application have been described hereinafter with reference to different functional units. It will be apparent, however, that the functional units can be implemented in one single unit or in different units, or as part of other functional units. For example, the functionality of a single unit can be performed by one or more different units. Thus, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization. Accordingly, the application can be embodied in single units, or can be physically and functionally distributed between different units and circuitries.

[0098] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these terms. These terms are only used to distinguish one device, element, component or part from another.

[0099] Although the application has been described in connection with some embodiments, it is not intended to be limited to the particular form set forth herein. Rather, the scope of the present application is limited only by the claims. Additionally, although individual features can be included in different claims, these can possibly, though need not, be combined, and the inclusion of different claims does not imply that a combination of features is not feasible and / or is not advantageous. The order in which the features of the claims are recited is not intended to prefer one feature before or over another. Furthermore, the use of the term "comprise" or "comprising" does not exclude the presence of other elements or steps than those listed in a claim. The use of the indefinite article "an" or "one" does not exclude the presence of several of these elements.

Claims

1. A method of generating multi-view images, comprising: obtaining a two-dimensional image and a corresponding disparity map based on the two-dimensional image; inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation, the neural network comprising a compression module and an expansion module, the compression module comprising a plurality of down-sampling structures, and the expansion module comprising a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and rendering generating two-dimensional images and disparity maps for different views based on the three-dimensional image implicit representation. 2.The method of claim 1, wherein the compression module further comprises a plurality of two-dimensional convolutional structures, and the expansion module further comprises a plurality of three-dimensional convolutional structures, each down-sampling structure is located between two-dimensional convolutional structures of adjacent layers in the plurality of two-dimensional convolutional structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures and a first three-dimensional convolutional structure in the plurality of three-dimensional convolutional structures, and the second up-sampling structure is located between three-dimensional convolutional structures of adjacent layers in the plurality of three-dimensional convolutional structures, wherein inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation comprises: providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures; fusing the first feature map with an up-sampled feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure has a same size as the first feature map; providing the fused feature map to a three-dimensional convolutional structure of a corresponding layer in the plurality of three-dimensional convolutional structures to output a corresponding second feature map; and obtaining the three-dimensional image implicit representation based on a second feature map output by a last three-dimensional convolutional structure in the plurality of three-dimensional convolutional structures. 3.The method of claim 2, wherein providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures comprises: down-sampling a first feature map output by a preceding two-dimensional convolutional structure in two adjacent two-dimensional convolutional structures in the plurality of two-dimensional convolutional structures using each down-sampling structure in the plurality of down-sampling structures to obtain a down-sampled feature map; and inputting the down-sampled feature map into a following two-dimensional convolutional structure in the two adjacent two-dimensional convolutional structures to output the corresponding first feature map. 4.The method of claim 2, wherein the up-sampled feature map comprises a first up-sampled feature map output by the first up-sampling structure and a second up-sampled feature map output by the second up-sampling structure, and wherein the method further comprises: up-sampling a first feature map output by a last two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures using the first up-sampling structure in the plurality of up-sampling structures to obtain the first up-sampled feature map; and ​ ​ ​ ​ The second feature map output by a front-stage three-dimensional convolution structure in the three-dimensional convolution structure of the adjacent two layers is up-sampled by using the second up-sampling structure to obtain the second up-sampled feature map.

5. The method of claim 4, wherein fusing the first feature map and the up-sampled feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map comprises: fusing the first feature map output by a front-stage two-dimensional convolution structure of the two-dimensional convolution structure of the last layer and the first up-sampled feature map to obtain a fused feature map provided to a first three-dimensional convolution structure in the plurality of three-dimensional convolution structures; and fusing the second up-sampled feature map and a first feature map output by a front-stage two-dimensional convolution structure of a first down-sampling structure corresponding to the second up-sampling structure to obtain a fused feature map provided to a back-stage three-dimensional convolution structure of the second up-sampling structure.

6. The method of claim 2, wherein fusing the first feature map and the up-sampled feature map output by the corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map comprises: stacking the first feature map and the up-sampled feature map in a direction along a depth dimension.

7. The method of claim 1, wherein the implicit representation of the three-dimensional image comprises a multi-plane image.

8. The method of claim 1, wherein the different viewpoints comprise a reference viewpoint corresponding to the two-dimensional image and a target viewpoint different from the reference viewpoint, and wherein rendering generating two-dimensional images and disparity maps for different viewpoints based on the implicit representation of the three-dimensional image comprises: rendering generating two-dimensional images and disparity maps for the reference viewpoint and the target viewpoint based on the implicit representation of the three-dimensional image.

9. The method of claim 1, wherein obtaining a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image comprises: generating a disparity map corresponding to the two-dimensional image based on the two-dimensional image using a monocular depth estimation network.

10. A method for obtaining training data for a neural network as claimed in claim 1, comprising: obtaining a plurality of reference viewpoint two-dimensional images corresponding to a reference viewpoint and a reference viewpoint disparity map corresponding to each reference viewpoint two-dimensional image in the plurality of reference viewpoint two-dimensional images; determining a viewpoint transformation matrix and a target viewpoint for each reference viewpoint two-dimensional image, the viewpoint transformation matrix representing a spatial positional offset of the target viewpoint relative to the reference viewpoint; determining a target viewpoint two-dimensional image corresponding to the target viewpoint based on the viewpoint transformation matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image, and camera intrinsic parameters; obtaining a target viewpoint disparity map corresponding to the target viewpoint two-dimensional image; and forming training data based on the reference viewpoint two-dimensional images, the reference viewpoint disparity map, the target viewpoint two-dimensional image, and the target viewpoint disparity map.

11. The method of claim 10, wherein forming training data based on the reference viewpoint two-dimensional images, the reference viewpoint disparity map, the target viewpoint two-dimensional image, and the target viewpoint disparity map comprises: ​ ​ holes in the target view two-dimensional image and the target view disparity map are repaired to obtain a repaired target view two-dimensional image and a repaired target view disparity map, and the reference view two-dimensional image, the reference view disparity map, the repaired target view two-dimensional image and the repaired target view disparity map are taken as training data. 12.The method of claim 10, wherein determining a target view two-dimensional image corresponding to the target view according to the view transformation matrix, the reference view disparity map, the reference view two-dimensional image and camera intrinsic parameters comprises: determining pixel values of respective pixels of the target view two-dimensional image according to pixel values of respective pixels of the reference view two-dimensional image, the reference view disparity map, the view transformation matrix and camera intrinsic parameters. 13.The method of claim 10, wherein determining a view transformation matrix and a target view for each reference view two-dimensional image comprises: randomly determining a rotation vector and a translation vector for a reference view corresponding to the reference view two-dimensional image, the rotation vector representing an Euler angle offset of the target view relative to the reference view, and the translation vector representing a coordinate offset of the target view relative to the reference view. 14.A method of training the neural network of claim 1 with the training data of claim 10, comprising: providing a reference view two-dimensional image and a reference view disparity map to the neural network to generate an implicit representation of a three-dimensional image, rendering a reference view target two-dimensional image and a target view target two-dimensional image at a reference view and a target view respectively based on the implicit representation of the three-dimensional image; obtaining a reference view target disparity map and a target view target disparity map respectively based on the reference view target two-dimensional image and the target view target two-dimensional image; determining a loss function based on a difference between the reference view two-dimensional image and the reference view target two-dimensional image, a difference between the target view two-dimensional image and the target view target two-dimensional image, a difference between the reference view disparity map and the reference view target disparity map, and a difference between the target view disparity map and the target view target disparity map; and training the neural network with the training data until a value of the loss function is less than a threshold. 15.The method of claim 14, wherein determining a loss function based on a difference between the reference view two-dimensional image and the reference view target two-dimensional image, a difference between the target view two-dimensional image and the target view target two-dimensional image, a difference between the reference view disparity map and the reference view target disparity map, and a difference between the target view disparity map and the target view target disparity map comprises: determining a first loss sub-function, a second loss sub-function, a third loss sub-function and a fourth loss sub-function according to a difference between the reference-view two-dimensional image and the reference-view target two-dimensional image, a difference between the target-view two-dimensional image and the target-view target two-dimensional image, a difference between the reference-view disparity map and the reference-view target disparity map, and a difference between the target-view disparity map and the target-view target disparity map; determining a sum of the first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function as the loss function.

16. An apparatus for generating multi-view images, comprising: an image acquisition module configured to acquire a two-dimensional image and acquire a corresponding disparity map based on the two-dimensional image; a three-dimensional image implicit representation generation module configured to generate a three-dimensional image implicit representation based on the two-dimensional image and the disparity map, the three-dimensional image implicit representation generation module comprising a compression module and an expansion module, the compression module comprising a plurality of down-sampling structures, and the expansion module comprising a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and a rendering module configured to render, based on the three-dimensional image implicit representation, two-dimensional images and disparity maps for different views.

17. The apparatus of claim 16, wherein the compression module further comprises a plurality of two-dimensional convolutional structures, the expansion module further comprises a plurality of three-dimensional convolutional structures, each down-sampling structure is located between two-dimensional convolutional structures of adjacent layers in the plurality of two-dimensional convolutional structures, the plurality of up-sampling structures comprises a first up-sampling structure and a second up-sampling structure, the first up-sampling structure is located between a last two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures and a first three-dimensional convolutional structure in the plurality of three-dimensional convolutional structures, the second up-sampling structure is located between three-dimensional convolutional structures of adjacent layers in the plurality of three-dimensional convolutional structures, wherein the compression module is configured to receive the two-dimensional image and the disparity map to output a corresponding first feature map from each two-dimensional convolutional structure in the plurality of two-dimensional convolutional structures, wherein the three-dimensional image implicit representation generation module further comprises a feature map fusion module configured to fuse the first feature map with an up-sampled feature map output by a corresponding up-sampling structure in the plurality of up-sampling structures to obtain a fused feature map, the up-sampled feature map output by the corresponding up-sampling structure has a same size as the first feature map, wherein a three-dimensional convolutional structure of a corresponding layer in the plurality of three-dimensional convolutional structures receives the fused feature map to output a corresponding second feature map, the expansion module obtains the implicit representation of the three-dimensional image based on a second feature map output by a last three-dimensional convolutional structure in the plurality of three-dimensional convolutional structures.

18. A computing device, comprising: a memory configured to store computer-executable instructions; and a processor configured to perform the method of any one of claims 1-15 when the computer-executable instructions are executed by the processor.

19. A computer-readable storage medium storing computer-executable instructions that, when executed, implement the method of any one of claims 1-15.

20. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 15.

Images