Multi-viewpoint image generation method, training data acquisition method, and neural network training method

Abstract

Embodiments of the present application at least provide a method for generating a multi-viewpoint image. The method comprises: acquiring a two-dimensional image and acquiring a corresponding disparity map on the basis of the two-dimensional image; inputting the two-dimensional image and the disparity map into a neural network to obtain a three-dimensional image implicit representation, wherein the neural network comprises a compression module and an expansion module, the compression module comprises a plurality of down-sampling structures, and the expansion module comprises a plurality of up-sampling structures corresponding to the plurality of down-sampling structures; and on the basis of the three-dimensional image implicit representation, performing rendering to generate two-dimensional images and disparity maps for different viewpoints. According to the method, the neural network is used to generate the implicit representation of a three-dimensional image on the basis of the two-dimensional image and the disparity map corresponding to the two-dimensional image, and on this basis, the two-dimensional images and the disparity maps for different viewpoints are generated by means of rendering, so that a three-dimensional display device can output three-dimensional content better conforming to a real depth-of-field relationship, and thus, the method has the characteristic of flexibility and can be suitable for three-dimensional display devices with different numbers of viewpoints.

Description

Multi-view image generation method, training data acquisition method, and neural network training method Technical Field

[0001] The present application relates to the field of display technology, and more specifically, to a method for generating multi-viewpoint images, a method for obtaining training data for a neural network, a method for training a neural network using training data, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Autostereoscopic display technology, also known as naked-eye stereoscopic display technology, aims to enable viewers to view three-dimensional images without the use of special glasses. The principle behind autostereoscopic display technology involves using a specially designed display screen capable of simultaneously displaying multiple viewpoint images. Each viewpoint image corresponds to an image viewed from a different position on the viewer's screen. By dividing the screen into multiple areas, each displaying a different image, the viewer's left and right eyes simultaneously see different images, creating a three-dimensional effect. Depending on the optical design of the autostereoscopic display device, different autostereoscopic devices can have different numbers of viewpoints, requiring a correspondingly different number of virtual cameras. For example, a 49-viewpoint autostereoscopic display device requires 49 virtual cameras. After acquiring images captured by multiple virtual cameras, these images can be merged into a single image. This merged image can then be copied or downloaded to the autostereoscopic display device, allowing the viewer to perceive the three-dimensional display effect. However, conventional technologies for generating three-dimensional images from two-dimensional images suffer from high costs and limited applicability.

[0003] Summary of the Invention

[0004] An embodiment of the present application proposes a method for generating multi-viewpoint images, the method comprising: obtaining a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image; inputting the two-dimensional image and the disparity map into a neural network to obtain an implicit representation of a three-dimensional image, the neural network comprising a compression module and an expansion module, the compression module comprising multiple downsampling structures, the expansion module comprising multiple upsampling structures corresponding to the multiple downsampling structures; and rendering and generating two-dimensional images and disparity maps for different viewpoints based on the implicit representation of the three-dimensional image.

[0005] According to some embodiments of the present application, the compression module further includes a multi-layer two-dimensional convolution structure, the expansion module further includes a multi-layer three-dimensional convolution structure, each downsampling structure is located between the two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first upsampling structure is located between the last layer of two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, the second upsampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure, wherein the two-dimensional image and the disparity map are input into a neural network to obtain an implicit representation of the three-dimensional image The method includes: providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure; fusing the first feature map with the up-sampling feature map output by the corresponding up-sampling structure in the multiple up-sampling structures to obtain a fused feature map, and the up-sampling feature map output by the corresponding up-sampling structure has the same size as the first feature map; providing the fused feature map to the three-dimensional convolution structure of the corresponding layer in the multi-layer three-dimensional convolution structure to output a corresponding second feature map; and obtaining an implicit representation of the three-dimensional image based on the second feature map output by the last layer of the three-dimensional convolution structure in the multi-layer three-dimensional convolution structure.

[0006] According to some embodiments of the present application, providing the two-dimensional image and the disparity map to the compression module to output the corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure includes: using each downsampling structure in the multiple downsampling structures to downsample the first feature map output by the previous two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to obtain a downsampled feature map; and inputting the downsampled feature map into the next two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to output the corresponding first feature map.

[0007] According to some embodiments of the present application, the up-sampled feature map includes a first up-sampled feature map output by the first up-sampling structure and a second up-sampled feature map output by the second up-sampling structure, wherein the method further includes: using the first up-sampling structure among the multiple up-sampling structures to up-sample the first feature map output by the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure to obtain the first up-sampled feature map; and using the second up-sampling structure to up-sample the second feature map output by the previous three-dimensional convolution structure in the three-dimensional convolution structure of two adjacent layers to obtain the second up-sampling feature map.

[0008] According to some embodiments of the present application, fusing the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map includes: fusing the first feature map output by the previous two-dimensional convolution structure of the last layer of the two-dimensional convolution structure with the first upsampling feature map to obtain a fused feature map provided to the first layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure; and fusing the second upsampling feature map with the first feature map output by the previous two-dimensional convolution structure of the first downsampling structure corresponding to the second upsampling structure to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second upsampling structure.

[0009] According to some embodiments of the present application, fusing the first feature map with the upsampled feature map output by the corresponding upsampled structure in the multiple upsampled structures to obtain a fused feature map includes: superimposing the first feature map and the upsampled feature map in a direction along the depth dimension.

[0010] According to some embodiments of the present application, the implicit representation of the three-dimensional image includes a multi-plane image.

[0011] According to some embodiments of the present application, the different viewpoints include a reference viewpoint corresponding to the two-dimensional image and a target viewpoint different from the reference viewpoint, wherein generating two-dimensional images and disparity maps for different viewpoints based on the implicit representation rendering of the three-dimensional image includes: generating two-dimensional images and disparity maps of the reference viewpoint and the target viewpoint based on the implicit representation rendering of the three-dimensional image.

[0012] According to some embodiments of the present application, acquiring a two-dimensional image and acquiring a corresponding disparity map based on the two-dimensional image includes: generating a disparity map corresponding to the two-dimensional image based on the two-dimensional image using a monocular depth estimation network.

[0013] Another embodiment of the present application provides a method for obtaining training data for a neural network, which can be the neural network described in any of the aforementioned embodiments, the method comprising: obtaining multiple reference viewpoint two-dimensional images corresponding to a reference viewpoint and a reference viewpoint disparity map corresponding to each reference viewpoint two-dimensional image in the multiple reference viewpoint two-dimensional images; determining a viewpoint conversion matrix and a target viewpoint for each reference viewpoint two-dimensional image, the viewpoint conversion matrix representing the spatial position offset of the target viewpoint relative to the reference viewpoint; determining a target viewpoint two-dimensional image corresponding to the target viewpoint based on the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image and camera intrinsic parameters; obtaining a target viewpoint disparity map corresponding to the target viewpoint two-dimensional image; and forming training data based on the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the target viewpoint two-dimensional image and the target viewpoint disparity map.

[0014] According to some embodiments of the present application, forming training data based on the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the target viewpoint two-dimensional image and the target viewpoint disparity map includes: repairing holes in the target viewpoint two-dimensional image and the target viewpoint disparity map to obtain a repaired target viewpoint two-dimensional image and a repaired target viewpoint disparity map, and using the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the repaired target viewpoint two-dimensional image and the repaired target viewpoint disparity map as training data.

[0015] According to some embodiments of the present application, determining the target viewpoint two-dimensional image corresponding to the target viewpoint based on the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image and the camera intrinsic parameters includes: determining the pixel value of each pixel of the target viewpoint two-dimensional image based on the viewpoint conversion matrix, the reference viewpoint disparity map, the pixel value of each pixel of the reference viewpoint two-dimensional image and the camera intrinsic parameters.

[0016] According to some embodiments of the present application, determining the viewpoint transformation matrix and the target viewpoint for each reference viewpoint two-dimensional image includes: randomly determining a rotation vector and a translation vector for the reference viewpoint corresponding to the reference viewpoint two-dimensional image, the rotation vector representing the Euler angle offset of the target viewpoint relative to the reference viewpoint, and the translation vector representing the coordinate offset of the target viewpoint relative to the reference viewpoint.

[0017] Another embodiment of the present application provides a method for training a neural network using the training data described in the aforementioned embodiment of the method for obtaining training data for a neural network. The neural network may be the neural network described in the aforementioned embodiment of the method for generating multi-viewpoint images. The method for training the neural network includes: providing the reference viewpoint two-dimensional image and the reference viewpoint disparity map to the neural network to generate an implicit representation of a three-dimensional image; rendering a two-dimensional image of a reference viewpoint marker and a two-dimensional image of a target viewpoint marker at the reference viewpoint and the target viewpoint, respectively, based on the implicit representation of the three-dimensional image; obtaining a disparity map of the reference viewpoint marker and a disparity map of the target viewpoint marker based on the two-dimensional image of the reference viewpoint marker and the two-dimensional image of the target viewpoint marker, respectively; determining a loss function based on the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint marker, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint marker, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint marker, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint marker; and training the neural network using the training data until the value of the loss function is less than a threshold.

[0018] According to some embodiments of the present application, determining a loss function based on the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint mark, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint mark, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint mark, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint mark includes: determining a first loss sub-function, a second loss sub-function, a third loss sub-function and a fourth loss sub-function according to the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint mark, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint mark, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint mark, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint mark; and determining the sum of the first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function as the loss function.

[0019] Another embodiment of the present application provides a device for generating multi-viewpoint images, which includes: an image acquisition module, which is configured to acquire a two-dimensional image and acquire a corresponding disparity map based on the two-dimensional image; a three-dimensional image implicit representation generation module, which is configured to generate a three-dimensional image implicit representation based on the two-dimensional image and the disparity map, the three-dimensional image implicit representation generation module including a compression module and an expansion module, the compression module including multiple downsampling structures, and the expansion module including multiple upsampling structures corresponding to the multiple downsampling structures; and a rendering module, which is configured to generate two-dimensional images and disparity maps for different viewpoints based on the three-dimensional image implicit representation rendering.

[0020] According to some embodiments of the present application, the compression module further includes a multi-layer two-dimensional convolution structure, the expansion module further includes a multi-layer three-dimensional convolution structure, each downsampling structure is located between the two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first upsampling structure is located between the last layer of two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, the second upsampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure, wherein the compression module is configured to receive the two-dimensional image and the disparity map to obtain the disparity map from the multi-layer two-dimensional convolution structure. Each layer of two-dimensional convolution structure in the structure outputs a corresponding first feature map, and the three-dimensional image implicit representation generation module also includes a feature map fusion module, which is configured to fuse the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map, and the upsampling feature map output by the corresponding upsampling structure has the same size as the first feature map, wherein the three-dimensional convolution structure of the corresponding layer in the multi-layer three-dimensional convolution structure receives the fused feature map to output a corresponding second feature map, and the expansion module obtains the implicit representation of the three-dimensional image based on the second feature map output by the last layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure.

[0021] Another embodiment of the present application provides a computing device, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute a method as described in any one of the aforementioned method embodiments when the computer-executable instructions are executed by the processor.

[0022] Yet another embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed, implement a method as described in any one of the aforementioned method embodiments.

[0023] Yet another embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements a method as described in any one of the aforementioned method embodiments.

[0024] Compared to conventional technologies based on converting two-dimensional images into three-dimensional images, the method and device for generating multi-viewpoint images provided in the embodiments of the present application utilize a neural network to generate an implicit representation of a three-dimensional image based on a two-dimensional image and a disparity map corresponding to the two-dimensional image. On this basis, two-dimensional images and disparity maps for different viewpoints are generated through rendering. This allows a three-dimensional display device to output three-dimensional content that is more consistent with a real depth of field relationship, promotes consistent generation results from different perspectives, and at the same time has the characteristics of flexibility and is applicable to three-dimensional display devices with different numbers of viewpoints. These and other advantages of the present application will become clear based on the embodiments described below, and these and other advantages of the present application will be clarified with reference to the embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Embodiments of the present application will now be described in more detail and with reference to the accompanying drawings, in which:

[0026] FIG1 illustrates some steps in a method for generating multi-view images according to an embodiment of the present application;

[0027] FIG2 is used to schematically illustrate the calculation of disparity;

[0028] FIG3 is used to schematically illustrate zero parallax, positive parallax, and negative parallax;

[0029] FIG4 schematically shows an example of a reference viewpoint and a target viewpoint different from the reference viewpoint;

[0030] FIG5 schematically illustrates some steps involved in inputting a two-dimensional image and a disparity map into a neural network to obtain an implicit representation of a three-dimensional image according to an embodiment of the present application;

[0031] FIG6 illustrates some steps in a method for generating multi-view images according to another embodiment of the present application;

[0032] FIG7 schematically shows an example of a model of a neural network including a compression module and an expansion module;

[0033] FIG8 is used to schematically illustrate a multi-plane image;

[0034] FIG9 illustrates some steps included in an embodiment of a method for obtaining training data for a neural network,

[0035] FIG10 a schematically shows an example of a reference viewpoint two-dimensional image, a reference viewpoint disparity map, a target viewpoint two-dimensional image, and a target viewpoint disparity map;

[0036] FIG10 b schematically shows a comparison between the target viewpoint 2D image and the target viewpoint disparity map before and after restoration;

[0037] FIG11 illustrates some steps included in a method for training a neural network according to an embodiment of the present application;

[0038] FIG12 is a diagram schematically illustrating the training of the neural network model shown in FIG7 ;

[0039] FIG13 illustrates an apparatus for generating multi-viewpoint images;

[0040] FIG. 14 illustrates an example system including an example computing device representative of one or more systems and / or devices that may implement the various methods or apparatus described herein. DETAILED DESCRIPTION

[0041] The following description provides specific details of various embodiments of the present application so that those skilled in the art can fully understand and implement the various embodiments of the present application. In some cases, this application does not illustrate or describe in detail some structures or functions well known in the art to avoid these unnecessary descriptions from obscuring the description of the embodiments of the present application. The technical solutions of the present application can be embodied in many different forms and purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make the technical solutions of the present application clear and complete, but the embodiments do not limit the scope of protection of this patent application.

[0042] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the technical solutions of the present application can be implemented without one or more of these details. Detailed steps and detailed structures are provided in the following description to illustrate the technical solutions proposed by the present application. The terms used in this application should be understood in their broadest reasonable manner, even when used in conjunction with specific embodiments of the present application. A neural network is a computational model that simulates the human nervous system. It learns and trains on large amounts of data to predict and classify unknown data. The basic building block of a neural network is a neuron. Each neuron receives input signals from other neurons and processes and outputs them according to specific rules. These neurons are interconnected through connection weights, forming a complex network structure. During the training phase, the neural network receives a large amount of labeled data and optimizes the network's performance by continuously adjusting the connection weights of the neurons. This learning and prediction process is the core function of a neural network. When a neural network receives new data, it predicts and classifies the data based on the knowledge it has already learned.

[0043] An embodiment of the present application provides a method for generating multi-viewpoint images. FIG1 illustrates some steps included in an embodiment of the method for generating multi-viewpoint images. As shown in FIG1 , the method for generating multi-viewpoint images may include the following steps: S101, obtaining a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image; S102, inputting the two-dimensional image and the disparity map into a neural network to obtain an implicit representation of a three-dimensional image, wherein the neural network includes a compression module and an expansion module, wherein the compression module includes multiple downsampling structures, and the expansion module includes multiple upsampling structures corresponding to the multiple downsampling structures; S103, rendering and generating two-dimensional images and disparity maps for different viewpoints based on the implicit representation of the three-dimensional image.

[0044] Three-dimensional representation is a key technology in machine vision. It can intuitively reflect the shape of objects. Compared with two-dimensional representation, three-dimensional representation contains depth information. Therefore, effective three-dimensional representation is crucial for achieving machine vision tasks such as three-dimensional model reconstruction, three-dimensional object detection, and scene semantic segmentation. It has broad application prospects in robotics, AR / VR, human-computer interaction, remote sensing mapping, and other fields. Three-dimensional spatial representation can be expressed in two forms: explicit representation and implicit representation. Common explicit representations include voxels, point clouds, and meshes. The characteristic of explicit representation is that the description of a scene is explicit, and the three-dimensional representation of the scene can be directly viewed. For example, the corresponding scene can be directly visualized based on meshes. Common implicit representations include signed distance function, occupancy field, neural radiance field (NeRF), and multi-plane image (MPI).

[0045] In a spatial scene, the image points formed by the same object point in the left and right viewpoint images are called homologous image points, and the position difference of the homologous image points is defined as parallax. The difference between homologous image points in the vertical direction is called vertical parallax, and the difference in the horizontal direction is called horizontal parallax. Horizontal parallax is the main factor in achieving three-dimensional display, while vertical parallax has no practical effect on three-dimensional display and can easily cause fatigue to the viewer. The parallax involved in the embodiments of this article refers to horizontal parallax. The parallax is further explained below using two viewpoints as an example. As shown in Figure 2, P represents an object point in a three-dimensional scene, and the object point P forms a homologous image point p in the left and right viewpoint images. u 、p' u , O1 and O2 represent the human eyes, z represents the vertical distance from the human eye to the object point, f represents the vertical distance from the human eye to the viewpoint u, u', B represents the distance between the human eyes, then the parallax d is

[0046] It can be seen that the closer the object point p is to the human eye, the larger the parallax d is. The distance between the object point and the human eye can be inferred by the size of the parallax. Horizontal parallax can be divided into zero parallax, positive parallax, and negative parallax. The following diagram schematically illustrates zero parallax, positive parallax, and negative parallax. As shown in Figure 3, point O L Indicates the position of the image point of the object point in the left viewpoint image (ie, the display plane), point O R Indicates the position of the image point of the object point in the right viewpoint image (i.e., display plane). L and point O R When they coincide, there is zero parallax. In a stereoscopic display scene, the viewer will perceive that the reconstructed object point is located on the display plane. R Located at point O L On the right side is positive parallax. In the stereoscopic display scene, the viewer will perceive that the reproduced object point is located behind the display plane, that is, "entering the screen"; when point O R Located at point O L On the left side is negative parallax. In a stereoscopic display scene, the viewer will perceive that the reproduced object point is in front of the display plane, that is, "out of screen".

[0047] As mentioned above, the number of viewpoints may vary depending on the optical design of the stereoscopic display device. Each viewpoint can be understood as corresponding to a virtual camera. Figure 4 schematically shows viewpoint V0, viewpoint V1, viewpoint V2, viewpoint V3 and viewpoint V4. Each viewpoint can be regarded as corresponding to a different virtual camera. The shooting directions of the virtual cameras can converge at point Q. By adjusting the convergence position of the shooting directions of the virtual cameras, it can be determined which parts of the three-dimensional display content are out of the screen and which parts are in the screen.

[0048] In the previous step S101—acquiring a two-dimensional image and obtaining a corresponding disparity map based on the two-dimensional image—a monocular depth estimation network can be used to estimate the disparity map of the two-dimensional image. Depth estimation is a fundamental problem in the field of computer vision. It involves obtaining the distance information from each point in the scene in the image to the camera. This distance information is called a depth map, which can be applied in fields such as robot navigation, augmented reality, three-dimensional reconstruction, and autonomous driving. Monocular depth estimation uses a single or unique RGB image to estimate the distance of each pixel in the image relative to the source. The human eye has a large amount of prior knowledge, so a large amount of depth information can be extracted from the image information obtained by a single eye. This application does not limit the specific algorithm of the monocular depth estimation network that can be used. For example, cue-based monocular depth estimation, machine learning-based monocular depth estimation, deep learning-based monocular depth estimation, etc. can be used. For example, the disparity data generation method disclosed in patent CN112991419A can be used to generate a disparity map. A monocular depth estimation network can be pre-trained, and then the trained monocular depth estimation network can be used to generate disparity data corresponding to the disparity map based on the 2D image. For example, a depth network can be used as a monocular depth estimation network, and the trained depth network can generate the corresponding disparity map based on the 2D image.

[0049] In step S102, a 2D image and a disparity map corresponding to the 2D image are input into a neural network. The neural network includes a compression module and an expansion module. The compression module includes multiple downsampling structures, and the expansion module includes multiple upsampling structures corresponding to the multiple downsampling structures. The number of upsampling layers is the same as the number of upsampling structures. This allows an implicit representation of a 3D image to be generated based on the combination of the output information of the compression module and the output information of the expansion module. Based on the implicit representation of the 3D image, in step S103, 2D images and disparity maps for different viewpoints are generated through rendering.

[0050] Compared with conventional technologies based on converting two-dimensional images into three-dimensional images, the method for generating multi-viewpoint images provided in the embodiments of the present application uses a neural network to generate an implicit representation of a three-dimensional image based on a two-dimensional image and a disparity map corresponding to the two-dimensional image, and on this basis generates two-dimensional images and disparity maps for different viewpoints through rendering. This allows the three-dimensional display device to output three-dimensional content that is more consistent with the real depth of field relationship, promotes consistency in the generation results of different perspectives, and at the same time has the characteristics of flexibility and is applicable to three-dimensional display devices with different numbers of viewpoints.

[0051] According to some embodiments of the present application, the compression module of the neural network further includes a multi-layer two-dimensional convolution structure, and the expansion module further includes a multi-layer three-dimensional convolution structure, each downsampling structure is located between the two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, and the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first upsampling structure is located between the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, and the second upsampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure. In other words, each layer of two-dimensional convolution structure forms a cascaded multi-layer two-dimensional convolution structure via multiple downsampling structures, each layer of three-dimensional convolution structure forms a cascaded multi-layer three-dimensional convolution structure via multiple second upsampling structures, and the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first three-dimensional convolution structure in the multi-layer three-dimensional convolution structure are connected via the first upsampling structure. The difference between convolutions of different dimensions mainly lies in the different degrees of freedom of the convolution kernel. For example, two-dimensional convolution can perform sliding window operations in the width and height directions, while three-dimensional convolution can perform sliding window operations in the width, height, and channel directions. That is, two-dimensional convolution is a sliding window operation on a frame of a single channel image, and the input data is a two-dimensional matrix. The input of three-dimensional convolution is a three-dimensional matrix, which adds a channel dimension (depth) compared to two-dimensional convolution. In three-dimensional convolution, the filter can move in three directions (image width, height, and channel). Because the filter slides through a three-dimensional space, the output value is also a three-dimensional data. Three-dimensional convolution is widely used in scenarios such as video classification and image segmentation. After receiving the two-dimensional image and the corresponding disparity map, the compression module can output the corresponding first feature map from each layer of the two-dimensional convolution structure. The downsampling structure may include a pooling layer to reduce the height and width of the first feature map output by each layer of the two-dimensional convolution structure. The first upsampling structure or the second upsampling structure may include a deconvolution layer, a bilinear interpolation layer, or a depooling layer, which can achieve amplification of the second feature map output by the three-dimensional convolution structure, as well as a method for amplifying the first feature map output by the last layer of two-dimensional convolution structure in a multi-layer two-dimensional convolution structure.As shown in Figure 5, in some embodiments, the above-mentioned step S102 - inputting the two-dimensional image and the disparity map into the neural network to obtain an implicit representation of the three-dimensional image may include the following steps: S501, providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure; S502, fusing the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map, and the upsampling feature map output by the corresponding upsampling structure has the same size as the first feature map; S503, providing the fused feature map to the three-dimensional convolution structure of the corresponding layer to output a corresponding second feature map; and S504, obtaining an implicit representation of the three-dimensional image based on the second feature map output by the last layer of the three-dimensional convolution structure in the multi-layer three-dimensional convolution structure.

[0052] Each layer of the two-dimensional convolution structure in the compression module forms a cascaded multi-layer two-dimensional convolution structure via multiple downsampling structures. Each layer of the three-dimensional convolution structure in the expansion module forms a cascaded multi-layer three-dimensional convolution structure via multiple second upsampling structures. The first feature map output by each layer of the two-dimensional convolution structure is fused with the upsampled feature map output by the corresponding upsampling structure between each layer of the three-dimensional convolution structure to generate a fused feature map. The three-dimensional convolution structure of the corresponding layer in the expansion module outputs a corresponding second feature map based on the fused feature map. The second feature map output by the last layer of the three-dimensional convolution structure in the multi-layer three-dimensional convolution structure can be used as an implicit representation of the three-dimensional image. The fused feature map can be formed by superimposing the first feature map and the upsampled feature map in a depth direction that is different from the width and height directions of the feature map. Therefore, each layer of the three-dimensional convolution structure can receive a corresponding different fused feature map. Based on the fused feature map and the processing of the fused feature map by the three-dimensional convolution structure, the quality of the implicit representation of the generated three-dimensional image can be further improved, which is more conducive to the three-dimensional display device outputting three-dimensional content that is more consistent with the real depth of field relationship.

[0053] In some embodiments, the above-mentioned step S501 - providing the two-dimensional image and the disparity map to the compression module to output the corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure includes: using each downsampling structure of the multiple downsampling structures to downsample the first feature map output by the previous two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to obtain a downsampled feature map; and inputting the downsampled feature map into the next two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to output the corresponding first feature map.

[0054] In some embodiments, the upsampled feature map includes a first upsampled feature map output by the first upsampled structure and a second upsampled feature map output by the second upsampled structure. At this time, the above-mentioned step S102 - inputting the two-dimensional image and the disparity map into the neural network to obtain an implicit representation of the three-dimensional image may also include: using the first upsampled structure among the multiple upsampled structures to upsample the first feature map output by the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure to obtain the first upsampled feature map; and using the second upsampled structure to upsample the second feature map output by the previous three-dimensional convolution structure in the three-dimensional convolution structure of two adjacent layers to obtain the second upsampled feature map.

[0055] Therefore, in some embodiments, as shown in FIG6 , the above-mentioned steps of inputting the two-dimensional image and the disparity map into the neural network to obtain the implicit representation of the three-dimensional image may be summarized as follows: S601, using each downsampling structure of the multiple downsampling structures to downsample the first feature map output by the previous two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to obtain a downsampled feature map; S602, inputting the downsampled feature map into the next two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to output the corresponding first feature map; S603, using the first upsampling structure of the multiple upsampling structures to upsample the first feature map output by the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure to obtain a first Upsampling feature map; S604, using the second upsampling structure to upsample the second feature map output by the previous three-dimensional convolution structure in the two adjacent layers of the three-dimensional convolution structure to obtain a second upsampling feature map; S605, fusing the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map, wherein the upsampling feature map output by the corresponding upsampling structure has the same size as the first feature map; S606, providing the fused feature map to the three-dimensional convolution structure of the corresponding layer to output the corresponding second feature map; and S607, obtaining an implicit representation of the three-dimensional image based on the second feature map output by the last three-dimensional convolution structure in the multi-layer three-dimensional convolution structure. In some embodiments, the last three-dimensional convolution structure outputs multiple discrete planar images, i.e., multi-plane images MPI, as implicit representations of the three-dimensional image.

[0056] In some embodiments, the above-mentioned fusion of the first feature map with the upsampled feature map output by the corresponding upsampled structure in the multiple upsampled structures to obtain a fused feature map may include the following situations: fusing the first feature map output by the previous two-dimensional convolution structure of the last layer of the two-dimensional convolution structure with the first upsampled feature map to obtain a fused feature map provided to the first three-dimensional convolution structure of the multi-layer three-dimensional convolution structure; and fusing the second upsampled feature map with the first feature map output by the previous two-dimensional convolution structure of the first downsampled structure corresponding to the second upsampled structure to obtain a fused feature map provided to the next three-dimensional convolution structure of the second upsampled structure. In other words, the form of the feature map provided to the first upsampled structure in the multiple upsampled structures may be different from the form of the feature map provided to the second upsampled structure. The first upsampled structure outputs the first upsampled feature map based on the first feature map output by the two-dimensional convolution structure of the last layer of the multi-layer two-dimensional convolution structure, and each second upsampled structure outputs the corresponding second upsampled feature map based on the second feature map output by the previous three-dimensional convolution structure adjacent to the upsampled structure.

[0057] In some embodiments, fusing the first feature map with the upsampled feature map output by the corresponding upsampled structure in the multiple upsampled structures to obtain a fused feature map includes: superimposing the first feature map and the upsampled feature map in a direction along the depth dimension. The direction of the depth dimension mentioned here refers to another dimensional direction different from the width and height. For example, assuming that the first feature map and the upsampled feature map are both two-dimensional matrices of a×b, then the fused feature map obtained based on the first feature map and the upsampled feature map can be two two-dimensional matrices superimposed on each other, thereby obtaining a three-dimensional matrix of a×b×2, that is, the width and height of the fused feature map are still a and b respectively, but the depth dimension becomes 2. Similarly, in the case where the first feature map and the upsampled feature map are both three-dimensional matrices, the fused feature map obtained based on the first feature map and the upsampled feature map can be two three-dimensional matrices superimposed on each other in another depth direction different from the length, width and height of the three-dimensional matrix. The fused feature map has an additional dimension compared to the first feature map or the upsampled feature map.

[0058] Below, the model of the neural network in the embodiment of the present application is schematically illustrated with reference to Figure 7. As shown in Figure 7, a neural network including a multi-layer two-dimensional convolution structure, a multi-layer three-dimensional convolution structure, a plurality of upsampling structures and a plurality of downsampling structures can form a convolutional neural network that is approximately U-shaped. The input of the convolutional neural network is a two-dimensional image (for example, an RGB image) and a disparity map corresponding to the two-dimensional image, and the output of the convolutional neural network is an implicit representation of a three-dimensional image, such as a multi-plane image (MPI). The resolution of the two-dimensional image provided to the convolutional neural network is h×w, and the output is a set of discrete planes. In the example of Figure 7, the blue arrow represents a two-dimensional convolution structure, the yellow arrow represents a three-dimensional convolution structure, the red arrow represents a downsampling structure, the green arrow represents an upsampling structure, and the gray arrow represents a fusion operation. Figure 7 schematically shows a four-layer two-dimensional convolution structure, a three-layer three-dimensional convolution structure, three upsampling structures and three downsampling structures. As shown in Figure 7, each layer of two-dimensional convolutional structure may include two two-dimensional convolutions, and each layer of three-dimensional convolutional structure may include two three-dimensional convolutions. The initial data provided to the convolutional neural network includes four channels. The first two-dimensional convolution in each layer of two-dimensional convolutional structure increases the channel data, and the second two-dimensional convolution does not change the size of the feature map output by the first two-dimensional convolution. In the example of Figure 7, the compression module of the neural network includes three downsampling structures, and correspondingly, the expansion module includes three upsampling structures, thereby ensuring that the resolution of the feature map output by the expansion module is the same as the resolution of the input image provided to the compression module. The compression module also includes four layers of two-dimensional convolutional structures, each downsampling structure located between the two-dimensional convolutional structures of adjacent layers in the four-layer two-dimensional convolutional structure. Each downsampling structure downsamples the first feature map output by the previous two-dimensional convolutional structure in the two adjacent layers of the two-dimensional convolutional structure to obtain a downsampled feature map. The downsampled feature map is input into the next two-dimensional convolutional structure in the two adjacent layers of the two-dimensional convolutional structure to output the corresponding first feature map. The expansion module also includes a three-layer three-dimensional convolutional structure. The three upsampling structures include one first upsampling structure and two second upsampling structures. The first upsampling structure is located between the last two-dimensional convolutional structure in the four-layer two-dimensional convolutional structure and the first three-dimensional convolutional structure in the three-layer three-dimensional convolutional structure. Each second upsampling structure is located between the three-dimensional convolutional structures of adjacent layers in the multi-layer three-dimensional convolutional structure. The first upsampling structure upsamples the first feature map output by the last two-dimensional convolutional structure in the four-layer two-dimensional convolutional structure to obtain a first upsampled feature map. Each second upsampling structure upsamples the second feature map output by the previous three-dimensional convolutional structure in two adjacent three-dimensional convolutional layers to obtain a second upsampled feature map.7 , the first feature map output by the preceding two-dimensional convolution structure of the last two-dimensional convolution structure in the four-layer two-dimensional convolution structure (i.e., the third-layer two-dimensional convolution structure in FIG7 ) is fused with the first up-sampled feature map output by the first up-sampling structure (as shown by the green arrow at the bottom of FIG7 ) (as shown by the gray arrow at the bottom of FIG7 ) to obtain a fused feature map provided to the first three-dimensional convolution structure in the three-layer three-dimensional convolution structure. The second up-sampled feature map output by the second up-sampling structure (e.g., the middle green arrow in FIG7 ) is fused with the first feature map output by the preceding two-dimensional convolution structure of the first down-sampling structure (as shown by the middle red arrow in FIG7 ) corresponding to the second up-sampling structure (as shown by the middle gray arrow in FIG7 ) to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second up-sampling structure (i.e., the second three-dimensional convolution structure in FIG7 ). Continuing to refer to Figure 7, each downsampling structure can reduce the size of the first feature map output by the previous two-dimensional convolution structure by half. Therefore, the size or resolution of the downsampled feature map provided to the last layer of two-dimensional convolution structure (i.e., the bottom two-dimensional convolution structure in Figure 7) is one-eighth of the input two-dimensional image. In the expansion module, the fused feature map received by the three-dimensional convolution structure has an increased dimension in the depth direction compared to the first feature map output by the two-dimensional convolution structure. Therefore, the feature map in the expansion module shown on the right side of Figure 7 is illustrated as a three-dimensional structure. For the last layer of two-dimensional convolution structure, the first feature map it outputs is not fused with other feature maps, so it still does not have a dimension in the depth direction, or, the dimension of the first feature map in the depth direction can be considered to be 1. As shown in Figure 7, the first feature map output by the first layer of two-dimensional convolution structure includes 64 channels and has a resolution of h×w. After the first downsampling structure is downsampled, the resolution of the first feature map becomes half of the previous one, i.e., . Correspondingly, the resolution of the first feature map output by the second-layer two-dimensional convolutional structure is The number of channels increases to 128. After the downsampling process of the three downsampling structures, the resolution of the first feature map output by the last layer of two-dimensional convolution structure is However, after the two-dimensional convolution operation of multiple layers of two-dimensional convolution structures, the first feature map output by the last layer of two-dimensional convolution structure includes 512 channels. The first feature map output by each layer of two-dimensional convolution structure does not have a dimension in the depth direction, or in other words, the dimension of the first feature map output by each layer of two-dimensional convolution structure in the depth direction is 1. In order to illustrate this more clearly, as shown in Figure 7, the first feature map output by the last layer of two-dimensional convolution structure is marked as "512×1". The first upsampling structure in the expansion module receives the first feature map output by the last layer of two-dimensional convolution structure, and expands the size of the first feature map by two times through upsampling (for example, deconvolution), and reduces the number of channels by half, thereby obtaining a channel of 256 and a size or resolution of The first up-sampled feature map is the same size as the first feature map output by the previous two-dimensional convolution structure of the last two-dimensional convolution structure. The two are fused to obtain a fused feature map with 256 channels and 2 dimensions in the depth direction. The fused feature map is marked as "256×2" in Figure 7. Similarly, the second up-sampling structure can double the size of the second feature map output by the first three-dimensional convolution structure through deconvolution and reduce the number of channels by half, thereby obtaining a channel of 128 and a size or resolution of The upsampled feature map is the same size as the first feature map output by the second-layer two-dimensional convolution structure. The two are fused to obtain a fused feature map with 128 channels and a dimension of 3 in the depth direction. The fused feature map is identified as "128×3" in Figure 7. For the last layer of three-dimensional convolution structure, the size of the fused feature map it receives is the same as the size of the two-dimensional image and the corresponding disparity map provided to the neural network. The dimension of the fused feature map in the depth direction is 4. The fused feature map is identified as "64×4" in Figure 7. The parameters of the three-dimensional convolution structure can be set so that the number of channels of the feature map output by the last layer of three-dimensional convolution structure is the number of layers N of the planar image of the multi-plane image. In other words, the number of layers of the planar image of the multi-plane image (MPI) output by the convolutional neural network can be set as needed, and the embodiments of the present application do not limit this. In addition, FIG7 only gives an example of a neural network model, wherein the number of layers of the two-dimensional convolution structure, the number of layers of the three-dimensional convolution structure, the number of two-dimensional convolutions in each layer of the two-dimensional convolution structure, and the number of three-dimensional convolutions in each layer of the three-dimensional convolution structure are only exemplary and do not constitute a limitation on the scope of protection of this application. The upsampling structure (for example, deconvolution) can play a role in expanding the feature map, but it cannot completely restore the feature map. Through the above-mentioned fusion operation, the features processed by the downsampling structure corresponding to the upsampling structure can be superimposed on each other, reducing data loss. In addition, this fusion operation allows deep information and shallow information to be combined. The deep information includes low-resolution information after the input image has been downsampled multiple times by multiple downsampling structures, and the shallow information includes high-resolution information transferred from the compression module to the expansion module through the fusion operation, which can provide more refined features.

[0059] In some embodiments, the implicit representation of a 3D image includes a multi-plane image (MPI). A multi-plane image uses multiple RGBA planes within the camera's view frustum to represent the scene, supporting differentiable rendering. Figure 8 schematically illustrates a multi-plane image. A multi-plane image consists of multiple RGBA images located on multiple parallel planes. These RGBA images are located in different layers, also known as multi-layer RGBA images. Each layer of RGBA images has different parallaxes and can be evenly spaced in space. For each layer of RGBA images, each plane can encode two parameters: the color C of an RGB image and a transparency α. That is, each RGBA image layer can be represented as (Ci, αi), where Ci and αi represent the color channel and alpha channel of the RGBA image in the i-th layer, respectively. A multi-plane image is a collection of RGBA images. In computer graphics, true color graphics in the RGB color model are synthesized from three color channels: red, green, and blue. To achieve transparency, another channel of information is added during image processing and storage. This additional channel representing the transparency of each pixel in the image is called the alpha channel. For example, using an 8-bit binary number, the alpha channel can represent 256 levels of grayscale, or 256 levels of transparency. Each layer of RGBA images in the MPI is fixed to a certain parallax, and the alpha channel of each layer of RGBA images represents its transparency or visibility. Figure 8 also schematically illustrates an example of a reference viewpoint 2D image at a reference viewpoint and a target viewpoint 2D image at a target viewpoint. Given a multi-plane image (MPI), a reference viewpoint, and a target viewpoint, these reference viewpoint 2D images and target viewpoint 2D images can be generated through rendering.

[0060] As mentioned above, each viewpoint can be understood as corresponding to a virtual camera. Accordingly, the two-dimensional image provided to the neural network can be regarded as being taken by a virtual camera corresponding to the reference viewpoint. That is, the two-dimensional image provided to the neural network can correspond to the reference viewpoint. The aforementioned different viewpoints can include the reference viewpoint corresponding to the two-dimensional image and other target viewpoints different from the reference viewpoint. Therefore, based on the obtained three-dimensional image implicit representation rendering, a reference viewpoint two-dimensional image (RGB image) and a corresponding disparity map as well as a target viewpoint two-dimensional image (RGB image) and a corresponding disparity map can be generated.

[0061] Different algorithms in this field can be used to render a two-dimensional image of the corresponding viewpoint based on a multi-plane image (MPI). The main process of rendering is actually to calculate the RGB value of each pixel on the virtual camera imaging plane corresponding to the target viewpoint. For example, for any pixel X on the virtual camera imaging plane, the calculation method for obtaining its RGB value can be: connect the optical center of the virtual camera and the pixel X to form a straight line, and the straight line passes through each layer of RGBA image in the multi-plane image, and determine the intersection of the straight line and each layer of RGBA image. Therefore, each intersection will correspond to a color channel Ci of an RGBA image, that is, an RGB value. Assuming that the multi-plane image includes N layers of RGBA images, the above straight line will form N intersections with the multi-plane image, then the RGB value of pixel X is the weighted sum of the RGB values ​​of the N intersections, and different weights can be set for the RGB values ​​of different intersections. For example, assuming that the RGB value of each intersection is Y i , then the RGB value of pixel X where ρ i Represents the weight coefficient for the RGB value of each intersection. Based on the two-dimensional image of the corresponding viewpoint, a monocular depth estimation network can be used to estimate the disparity map of the two-dimensional image.

[0062] The neural network mentioned in the above embodiments of the present application can be obtained by training with training data. Another embodiment of the present application provides a method for obtaining training data for a neural network, which includes but is not limited to the convolutional neural network described in the previous embodiments. Figure 9 illustrates some steps included in an embodiment of a method for obtaining training data for a neural network. As shown in Figure 9, the method for obtaining training data for a neural network may include the following steps: Step S901, obtaining multiple reference viewpoint two-dimensional images corresponding to a reference viewpoint and a reference viewpoint disparity map corresponding to each of the multiple reference viewpoint two-dimensional images. The multiple reference viewpoint two-dimensional images can be regarded as multiple two-dimensional images captured by a virtual camera at the reference viewpoint. Similarly, a monocular depth estimation network can be used to estimate the reference viewpoint disparity map corresponding to each of the multiple reference viewpoint two-dimensional images. The disparity map is a single-channel image. The resolution of the disparity map can be the same as the resolution of the corresponding two-dimensional image. The value of each pixel in the disparity map represents the disparity at that pixel. Step S902: Determine a viewpoint conversion matrix and a target viewpoint for each reference viewpoint 2D image, wherein the viewpoint conversion matrix represents the spatial position offset of the target viewpoint relative to the reference viewpoint. Step S903: Determine a target viewpoint 2D image corresponding to the target viewpoint based on the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint 2D image, and camera intrinsic parameters. Step S904: Obtain a target viewpoint disparity map corresponding to the target viewpoint 2D image. Similarly, the target viewpoint disparity map corresponding to the target viewpoint 2D image can be estimated using a monocular depth estimation network. And step S905: Generate training data based on the reference viewpoint 2D image, the reference viewpoint disparity map, the target viewpoint 2D image, and the target viewpoint disparity map. That is, for each reference viewpoint 2D image, a corresponding set of training data can be obtained through the above steps. This set of training data includes the reference viewpoint 2D image, the reference viewpoint disparity map corresponding to the reference viewpoint 2D image, the target viewpoint 2D image, and the target viewpoint disparity map corresponding to the target viewpoint 2D image. Based on multiple reference viewpoint two-dimensional images, multiple groups of similar training data can be obtained, thereby enabling effective training of the neural network.

[0063] In some cases, the target viewpoint 2D image may contain content that is invisible to the reference viewpoint 2D image. That is, the target viewpoint 2D image may contain holes, and accordingly, the target viewpoint disparity map may also contain holes. Therefore, in some embodiments, step S905 may include repairing holes in the target viewpoint 2D image and the target viewpoint disparity map to obtain a repaired target viewpoint 2D image and a repaired target viewpoint disparity map, and using the reference viewpoint 2D image, the reference viewpoint disparity map, the repaired target viewpoint 2D image, and the repaired target viewpoint disparity map as training data. This improves the sample quality of the training data. This document does not limit the specific method for hole repair; either deep learning-based or non-deep learning methods, such as the PatchMatch algorithm, can be used. The PatchMatch algorithm was developed for image editing and fills the boundary area by finding the closest match to the edge portion, thereby achieving a natural effect. The core purpose of this algorithm is to quickly find corresponding small areas between two images. In application, the patchmatch algorithm can be combined with image reconstruction and other technologies to achieve functions such as image restoration, image fusion, and watermark removal.

[0064] Similar to the previous embodiment, in the above step S901, the reference viewpoint disparity map can be generated based on the reference viewpoint two-dimensional image using a monocular depth estimation network. In some embodiments, the above step S902 - determining the viewpoint conversion matrix and the target viewpoint for each reference viewpoint two-dimensional image may include: randomly determining a rotation vector and a translation vector for the reference viewpoint corresponding to the reference viewpoint two-dimensional image, the rotation vector representing the Euler angle offset of the target viewpoint relative to the reference viewpoint, and the translation vector representing the coordinate offset of the target viewpoint relative to the reference viewpoint. The above step S903 - determining the target viewpoint two-dimensional image corresponding to the target viewpoint based on the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image, and the camera intrinsic parameters may include: determining the pixel value of each pixel of the target viewpoint two-dimensional image based on the viewpoint conversion matrix, the reference viewpoint disparity map, the pixel value of each pixel of the reference viewpoint two-dimensional image, and the camera intrinsic parameters.

[0065] It can be understood that for each reference viewpoint two-dimensional image among the multiple reference viewpoint two-dimensional images, different viewpoint transformation matrices and corresponding multiple target viewpoints can be determined respectively. At this time, the method for obtaining training data for a neural network may include: determining multiple target viewpoint two-dimensional images corresponding to the multiple target viewpoints respectively according to each reference viewpoint two-dimensional image, the camera intrinsic parameters, the viewpoint transformation matrix corresponding to each reference viewpoint two-dimensional image, and the reference viewpoint disparity map; obtaining multiple target viewpoint disparity maps corresponding to the multiple target viewpoint two-dimensional images; and forming multiple groups of training data based on the multiple reference viewpoint two-dimensional images, the multiple reference viewpoint disparity maps, the multiple target viewpoint two-dimensional images and the multiple target viewpoint disparity maps, each group of training data including one reference viewpoint two-dimensional image among the multiple reference viewpoint two-dimensional images, and the corresponding reference viewpoint disparity map, the target viewpoint two-dimensional image and the target viewpoint disparity map.

[0066] The following further describes steps S902 and S903 using examples. The viewpoint transformation matrix and target viewpoint can be further illustrated using the different viewpoints shown in Figure 4. Referring back to Figure 4, if viewpoint V0 is considered the reference viewpoint and viewpoints V1, V2, V3, and V4 are the target viewpoints, the viewpoint transformation matrix represents the spatial position offset of target viewpoint V1, V2, V3, or V4 relative to reference viewpoint V0. Since each viewpoint corresponds to a virtual camera, the spatial position offset is actually the spatial position offset of the virtual camera at the target viewpoint relative to the virtual camera at the reference viewpoint. The spatial position offset may include two vectors: the rotation and translation of the virtual camera. In some embodiments, the rotation vector R can be generated by sampling a random Euler angle. Euler angles are a method for describing the rotation of an object in three-dimensional space. They use three angles to represent the rotation, typically denoted as yaw, pitch, and roll. The yaw angle represents the rotation of an object around the vertical axis, the pitch angle represents the rotation of an object around the horizontal axis, and the roll angle represents the rotation of an object around the fore-aft axis. In an aircraft, the yaw angle represents the angle at which the aircraft turns left or right, the pitch angle represents the angle at which the aircraft rolls up and down, and the roll angle represents the angle at which the aircraft rolls sideways. At the same time, a translation vector t is generated by sampling a random three-dimensional vector to obtain the viewpoint transformation matrix (R|t). Assuming that the Euler angle triplet sampled at a certain time is (α, β, γ), that is, the rotation angles of the virtual camera around the X, Y, and Z axes are α, β, and γ respectively, the rotation R vector can be generated according to the following formula: R = R z (γ)*R y (β)*R x (α) (Formula 2)

[0067] in:

[0068] The above-mentioned translation vector t can be expressed as t = (x, y, z), which represents the translation of the virtual camera relative to the reference viewpoint, and x, y, z represent the translation of the virtual camera on the X, Y, and Z axes respectively. The values ​​of x, y, and z can be randomly sampled within a preset range. The viewpoint conversion matrix T = (R|t) can be obtained by splicing the rotation vector R and the translation vector t. For example, the rotation vector R can be a 3×3 matrix, t can be a 3×1 matrix, and the viewpoint conversion matrix T after splicing the rotation vector R and the translation vector t is a 3×4 matrix. Therefore, the viewpoint conversion matrix T can characterize the spatial position offset of the target viewpoint relative to the reference viewpoint. In step S903, the target viewpoint two-dimensional image corresponding to the target viewpoint is determined according to the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image and the camera internal parameters. The process of determining the target viewpoint two-dimensional image is actually the process of determining the RGB values ​​of each pixel of the target viewpoint two-dimensional image. In some embodiments, the target viewpoint two-dimensional image corresponding to the target viewpoint can be determined according to the following formula. tgt ~KT src→tgt D src (p src )K -1 p src (Formula 4)

[0069] Among them, p tgt Represents the pixel RGB value of the target viewpoint two-dimensional image, p src Represents the pixel RGB value of the reference viewpoint 2D image, D src (psrc) represents the disparity of the corresponding pixel in the reference viewpoint disparity map, and K represents the camera intrinsic parameter. The camera intrinsic parameter K can be the actual intrinsic parameter of a conventional camera in the field. Intrinsic parameters are parameters that describe the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc., which characterize the laws of camera imaging. Once the camera intrinsic parameters are determined, they generally remain unchanged during the use of the camera. The camera intrinsic parameter K can be expressed by the following formula 5.

[0070] Assume that the focal length of the camera is f, the pixels on the imaging plane are rectangular, and the pixel size in the x direction (horizontal direction) (i.e., the physical size of a pixel) is d x , the pixel size in the y direction (vertical direction) is d y , then the parameter f in the above formula is x 、f y Can be respectively Parameter f x 、f y Indicates the conversion of the focal length f in physical size to the focal length in pixels. x,c y In some embodiments, the camera intrinsic parameter K is set as:

[0071] It can be understood that for different target viewpoints, different viewpoint transformation matrices T can be set, but the same camera intrinsic parameters can be used.

[0072] Figure 10a schematically shows an example of a reference viewpoint two-dimensional image, a reference viewpoint disparity map, a target viewpoint two-dimensional image, and a target viewpoint disparity map. As shown in Figure 10a, the two pictures on the right side of Figure 10a respectively represent the reference viewpoint two-dimensional image (elephant) captured by the virtual camera at the reference viewpoint and the corresponding reference viewpoint disparity map obtained based on the reference viewpoint two-dimensional image. The virtual camera is moved to the target viewpoint using the viewpoint transformation matrix T, and the target viewpoint two-dimensional image and the target viewpoint disparity map obtained based on the target viewpoint two-dimensional image are captured at the target viewpoint. The two pictures on the left side of Figure 10a respectively represent the target viewpoint two-dimensional image and the target viewpoint disparity map. Figure 10b schematically shows a comparison of the target viewpoint two-dimensional image and the target viewpoint disparity map before and after restoration. The figure on the right side of Figure 10b shows the restored target viewpoint two-dimensional image and the restored target viewpoint disparity map. The figure on the left side of Figure 10b shows the target viewpoint two-dimensional image and the target viewpoint disparity map before restoration.

[0073] Another embodiment of the present application provides a method for training a neural network using training data obtained in the aforementioned method for obtaining training data. The neural network may be a convolutional neural network as described in the aforementioned embodiment. The training data may be provided in the form of training data sets, each set of training data including a reference viewpoint 2D image, a reference viewpoint disparity map corresponding to the reference viewpoint 2D image, a target viewpoint 2D image, and a target viewpoint disparity map corresponding to the target viewpoint 2D image. It may also include a viewpoint transformation matrix corresponding to the target viewpoint. As shown in FIG11 , the method for training a neural network may include the following steps: S1101, providing the reference viewpoint 2D image and the reference viewpoint disparity map to the neural network to generate an implicit representation of a 3D image. The implicit representation of the 3D image may include a multi-plane image MPI, i.e., a collection of N-layer RGBA images, each plane image having a color channel Ci and an alpha channel. S1102, based on the implicit representation of the 3D image, rendering generates a 2D image of the reference viewpoint target and a 2D image of the target viewpoint target at the reference viewpoint and the target viewpoint, respectively. FIG12 schematically illustrates the training of the convolutional neural network shown in FIG7 . By providing a reference viewpoint two-dimensional image and a reference viewpoint disparity map to a convolutional neural network, a multi-plane image MPI is generated at the output end of the neural network. S1103, based on the two-dimensional image of the reference viewpoint mark and the two-dimensional image of the target viewpoint mark, a disparity map of the reference viewpoint mark and a disparity map of the target viewpoint mark are obtained respectively; S1104, based on the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint mark, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint mark, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint mark, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint mark; and S1105, using the training data to train the neural network until the value of the loss function is less than a threshold. In the above-mentioned step S1102, the position of the target viewpoint, that is, the orientation of the virtual camera corresponding to the target viewpoint, can be determined based on the reference viewpoint and the viewpoint transformation matrix, so that a two-dimensional image captured by the virtual camera at the target viewpoint can be generated based on the implicit representation rendering of the three-dimensional image (hereinafter referred to as the two-dimensional image of the target viewpoint mark). At the same time, a two-dimensional image captured by the virtual camera at the reference viewpoint can be generated based on the implicit representation rendering of the three-dimensional image (hereinafter referred to as the two-dimensional image of the reference viewpoint mark). The rendering method used here can be similar to the rendering process described above and will not be repeated here. Similarly, the method for generating a disparity map mentioned in the above embodiment can be used to obtain a disparity map of the reference viewpoint mark and a disparity map of the target viewpoint mark based on the two-dimensional image of the reference viewpoint mark and the two-dimensional image of the target viewpoint mark, respectively.In step S1104, the difference between the reference viewpoint two-dimensional image and the reference target two-dimensional image, the difference between the target viewpoint two-dimensional image and the target viewpoint target two-dimensional image, the difference between the reference viewpoint disparity map and the reference viewpoint target disparity map, and the difference between the target viewpoint disparity map and the target viewpoint target disparity map can be determined first, and then the overall loss function for the neural network can be determined based on the above-mentioned differences. These differences can also be measured using appropriate loss functions in the art. For example, the absolute value loss function (L1 loss function) or the mean square error loss function (L2 loss function) can be used to calculate the above-mentioned differences. In step S1105, a threshold can be set for the loss function, and the neural network can be trained using multiple sets of training data. When the value of the loss function is less than the above-mentioned threshold, the training process is stopped to obtain a trained neural network.

[0074] In some embodiments, the above-mentioned step S1104 - determining the loss function based on the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint mark, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint mark, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint mark, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint mark may include: respectively determining the loss function based on the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint mark, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint mark, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint mark, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint mark The difference between the disparity maps of the point marks determines the first loss subfunction Ls1, the second loss subfunction Ls2, the third loss subfunction Ls3 and the fourth loss subfunction Ls4, that is, the first loss subfunction Ls1 represents the difference between the reference viewpoint two-dimensional image and the reference viewpoint mark two-dimensional image, the second loss subfunction Ls2 represents the difference between the target viewpoint two-dimensional image and the target viewpoint mark two-dimensional image, the third loss subfunction Ls3 represents the difference between the reference viewpoint disparity map and the reference viewpoint mark disparity map, and the fourth loss subfunction Ls4 represents the difference between the target viewpoint disparity map and the target viewpoint mark disparity map; the sum of the first loss subfunction, the second loss subfunction, the third loss subfunction and the fourth loss subfunction is determined as the loss function. That is, the loss function for the neural network is Ls=Ls1+Ls2+Ls3+Ls4. The first loss subfunction, the second loss subfunction, the third loss subfunction and the fourth loss subfunction can adopt the absolute value loss function (L1 loss function) or the mean square error loss function (L2 loss function).

[0075] Another embodiment of the present application provides an apparatus for generating multi-view images. As shown in FIG13 , the apparatus 1300 includes: an image acquisition module 1310 configured to acquire a two-dimensional image and obtain a corresponding disparity map based on the two-dimensional image; a three-dimensional image implicit representation generation module 1320 configured to generate a three-dimensional image implicit representation based on the two-dimensional image and the disparity map, wherein the three-dimensional image implicit representation generation module includes a compression module and an expansion module, wherein the compression module includes multiple downsampling structures, and the expansion module includes multiple upsampling structures corresponding to the multiple downsampling structures; and a rendering module 1330 configured to generate two-dimensional images and disparity maps for different viewpoints based on the three-dimensional image implicit representation. Each module in the apparatus for generating multi-view images corresponds to each step in the above-mentioned method for generating multi-view images. These modules can be implemented in software programming, hardware circuits, or a combination of software programming and hardware circuits. The device uses a neural network to generate an implicit representation of a three-dimensional image based on a two-dimensional image and the disparity map corresponding to the two-dimensional image. On this basis, two-dimensional images and disparity maps for different viewpoints are generated through rendering. This allows the three-dimensional display device to output three-dimensional content that is more consistent with the real depth of field relationship. At the same time, it has the characteristics of flexibility and can be applied to three-dimensional display devices with different numbers of viewpoints.

[0076] In some embodiments, the compression module in the device further includes a multi-layer two-dimensional convolution structure, and the expansion module further includes a multi-layer three-dimensional convolution structure, each downsampling structure is located between the two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, and the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first sampling structure is located between the last layer of two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, and the second upsampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure, wherein the compression module is configured to receive the two-dimensional image and the disparity map to output a corresponding first feature map from each layer of two-dimensional convolution structure in the multi-layer two-dimensional convolution structure, wherein the three-dimensional image implicit representation generation module further includes a feature map fusion module, and the feature map fusion module is configured to fuse the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map, and the upsampling feature map output by the corresponding upsampling structure has the same size as the first feature map. The 3D convolution structure of the corresponding layer in the multi-layer 3D convolution structure receives the fused feature map and outputs a corresponding second feature map. The expansion module obtains an implicit representation of the 3D image based on the second feature map output by the 3D convolution structure of the last layer in the multi-layer 3D convolution structure. The fused feature map generated by the feature map fusion module and the processing of the fused feature map by the 3D convolution structure can further improve the quality of the implicit representation of the generated 3D image, further facilitating the 3D display device to output 3D content that better conforms to realistic depth of field relationships.

[0077] In some embodiments, the compression module provides the two-dimensional image and the disparity map to the compression module to output the corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure, which specifically may include: using each downsampling structure in the multiple downsampling structures to downsample the first feature map output by the previous two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to obtain a downsampled feature map; and inputting the downsampled feature map into the next two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to output the corresponding first feature map. In some embodiments, the upsampled feature map output by the upsampling structure in the expansion module includes the first upsampled feature map output by the first upsampling structure and the second upsampled feature map output by the second upsampling structure. The expansion module is further configured to use the first upsampling structure in the multiple upsampling structures to upsample the first feature map output by the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure to obtain the first upsampled feature map; and use the second upsampling structure to upsample the second feature map output by the previous three-dimensional convolution structure in the three-dimensional convolution structure of the adjacent two layers to obtain the second upsampled feature map.

[0078] In some embodiments, the feature map fusion module fuses the first feature map with the upsampling feature map output by the corresponding upsampling structure in the multiple upsampling structures to obtain a fused feature map, including: fusing the first feature map output by the previous two-dimensional convolution structure of the last layer of the two-dimensional convolution structure with the first upsampling feature map to obtain a fused feature map provided to the first layer of three-dimensional convolution structure in the multi-layer three-dimensional convolution structure; and fusing the second upsampling feature map with the first feature map output by the previous two-dimensional convolution structure of the first downsampling structure corresponding to the second upsampling structure to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second upsampling structure.

[0079] In some embodiments, the feature map fusion module fuses the first feature map with the upsampled feature map output by the corresponding upsampled structure in the multiple upsampled structures to obtain a fused feature map, including: superimposing the first feature map and the upsampled feature map in a direction along the depth dimension. The implicit representation of the three-dimensional image includes a multi-plane image.

[0080] Another embodiment of the present application provides a computing device, which includes: a memory configured to store computer-executable instructions; a processor configured to perform the steps of the method described in any of the aforementioned embodiments of the method for generating multi-view images, the method for obtaining training data for a neural network, and the method for training a neural network using training data when the computer-executable instructions are executed by the processor.

[0081] In particular, the methods described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of the present application provide a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing at least one step of the method for generating multi-viewpoint images, the method for obtaining training data for a neural network, or the method for training a neural network using training data described in the above embodiments.

[0082] Another embodiment of the present application provides one or more computer-readable storage media having computer-readable instructions stored thereon, which, when executed, implement the method for generating multi-view images, the method example for obtaining training data for a neural network, or the method for training a neural network using training data according to some embodiments of the present application. The various steps of the method for generating multi-view images, the method example for obtaining training data for a neural network, or the method for training a neural network using training data can be converted into computer-readable instructions through programming and thus stored in a computer-readable storage medium. When such a computer-readable storage medium is read or accessed by a computing device or a computer, the computer-readable instructions therein are executed by a processor on the computing device or the computer to implement the method for generating multi-view images, the method example for obtaining training data for a neural network, or the method for training a neural network using training data.

[0083] FIG14 illustrates an exemplary system including a computing device 1410, which is an example computing device in one or more systems and / or devices that can implement the technical solutions described in various embodiments herein. Computing device 1410 can be, for example, a server of a service provider, a device associated with a server, a system on a chip, and / or any other suitable computing device or computing system. The apparatus for generating multi-viewpoint images described above with reference to FIG13 can take the form of computing device 1410. Alternatively, the apparatus for generating multi-viewpoint images can be implemented as a computer program in the form of application 1416.

[0084] As shown in Figure 14, the example computing device 1410 includes a processing system 1411, one or more computer-readable media 1412, and one or more I / O interfaces 1413 that are communicatively coupled to each other. Although not shown, the computing device 1410 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures.

[0085] Processing system 1411 represents the functionality of performing one or more operations using hardware. Thus, processing system 1411 is illustrated as including hardware elements 1414 that can be configured as processors, functional blocks, and the like. This can include hardware implementations as application-specific integrated circuits or other logic devices formed using one or more semiconductors. Hardware elements 1414 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can be comprised of (a plurality of) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions. Computer-readable media 1412 is illustrated as including memory / storage 1415. Memory / storage 1415 represents memory / storage capacity associated with one or more computer-readable media. Memory / storage 1415 can include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, and the like). Memory / storage 1415 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer-readable media 1412 may be configured in various other ways, as further described below. One or more I / O interfaces 1413 represent functionality that allows a user to input commands and information to the computing device 1410 using various input devices and, optionally, also allows information to be presented to the user and / or other components or devices using various output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can use visible or invisible wavelengths (such as infrared frequencies) to detect motion that does not involve touch as gestures), and the like. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, a tactile response device, and the like. Thus, the computing device 1410 may be configured in various ways, as further described below, to support user interaction.

[0086] The computing device 1410 also includes an application 1416. The application 1416 may be, for example, a software instance of the apparatus 1300 for generating multi-viewpoint images described with reference to FIG. 13 , and may implement the techniques described herein in combination with other elements in the computing device 1410.

[0087] Various techniques may be described herein in the general context of software, hardware, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, and the like that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," and "component" generally refer to software, firmware, hardware, or a combination thereof. The techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0088] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1410. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0089] As opposed to a mere signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device, and / or tangible storage device, capable of persistently storing information. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing desired information and accessible by a computer.

[0090] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 1410, such as via a network. Signal media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0091] As previously described, hardware elements 1414 and computer-readable media 1412 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or other hardware devices. In this context, hardware elements can be used as processing equipment for executing program tasks defined by the instructions, modules, and / or logic embodied by the hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media previously described.

[0092] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1414. Computing device 1410 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware elements 1414 of a processing system, a module may be implemented as a module that can be executed by computing device 1410 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles of manufacture (e.g., one or more computing devices 1410 and / or processing systems 1411) to implement the techniques, modules, and examples described herein.

[0093] In various implementations, computing device 1410 can be implemented in a variety of different configurations. For example, computing device 1410 can be implemented as a computer-type device including a personal computer, a desktop computer, a multi-screen computer, a laptop computer, a netbook, etc. Computing device 1410 can also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable gaming devices, tablet computers, multi-screen computers, etc. Computing device 1410 can also be implemented as a television-type device, which includes devices having or connected to generally larger screens in casual viewing environments. These devices include televisions, set-top boxes, game consoles, etc.

[0094] The technology described herein can be supported by these various configurations of computing device 1410 and is not limited to the specific examples of the technology described herein. Functionality can also be implemented in whole or in part on a "cloud" 1420 using a distributed system, such as through a platform 1422 as described below. Cloud 1420 includes and / or represents a platform 1422 for resources 1424. Platform 1422 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 1420. Resources 1424 may include other applications and / or data that can be used when performing computer processing on a server remote from computing device 1410. Resources 1424 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0095] The platform 1422 can abstract resources and functionality to connect the computing device 1410 with other computing devices. The platform 1422 can also be used to abstract hierarchies of resources to provide a corresponding level of hierarchy in the demand encountered for resources 1424 implemented via the platform 1422. Thus, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 1400. For example, functionality can be implemented partially on the computing device 1410 and through the platform 1422 that abstracts the functionality of the cloud 1420.

[0096] By using the method and apparatus for generating multi-viewpoint images provided in the embodiments of the present application, or a computing device, computer-readable medium, and computer program product capable of executing the aforementioned method, an implicit representation of a three-dimensional image (e.g., a multi-plane image MPI) can be generated based on the provided two-dimensional image, and two-dimensional images and disparity maps for different viewpoints can be rendered based on the implicit representation of the three-dimensional image. These two-dimensional images from different viewpoints can be regarded as images taken by virtual cameras at different positions and orientations. After obtaining these two-dimensional images from different viewpoints, they can be synthesized into a single image, and finally this synthesized image can be copied or sent to a three-dimensional display terminal to see the stereoscopic display effect. The total number of viewpoints, including reference viewpoints and target viewpoints, may vary with different stereoscopic display devices. The technical solution provided in the embodiments of the present application is not limited by the number of viewpoints required by the stereoscopic display device, and can be applied to stereoscopic display devices requiring different numbers of viewpoints.

[0097] Should be understood that, for the sake of clarity, the embodiments of the present application are described with reference to different functional units. However, it will be apparent that, without departing from the present application, the functionality of each functional unit can be implemented in a single unit, implemented in multiple units or implemented as a part for other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating strict logical or physical structure or organization. Therefore, the application can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0098] It will be understood that although the terms first, second, third, etc. may be used herein to describe various devices, elements, parts, or portions, these devices, elements, parts, or portions should not be limited by these terms. These terms are only used to distinguish one device, element, part, or portion from another device, element, part, or portion.

[0099] Although the present application has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present application is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and inclusion in different claims does not imply that a combination of features is not feasible and / or advantageous. The order of features in the claims does not imply any specific order in which the features must work. Furthermore, in the claims, the word "comprising" does not exclude other elements, and the term "a" or "an" does not exclude a plurality.

Claims

1. A method for generating a multi-view image, comprising: Acquire a two-dimensional image and acquire a corresponding disparity map based on the two-dimensional image; Inputting the two-dimensional image and the disparity map into a neural network to obtain an implicit representation of a three-dimensional image, the neural network comprising a compression module and an expansion module, the compression module comprising a plurality of downsampling structures, and the expansion module comprising a plurality of upsampling structures corresponding to the plurality of downsampling structures; and Two-dimensional images and disparity maps for different viewpoints are generated based on the three-dimensional image implicit representation rendering.

2. The method according to claim 1, wherein the compression module further comprises a multi-layer two-dimensional convolution structure, the expansion module further comprises a multi-layer three-dimensional convolution structure, each downsampling structure is located between two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first upsampling structure is located between the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, and the second upsampling structure is located between three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure, wherein inputting the two-dimensional image and the disparity map into a neural network to obtain an implicit representation of the three-dimensional image comprises: Providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each layer of the two-dimensional convolutional structure in the multi-layer two-dimensional convolutional structure; Fusing the first feature map with an up-sampled feature map output by a corresponding up-sampling structure among the multiple up-sampling structures to obtain a fused feature map, where the up-sampled feature map output by the corresponding up-sampling structure has the same size as the first feature map; Providing the fused feature map to a three-dimensional convolutional structure of a corresponding layer in the multi-layer three-dimensional convolutional structure to output a corresponding second feature map; as well as An implicit representation of the three-dimensional image is obtained based on a second feature map output by a last layer of three-dimensional convolutional structure in the multi-layer three-dimensional convolutional structure.

3. The method according to claim 2, wherein providing the two-dimensional image and the disparity map to the compression module to output a corresponding first feature map from each layer of the two-dimensional convolutional structure comprises: Using each of the multiple downsampling structures to downsample the first feature map output by the previous two-dimensional convolution structure in the two-dimensional convolution structure of the adjacent two layers to obtain Downsample feature maps; as well as The downsampled feature map is input into the subsequent two-dimensional convolution structure in the two-dimensional convolution structures of the adjacent two layers to output the corresponding first feature map.

4. The method according to claim 2, wherein the upsampled feature map comprises a first upsampled feature map output by the first upsampled structure and a second upsampled feature map output by the second upsampled structure, wherein the method further comprises: Upsampling a first feature map output by a last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure using a first upsampling structure among the multiple upsampling structures to obtain the first upsampling feature map; as well as The second upsampling structure is used to upsample the second feature map output by the previous three-dimensional convolution structure in the three-dimensional convolution structures of two adjacent layers to obtain the second up-sampled feature map.

5. The method according to claim 4, wherein fusing the first feature map with the upsampled feature map output by the corresponding upsampled structure in the multiple upsampled structures to obtain a fused feature map comprises: Fusing a first feature map output by a preceding two-dimensional convolutional structure of the last two-dimensional convolutional structure with the first up-sampled feature map to obtain a fused feature map provided to a first three-dimensional convolutional structure in the multi-layer three-dimensional convolutional structure; as well as The second up-sampling feature map is fused with the first feature map output by the preceding two-dimensional convolution structure of the first down-sampling structure corresponding to the second up-sampling structure to obtain a fused feature map provided to the subsequent three-dimensional convolution structure of the second up-sampling structure.

6. The method according to claim 2, wherein fusing the first feature map with an upsampled feature map output by a corresponding upsampled structure in the plurality of upsampled structures to obtain a fused feature map comprises: The first feature map and the upsampled feature map are superimposed along the depth dimension. The method of claim 1 , wherein the implicit representation of the three-dimensional image comprises a multi-planar image.

8. The method according to claim 1, wherein the different viewpoints include a reference viewpoint corresponding to the two-dimensional image and a target viewpoint different from the reference viewpoint, wherein generating the two-dimensional image and the disparity map for the different viewpoints based on the implicit representation rendering of the three-dimensional image comprises: The two-dimensional images and the disparity map of the reference viewpoint and the target viewpoint are generated by implicit rendering according to the three-dimensional image.

9. The method according to claim 1, wherein acquiring a two-dimensional image and acquiring a corresponding disparity map based on the two-dimensional image comprises: A monocular depth estimation network is used to generate a disparity map corresponding to the two-dimensional image based on the two-dimensional image.

10. A method for obtaining training data for a neural network as claimed in claim 1, comprising: Acquire a plurality of reference viewpoint two-dimensional images corresponding to a reference viewpoint and a reference viewpoint disparity map corresponding to each of the plurality of reference viewpoint two-dimensional images; Determining a viewpoint conversion matrix and a target viewpoint for each reference viewpoint two-dimensional image, wherein the viewpoint conversion matrix represents a spatial position offset of the target viewpoint relative to the reference viewpoint; Determine a target viewpoint two-dimensional image corresponding to the target viewpoint according to the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image and camera intrinsic parameters; Acquire a target viewpoint disparity map corresponding to the target viewpoint two-dimensional image; as well as Training data is formed based on the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the target viewpoint two-dimensional image, and the target viewpoint disparity map.

11. The method according to claim 10, wherein forming training data based on the reference viewpoint 2D image, the reference viewpoint disparity map, the target viewpoint 2D image, and the target viewpoint disparity map comprises: The holes in the target viewpoint two-dimensional image and the target viewpoint disparity map are repaired to obtain a repaired target viewpoint two-dimensional image and a repaired target viewpoint disparity map, and the reference viewpoint two-dimensional image, the reference viewpoint disparity map, the repaired target viewpoint two-dimensional image and the repaired target viewpoint disparity map are used as training data.

12. The method according to claim 10, wherein determining a target viewpoint two-dimensional image corresponding to the target viewpoint according to the viewpoint conversion matrix, the reference viewpoint disparity map, the reference viewpoint two-dimensional image, and camera intrinsic parameters comprises: The pixel value of each pixel of the target viewpoint two-dimensional image is determined according to the viewpoint conversion matrix, the reference viewpoint disparity map, the pixel value of each pixel of the reference viewpoint two-dimensional image and the camera intrinsic parameter.

13. The method according to claim 10, wherein determining the viewpoint conversion matrix and the target viewpoint for each reference viewpoint 2D image comprises: A rotation vector and a translation vector are randomly determined for a reference viewpoint corresponding to the reference viewpoint two-dimensional image, wherein the rotation vector represents the Euler angle offset of the target viewpoint relative to the reference viewpoint, and the translation vector represents the coordinate offset of the target viewpoint relative to the reference viewpoint.

14. A method for training the neural network of claim 1 using the training data of claim 10, comprising: Providing the reference viewpoint two-dimensional image and the reference viewpoint disparity map to the neural network to generate an implicit representation of the three-dimensional image, Rendering a two-dimensional image of a reference viewpoint marker and a two-dimensional image of a target viewpoint marker at a reference viewpoint and a target viewpoint respectively based on the implicit representation of the three-dimensional image; Obtaining a disparity map of the reference viewpoint mark and a disparity map of the target viewpoint mark based on the two-dimensional image of the reference viewpoint mark and the two-dimensional image of the target viewpoint mark respectively; determining a loss function based on a difference between the reference viewpoint two-dimensional image and the reference viewpoint target two-dimensional image, a difference between the target viewpoint two-dimensional image and the target viewpoint target two-dimensional image, a difference between the reference viewpoint disparity map and the reference viewpoint target disparity map, and a difference between the target viewpoint disparity map and the target viewpoint target disparity map; and The neural network is trained using the training data until the value of the loss function is less than a threshold.

15. The method according to claim 14 , wherein determining the loss function based on the difference between the reference viewpoint 2D image and the reference viewpoint target 2D image, the difference between the target viewpoint 2D image and the target viewpoint target 2D image, the difference between the reference viewpoint disparity map and the reference viewpoint target disparity map, and the difference between the target viewpoint disparity map and the target viewpoint target disparity map comprises: determining a first loss sub-function, a second loss sub-function, a third loss sub-function, and a fourth loss sub-function according to the difference between the reference viewpoint two-dimensional image and the two-dimensional image of the reference viewpoint target, the difference between the target viewpoint two-dimensional image and the two-dimensional image of the target viewpoint target, the difference between the reference viewpoint disparity map and the disparity map of the reference viewpoint target, and the difference between the target viewpoint disparity map and the disparity map of the target viewpoint target, respectively; The sum of the first loss sub-function, the second loss sub-function, the third loss sub-function and the fourth loss sub-function is determined as the loss function.

16. A device for generating multi-viewpoint images, comprising: an image acquisition module configured to acquire a two-dimensional image and acquire a corresponding disparity map based on the two-dimensional image; a 3D image implicit representation generation module, configured to generate a 3D image implicit representation based on the 2D image and the disparity map, the 3D image implicit representation generation module comprising a compression module and an expansion module, the compression module comprising a plurality of downsampling structures, the expansion module comprising a plurality of upsampling structures corresponding to the plurality of downsampling structures; as well as A rendering module is configured to generate two-dimensional images and disparity maps for different viewpoints based on the three-dimensional image implicit representation rendering.

17. The apparatus according to claim 16, wherein the compression module further comprises a multi-layer two-dimensional convolution structure, the expansion module further comprises a multi-layer three-dimensional convolution structure, each downsampling structure is located between the two-dimensional convolution structures of adjacent layers in the multi-layer two-dimensional convolution structure, the multiple upsampling structures include a first upsampling structure and a second upsampling structure, the first upsampling structure is located between the last two-dimensional convolution structure in the multi-layer two-dimensional convolution structure and the first three-dimensional convolution structure in the multi-layer three-dimensional convolution structure, the second upsampling structure is located between the three-dimensional convolution structures of adjacent layers in the multi-layer three-dimensional convolution structure, wherein the compression module is configured to receive the two-dimensional image and the disparity map to output a corresponding first feature map from each layer of the two-dimensional convolution structure in the multi-layer two-dimensional convolution structure, The three-dimensional image implicit representation generation module further includes a feature map fusion module, wherein the feature map fusion module is configured to fuse the first feature map with an up-sampled feature map output by a corresponding up-sampling structure in the multiple up-sampling structures to obtain a fused feature map, wherein the up-sampled feature map output by the corresponding up-sampling structure has the same size as the first feature map. The three-dimensional convolution structure of the corresponding layer in the multi-layer three-dimensional convolution structure receives the fused feature map to output a corresponding second feature map, and the expansion module obtains an implicit representation of the three-dimensional image based on the second feature map output by the last layer of the three-dimensional convolution structure in the multi-layer three-dimensional convolution structure.

18. A computing device comprising: a memory configured to store computer-executable instructions; as well as A processor configured to perform the method according to any one of claims 1 to 15 when the computer executable instructions are executed by the processor.

19. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions implement the method according to any one of claims 1 to 15 when executed.

20. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images