A method and device for generating parallax images of 2D video frames

Abstract

The embodiment of the present invention provides a method and device for generating a parallax image of a 2D video frame, which relates to the field of video processing technology. The method includes: performing cascade downsampling processing on the 2D video frame. Performing upsampling processing on the first image. Performing convolution processing on the upsampled image. Fusing the upsampled image, the convolution image and the second image to obtain a fused image, and performing upsampling processing on the fused image. If the size of the new upsampled image is different from the size of the 2D video frame, then returning to the step of performing convolution processing on the upsampled image. Otherwise, obtaining a convolution image based on the first convolution image, and fusing the upsampled image, the convolution image and the downsampled image to obtain an image with parallax with the 2D video frame. Applying the solution provided by the embodiment of the present invention can reduce the difference between the parallax of different 2D video frames and the generated image.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method and device for generating a parallax image of a 2D video frame. Background Art

[0002] From the perspective of visual perception, videos can be divided into two categories: 2D videos and 3D videos. Among them, each video frame of 2D videos is an image, and each video frame of 3D videos contains two views with parallax, where each view is an image. When watching 3D videos, the left and right eyes of users see different views. Since there is parallax between the two views, users can see a 3D stereoscopic effect. It can be seen that converting 2D videos to 3D videos allows users to obtain more 3D stereoscopic information while watching videos. 3D videos are currently widely used in many fields such as movies and television.

[0003] At present, in the process of converting 2D video frames into 3D videos, an image with parallax with the 2D video frame can be generated based on the 2D video frame in the 2D video, and then a 3D video frame can be obtained based on the 2D video frame and the generated image. However, in the process of generating an image with parallax with the 2D video frame based on the 2D video frame, there is a problem of parallax ambiguity, that is, images are generated based on similar 2D video frames, and there is parallax between the generated image and the 2D video frame, but the difference between the parallaxes corresponding to different 2D video frames is large. As a result, the 3D effect displayed by the 3D video generated based on the similar 2D video frame and the generated image is different, so the 3D effect of the 3D video obtained by the conversion of the prior art is unstable. In order to improve the stability of the 3D effect of the 3D video obtained by converting the 2D video, it is necessary to provide a parallax image generation method of a 2D video frame to reduce the difference between the parallax of different 2D video frames and the generated image. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and device for generating a disparity image of a 2D video frame, so as to reduce the difference between the disparity of different 2D video frames and the generated image. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a disparity image of a 2D video frame, the method comprising:

[0006] Performing a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process;

[0007] Performing up-sampling processing on the first image to obtain an up-sampled image, wherein the first image is: an image with the smallest size among the obtained down-sampled images;

[0008] Performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0009] The upsampled image, the convolution image and the second image with the same size as the upsampled image are subjected to pixel value fusion according to pixel positions to obtain a fused image, the fused image is subjected to upsampling processing to obtain a new upsampled image, if the size of the new upsampled image is different from the size of the 2D video frame, then the steps of performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image, and the upsampled image, the convolution image and the second image with the same size as the upsampled image are subjected to pixel value fusion according to pixel positions to obtain a fused image, and the fused image is subjected to upsampling processing to obtain a new upsampled image, wherein the second image is an image in the downsampled image other than the first image;

[0010] Otherwise, according to the first convolution image, a convolution image having the same size as the upsampled image is obtained, and pixel values ​​of the upsampled image, the obtained convolution image, and the image having the same size as the upsampled image in the downsampled image are fused according to pixel positions to obtain an image having parallax with the 2D video frame, wherein the first convolution image is: the image with the largest size in the second convolution image, and the second convolution image is: a convolution image having a size smaller than that of the upsampled image.

[0011] In one embodiment of the present invention, obtaining a convolution image having the same size as that of the upsampled image according to the first convolution image includes:

[0012] Performing convolution processing on the up-sampled image obtained according to the first convolution image to obtain a third convolution image having the same size as that of the up-sampled image;

[0013] Based on preset parameters, the pixel values ​​of the pixels in the obtained third convolution image are adjusted so that the mean and variance of the pixel values ​​of the pixels in the convolution image are changed, and a convolution image having the same size as the upsampled image is obtained.

[0014] In one embodiment of the present invention, adjusting the pixel values ​​of pixels in the obtained third convolution image based on preset parameters includes:

[0015] A preset disparity scaling value is multiplied by the pixel value of the pixel in the obtained third convolution image, and a preset disparity offset value is added to obtain a new pixel value.

[0016] In one embodiment of the present invention, obtaining a convolution image having the same size as that of the upsampled image according to the first convolution image includes:

[0017] The first convolution image is scaled up to obtain a convolution image having the same size as the upsampled image.

[0018] In one embodiment of the present invention, the obtaining of an image having parallax with the 2D video frame in the following manner includes:

[0019] The 2D video frame is input into a pre-trained image acquisition model to obtain an image with parallax to the 2D video frame, wherein the image acquisition model is used to: implement the cascade downsampling process of the 2D video frame including a preset number of downsampling processes, obtain the downsampled image obtained by each downsampling process, and perform pixel value fusion according to pixel position to obtain the image with parallax to the 2D video frame.

[0020] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model in the following manner:

[0021] Inputting a first sample view and a second sample view in the 3D sample video frame into a preset neural network model respectively, obtaining a first downsampled image of the first sample view and a second downsampled image of the second sample view obtained by each downsampling process, and a sample convolution image of the first sample view obtained by each convolution process, wherein the first sample view and the second sample view are two views in the 3D sample video frame respectively;

[0022] fusing pixel values ​​of the first down-sampled image with the smallest size and the second down-sampled image with the smallest size according to pixel positions to obtain a sample fused image;

[0023] Perform convolution processing on the sample fusion image to obtain a binocular convolution image with the same size as the sample fusion image;

[0024] The obtained binocular convolution image is enlarged to obtain an enlarged binocular convolution image having the same size as the target down-sampled image, wherein the size of the target down-sampled image is the smallest among the third down-sampled images, and the third down-sampled image is the first down-sampled image having a size larger than the binocular convolution image before enlargement;

[0025] The enlarged binocular convolution image, the first down-sampled image and the second down-sampled image having the same size as the enlarged binocular convolution image are fused according to pixel values ​​at pixel positions to obtain a new sample fused image;

[0026] Performing convolution processing on the sample fusion image to obtain a new binocular convolution image with the same size as the sample fusion image;

[0027] According to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating the disparity offset size between the binocular convolution image and the first sample convolution image and a disparity scaling value indicating the disparity scaling multiple between the binocular convolution image and the first sample convolution image are calculated, wherein the first sample convolution image is a sample convolution image having the same size as that of the binocular convolution image;

[0028] According to the calculated disparity offset value, disparity scaling value and the first sample convolution image, pixel values ​​of pixels in the binocular convolution image are adjusted to obtain an adjusted binocular convolution image;

[0029] If the adjusted binocular convolution image is different in size from the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as a new binocular convolution image, and the step of enlarging the size of the obtained binocular convolution image to obtain an enlarged binocular convolution image with the same size as the target down-sampled image is returned to the step of adjusting the pixel values ​​of the pixels in the binocular convolution image according to the calculated disparity offset value, the disparity scaling value and the first sample convolution image to obtain the adjusted binocular convolution image, wherein the fourth down-sampled image is: the first down-sampled image except the first down-sampled image with the largest size;

[0030] Otherwise, adjusting the pixel values ​​of the pixels in the sample convolution image with the largest size according to the binocular convolution image, and using the adjusted sample convolution image as an output image having a parallax with the first sample image;

[0031] Calculating the loss of the neural network model according to the output image and the second sample view;

[0032] Model parameters of the neural network model are adjusted according to the loss.

[0033] In one embodiment of the present invention, adjusting the pixel values ​​of pixels in the largest sample convolution image according to the binocular convolution image includes:

[0034] Calculating a disparity offset value and a disparity scaling value between the binocular convolution image and the first sample convolution image according to the binocular convolution image and the first sample convolution image;

[0035] According to the calculated disparity offset value and disparity scaling value, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted.

[0036] In one embodiment of the present invention, adjusting the pixel values ​​of pixels in the largest sample convolution image according to the binocular convolution image includes:

[0037] The obtained binocular convolution image is enlarged to obtain an enlarged binocular convolution image having the same size as the largest sample convolution image;

[0038] The enlarged binocular convolution image is fused with the sample convolution image with the largest size according to the pixel position to obtain the adjusted sample convolution image.

[0039] In one embodiment of the present invention, the calculating, according to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating a disparity offset between the binocular convolution image and the first sample convolution image and a disparity scaling value indicating a disparity scaling factor between the binocular convolution image and the first sample convolution image comprises:

[0040] Calculate the product between the first sample convolution image and the transposed value of the first convolution image as a first result;

[0041] Inverting the first result to obtain a second result;

[0042] Calculating the product of the first sample convolution image and the binocular convolution image as a third result;

[0043] The product of the second result and the third result is calculated to obtain the disparity offset value representing the disparity offset size between the binocular convolution image and the first sample convolution image and the disparity scaling value representing the disparity scaling multiple between the binocular convolution image and the first sample convolution image.

[0044] In one embodiment of the present invention, adjusting the pixel values ​​of pixels in the binocular convolution image according to the calculated disparity offset value, the disparity scaling value and the first sample convolution image to obtain the adjusted binocular convolution image includes:

[0045] Multiplying a pixel value of a pixel in the first sample convolution image by the disparity scaling value, and adding the disparity offset value to obtain a calculation result;

[0046] The average value of the above calculation result and the pixel value of the pixel in the binocular convolution image is calculated to obtain the pixel value of the pixel in the adjusted binocular convolution image, and the adjusted binocular convolution image is obtained.

[0047] In a second aspect, an embodiment of the present invention provides a device for generating a disparity image of a 2D video frame, the device comprising:

[0048] A downsampling module is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as a processed image processed by the next downsampling process;

[0049] An up-sampling module is used to perform up-sampling processing on the first image to obtain an up-sampled image, wherein the first image is: an image with the smallest size among the obtained down-sampled images;

[0050] A convolution module, used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0051] an image fusion module, configured to fuse the upsampled image, the convolution image, and a second image of the same size as the upsampled image according to pixel positions to obtain a fused image, perform upsampling on the fused image to obtain a new upsampled image, and return to trigger the execution of the convolution module and the image fusion module if the size of the new upsampled image is different from the size of the 2D video frame, wherein the second image is: an image in the downsampled image other than the first image;

[0052] an image acquisition module, for obtaining, according to the first convolution image, a convolution image having the same size as the upsampled image if the size of the new upsampled image is the same as the size of the 2D video frame, and fusing pixel values ​​of the upsampled image, the obtained convolution image, and an image having the same size as the upsampled image among the downsampled images according to pixel positions, to obtain an image having parallax with the 2D video frame, wherein the first convolution image is: an image having the largest size among the second convolution images, and the second convolution image is: a convolution image having a size smaller than that of the upsampled image.

[0053] In one embodiment of the present invention, the image acquisition module includes:

[0054] A convolution submodule, configured to perform convolution processing on the upsampled image obtained according to the first convolution image to obtain a third convolution image having the same size as that of the upsampled image;

[0055] A pixel value adjustment submodule, configured to adjust the pixel values ​​of the pixels in the obtained third convolution image based on preset parameters, so that the mean and variance of the pixel values ​​of the pixels in the convolution image are changed, and a convolution image having the same size as that of the upsampled image is obtained;

[0056] The image acquisition submodule is used to fuse the upsampled image, the obtained convolution image and the image of the downsampled image with the same size as the upsampled image according to the pixel position to obtain an image with parallax with the 2D video frame.

[0057] In one embodiment of the present invention, the pixel value adjustment submodule is specifically used for:

[0058] A preset disparity scaling value is multiplied by the pixel value of the pixel in the obtained third convolution image, and a preset disparity offset value is added to obtain a new pixel value.

[0059] In one embodiment of the present invention, the image acquisition module is specifically used to:

[0060] The first convolution image is enlarged to obtain a convolution image with the same size as the upsampled image, and the upsampled image, the obtained convolution image, and an image with the same size as the upsampled image in the downsampled image are fused according to pixel positions to obtain an image with parallax with the 2D video frame.

[0061] In one embodiment of the present invention, an image having parallax with the 2D video frame is obtained by the following image obtaining module, and the image determining module includes:

[0062] The 2D video frame is input into a pre-trained image acquisition model to obtain an image having parallax with the 2D video frame, wherein the image acquisition model is used to implement the downsampling module to the image acquisition module.

[0063] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model through the following model training module, and the model training module includes:

[0064] a convolution image acquisition submodule, used to input the first sample view and the second sample view in the 3D sample video frame into a preset neural network model respectively, to obtain a first down-sampled image of the first sample view and a second down-sampled image of the second sample view obtained by each down-sampling process, and a sample convolution image of the first sample view obtained by each convolution process, wherein the first sample view and the second sample view are two views in the 3D sample video frame respectively;

[0065] A first image fusion submodule is used to fuse pixel values ​​of the first down-sampled image with the smallest size and the second down-sampled image with the smallest size according to pixel positions to obtain a sample fused image;

[0066] The binocular convolution submodule is used to perform convolution processing on the sample fusion image to obtain a binocular convolution image with the same size as the sample fusion image;

[0067] A size enlargement submodule is used to enlarge the size of the obtained binocular convolution image to obtain an enlarged binocular convolution image with the same size as the target down-sampled image, wherein the size of the target down-sampled image is the smallest among the third down-sampled images, and the third down-sampled image is the first down-sampled image with a size larger than the binocular convolution image before enlargement;

[0068] A second image fusion submodule is used to fuse the enlarged binocular convolution image, the first down-sampled image with the same size as the enlarged binocular convolution image, and the second down-sampled image according to pixel positions to obtain a new sample fusion image;

[0069] An image convolution submodule, used for performing convolution processing on the sample fusion image to obtain a new binocular convolution image with the same size as the sample fusion image;

[0070] a disparity calculation submodule, configured to calculate, according to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating the size of the disparity offset between the binocular convolution image and the first sample convolution image and a disparity scaling value indicating the disparity scaling factor between the binocular convolution image and the first sample convolution image, wherein the first sample convolution image is a sample convolution image having the same size as that of the binocular convolution image;

[0071] a pixel value adjustment submodule, configured to adjust the pixel values ​​of the pixels in the binocular convolution image according to the calculated disparity offset value, the disparity scaling value and the first sample convolution image, to obtain an adjusted binocular convolution image; if the size of the adjusted binocular convolution image is different from the size of the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as a new binocular convolution image, and the size enlargement submodule is returned to execute the pixel value adjustment submodule, wherein the fourth down-sampled image is: the first down-sampled image except the first down-sampled image with the largest size;

[0072] An image output submodule, configured to adjust the pixel values ​​of pixels in the largest sample convolution image according to the binocular convolution image if the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, and use the adjusted sample convolution image as an output image having a parallax with the first sample image;

[0073] A loss calculation submodule, used to calculate the loss of the neural network model according to the output image and the second sample view;

[0074] A parameter adjustment submodule is used to adjust the model parameters of the neural network model according to the loss.

[0075] In one embodiment of the present invention, the image output submodule is specifically used for:

[0076] If the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, a disparity offset value and a disparity scaling value between the binocular convolution image and the first sample convolution image are calculated according to the binocular convolution image and the first sample convolution image;

[0077] According to the calculated disparity offset value and disparity scaling value, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted, and the adjusted sample convolution image is used as an output image having a disparity with the first sample image.

[0078] In one embodiment of the present invention, the image output submodule is specifically used for:

[0079] If the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, the obtained binocular convolution image is enlarged to obtain an enlarged binocular convolution image with the same size as the largest sample convolution image;

[0080] The enlarged binocular convolution image and the sample convolution image with the largest size are fused according to pixel positions to obtain an adjusted sample convolution image, and the adjusted sample convolution image is used as an output image with parallax with the first sample image.

[0081] In one embodiment of the present invention, the disparity calculation submodule is specifically used for:

[0082] Calculate the product between the first sample convolution image and the transposed value of the first convolution image as a first result;

[0083] Inverting the first result to obtain a second result;

[0084] Calculating the product of the first sample convolution image and the binocular convolution image as a third result;

[0085] The product of the second result and the third result is calculated to obtain the disparity offset value representing the disparity offset size between the binocular convolution image and the first sample convolution image and the disparity scaling value representing the disparity scaling multiple between the binocular convolution image and the first sample convolution image.

[0086] In one embodiment of the present invention, the pixel value adjustment submodule is specifically used for:

[0087] Multiplying a pixel value of a pixel in the first sample convolution image by the disparity scaling value, and adding the disparity offset value to obtain a calculation result;

[0088] Calculate the average value of the above calculation result and the pixel value of the pixel in the binocular convolution image to obtain the pixel value of the pixel in the adjusted binocular convolution image, and obtain the adjusted binocular convolution image;

[0089] If the size of the adjusted binocular convolution image is different from the size of the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as a new binocular convolution image, and the size enlargement submodule is returned to execute the pixel value adjustment submodule.

[0090] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0091] Memory, used to store computer programs;

[0092] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0093] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0094] In yet another aspect of the present invention, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the method steps described in the first aspect.

[0095] In the parallax image generation scheme of a 2D video frame provided by an embodiment of the present invention, a cascade downsampling process is performed on the 2D video frame to obtain a downsampled image obtained by each downsampling process. The first image is upsampled to obtain an upsampled image, and the upsampled image is convolved to obtain a convolution image of the same size as the upsampled image. The upsampled image, the convolution image, and the second image of the same size as the upsampled image are pixel-value fused to obtain a fused image. The fused image is upsampled to obtain a new upsampled image. The above process is repeated until the size of the new upsampled image is the same as that of the 2D video frame. According to the first convolution image, a convolution image of the same size as the upsampled image is obtained, and the upsampled image, the obtained convolution image, and the image of the same size as the upsampled image in the downsampled image are pixel-value fused according to the pixel position, and the finally obtained fused image is determined as an image with parallax with the 2D video frame.

[0096] As can be seen from the above, the solution provided by the embodiment of the present invention can obtain an image with parallax to the 2D video frame, and in the process of obtaining the image with parallax to the 2D video frame, the down-sampled image obtained based on the original 2D video frame is used to adjust the up-sampled image before generating the image with parallax to the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the parallax between the two is smaller, thereby maintaining the parallax of the generated 3D video within a smaller range, and reducing the difference between the parallax of different 2D video frames and the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0098] Figure 1 A schematic diagram of a process of generating a disparity image of a 2D video frame provided by an embodiment of the present invention;

[0099] Figure 2 A schematic diagram of a flow chart of a first image acquisition model training method provided in an embodiment of the present invention;

[0100] Figure 3 A schematic diagram of a flow chart of a second image acquisition model training method provided in an embodiment of the present invention;

[0101] Figure 4 A schematic structural diagram of a first 2D video frame disparity image generating device provided by an embodiment of the present invention;

[0102] Figure 5 A schematic structural diagram of a second 2D video frame disparity image generating device provided by an embodiment of the present invention;

[0103] Figure 6 A schematic diagram of the structure of a model training module provided by an embodiment of the present invention;

[0104] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0105] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0106] Since there is a problem in the prior art that the disparity between different 2D video frames and the generated images is relatively large, in order to solve this problem, an embodiment of the present invention provides a method and device for generating a disparity image of a 2D video frame.

[0107] In one embodiment of the present invention, a method for generating a disparity image of a 2D video frame is provided, the method comprising:

[0108] A cascade downsampling process including a preset number of downsampling processes is performed on the 2D video frame to obtain a downsampled image obtained by each downsampling process, and the result of each downsampling process in the above cascade downsampling process is used as a processed image processed by the next downsampling process.

[0109] The first image is upsampled to obtain an upsampled image, wherein the first image is an image with the smallest size among the obtained downsampled images.

[0110] The upsampled image is convolved to obtain a convolved image with the same size as the upsampled image.

[0111] The upsampled image, the convolution image and the second image with the same size as the upsampled image are fused according to pixel positions to obtain a fused image, and the fused image is upsampled to obtain a new upsampled image. If the size of the new upsampled image is different from the size of the 2D video frame, the steps of performing convolution on the upsampled image to obtain a convolution image with the same size as the upsampled image are returned to, and the upsampled image, the convolution image and the second image with the same size as the upsampled image are fused according to pixel positions to obtain a fused image, and the fused image is upsampled to obtain a new upsampled image, wherein the second image is: the image in the downsampled image other than the first image.

[0112] Otherwise, based on the first convolution image, a convolution image having the same size as the upsampled image is obtained, and the upsampled image, the obtained convolution image, and the image having the same size as the upsampled image in the downsampled image are fused according to pixel positions to obtain an image having parallax with the 2D video frame, the first convolution image being the largest image in the second convolution image, and the second convolution image being a convolution image having a size smaller than the upsampled image.

[0113] As can be seen from the above, the solution provided by the embodiment of the present invention can obtain an image that has parallax with the 2D video frame. The above process of converting 2D video into 3D video uses a down-sampled image obtained based on the original 2D video frame to adjust the up-sampled image before generating the image that has parallax with the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the parallax between the two is smaller, thereby maintaining the parallax of the generated 3D video within a smaller range, which can reduce the difference between the parallax of different 2D video frames and the generated image.

[0114] The following describes a method and device for generating a disparity image of a 2D video frame provided by an embodiment of the present invention through specific embodiments.

[0115] The embodiment of the present invention provides a method for generating a parallax image of a 2D video frame. When a user watches a 3D video, the left eye and the right eye see different views. Therefore, according to the different eyes corresponding to the views, the views of the 3D video frame can be divided into a left view and a right view. If the 2D video frame is used as the left view, the image obtained with a parallax to the 2D video frame is the right view. Conversely, if the 2D video frame is used as the right view, the image obtained with a parallax to the 2D video frame is the left view.

[0116] See also Figure 1 , an embodiment of the present invention provides a flow chart of a method for generating a disparity image of a 2D video frame, and the method can be implemented through the following steps S101-S105.

[0117] S101: performing a cascade downsampling process including a preset number of downsampling processes on a 2D video frame to obtain a downsampled image obtained by each downsampling process.

[0118] The result of each downsampling process in the cascade downsampling process is used as the processed image for the next downsampling process.

[0119] For example, the above preset number of times can be 5 times. After the 2D video frame is down-sampled for the first time, the result of the first down-sampling process is down-sampled for the second time, and the result of the second down-sampling process is down-sampled for the third time, and so on. A total of 5 down-sampling processes are performed to obtain 5 down-sampled images.

[0120] Since the downsampling process will reduce the size of the image, the sizes of the obtained downsampled images are different. Specifically, the downsampling process can be a process that reduces the size of the image after the downsampling process to 1 / 2 of the size of the original image.

[0121] For example, the size of the above-mentioned 2D video frame is 1080 pixels × 720 pixels. The size of the downsampled image obtained after one downsampling process is 540 pixels × 360 pixels. The size of the downsampled image obtained after the second downsampling process is 270 pixels × 180 pixels, and so on. The size of the downsampled image obtained after each downsampling process is half of the size before the downsampling process.

[0122] In one embodiment of the present invention, the downsampling process can be implemented through convolution processing and pooling processing.

[0123] S102: Perform up-sampling processing on the first image to obtain an up-sampled image.

[0124] The first image is the image with the smallest size among the obtained down-sampled images.

[0125] Specifically, the above-mentioned upsampling process is the inverse transformation process of the downsampling process in the above-mentioned cascade downsampling process, that is, the size of the first image processed by the upsampling process is the same as the size of the image obtained by the above-mentioned downsampling process, and the size of the upsampled image obtained by the above-mentioned upsampling process is the same as the size of the downsampled image processed by the above-mentioned downsampling process.

[0126] For example, if the downsampling process is to reduce the size of the image after the downsampling process to 1 / 2 of the original image, the upsampling process can increase the size of the image after the upsampling process to 2 times of the original image.

[0127] For example, if the size of the first image is 270 pixels×180 pixels, the size of the upsampled image obtained after upsampling the first image is 540 pixels×360 pixels.

[0128] The above-mentioned up-sampling process can be implemented by a bilinear interpolation method.

[0129] In addition, the above-mentioned downsampling process can be called an encoding process, and the above-mentioned upsampling process can be called a decoding process.

[0130] S103: Perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image.

[0131] Specifically, the convolution process may be a convolution process that keeps the image size unchanged. Then the size of the convolution image obtained after the convolution process is the same as the size of the upsampled image.

[0132] S104: fusing the upsampled image, the convolution image and a second image having the same size as the upsampled image according to pixel positions to obtain a fused image, and upsampling the fused image to obtain a new upsampled image.

[0133] The second image is an image of the downsampled image other than the first image. Since the upsampling process and the downsampling process are inverse transformation processes, there exists a downsampled image having the same size as the upsampled image obtained after the upsampling process.

[0134] Specifically, since the upsampled image, the convolution image and the downsampled image have the same size, the pixels of the above images correspond to each other according to the pixel positions, that is, the pixels with the same pixel positions correspond to each other, for example, the pixels in the upper left corners of the three correspond to each other. The upsampled image, the convolution image and the downsampled image with the same size are fused, and the size of the obtained fused image is also the same as the upsampled image.

[0135] The pixel values ​​of the corresponding pixels may be averaged, weighted averaged, etc., and the result of the calculation may be used as the pixel value of the pixel at the corresponding pixel position of the fused image, thereby obtaining the fused image. For example, the pixel values ​​of the pixels at the upper left corner of the three pixels may be weighted averaged, and the result of the calculation may be used as the pixel value of the pixel at the upper left corner of the fused image.

[0136] In addition, the above-mentioned up-sampled image, convolution image and down-sampled image may also be fused by using methods such as a logical filter algorithm, a pyramid decomposition algorithm, and a high-pass filter algorithm, which will not be described in detail here.

[0137] If the size of the new up-sampled image is different from the size of the 2D video frame, the process returns to execute the above steps S103 - S104 , otherwise, the process executes step S105 .

[0138] Specifically, the sizes of the two views of the 3D video frame are the same, so if the size of the upsampled image is different from the size of the 2D video frame, it means that an image with parallax with the 2D video frame has not been obtained, so it is necessary to continue processing the upsampled image and return to execute the above steps S103-S104. After returning to the above steps S103-S104, convolution processing will be performed on the upsampled image.

[0139] Otherwise, it means that an image having parallax with the 2D video frame can be obtained according to the up-sampled image, and then the process continues to step S105.

[0140] In addition, the process of performing cascade downsampling on the 2D video frame in steps S101 to S104 and then performing upsampling multiple times continuously until the size of the obtained fused image is the same as that of the 2D video frame can be called pyramid coding.

[0141] S105: According to the first convolution image, a convolution image having the same size as the up-sampled image is obtained, and pixel values ​​of the up-sampled image, the obtained convolution image, and the image having the same size as the up-sampled image in the down-sampled image are fused according to pixel positions to obtain an image having parallax with the 2D video frame.

[0142] Among them, the above-mentioned first convolution image is: the image with the largest size in the second convolution image, and the above-mentioned second convolution image is: a convolution image with a size smaller than the above-mentioned up-sampled image.

[0143] Since the size of the upsampled image obtained by each upsampling process increases gradually during the process of repeatedly executing the above steps S103-S104, the size of the convolution image with the same size as the upsampled image obtained by each convolution process also increases gradually. Then the above first convolution image is: the convolution image obtained by the last convolution process before executing the above step S105.

[0144] In one embodiment of the present invention, a convolution image having the same size as the upsampled image can be obtained through step A.

[0145] Step A: Enlarge the size of the first convolution image to obtain a convolution image with the same size as the upsampled image.

[0146] Specifically, the first convolution image may be upscaled by nearest neighbor interpolation, bilinear interpolation, quadratic interpolation, cubic interpolation, or the like to obtain a convolution image having the same size as the upsampled image.

[0147] In another embodiment of the present invention, a convolution image having the same size as the up-sampled image can be obtained through steps B to C.

[0148] Step B: performing convolution processing on the up-sampled image obtained according to the first convolution image to obtain a third convolution image having the same size as that of the up-sampled image.

[0149] Specifically, the convolution process is similar to the aforementioned step S103, which will not be described in detail here.

[0150] Step C: Based on preset parameters, the pixel values ​​of the pixels in the obtained third convolution image are adjusted so that the mean and variance of the pixel values ​​of the pixels in the convolution image are changed, and a convolution image having the same size as the upsampled image is obtained.

[0151] In one embodiment of the present invention, a preset disparity scaling value is multiplied by the pixel value of the pixel in the obtained third convolution image, and a preset disparity offset value is added to obtain a new pixel value.

[0152] The above-mentioned preset parallax scaling value and the preset parallax offset value are both the above-mentioned preset parameters.

[0153] Specifically, the pixel values ​​of the pixels in the obtained third convolution image may be adjusted according to the following formula:

[0154] disp′=disp*s+t

[0155] Wherein, disp is the pixel value of the pixel in the third convolution image, s and t are the above-mentioned preset parameters, and disp′ is the pixel value obtained after adjusting disp.

[0156] Specifically, by changing the above preset parameters, the calculated disp′ can be adjusted, so that the adjusted convolution image is changed.

[0157] Therefore, by adjusting the above preset parameters, the adjustment result of the convolution image can be changed, thereby changing the generated fusion result, so that the parallax between the obtained image and the 2D video frame changes, and further the parallax between the obtained image and the 2D video frame changes, so the parallax between the obtained image and the 2D video frame is adjustable.

[0158] As can be seen from the above, the solution provided by the embodiment of the present invention can obtain an image that has parallax with the 2D video frame. The above process of converting 2D video into 3D video uses a down-sampled image obtained based on the original 2D video frame to adjust the up-sampled image before generating the image that has parallax with the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the parallax between the two is smaller, thereby maintaining the parallax of the generated 3D video within a smaller range, which can reduce the difference between the parallax of different 2D video frames and the generated image.

[0159] In one embodiment of the present invention, the above steps S101-S105 can be implemented by the following step D.

[0160] Step D: Input the 2D video frame into a pre-trained image acquisition model to obtain an image with parallax to the 2D video frame.

[0161] The above image acquisition model is used to implement the above steps S101-S105.

[0162] Specifically, the above Figure 1 Compared with the embodiment shown, the above-mentioned downsampling processing can be implemented by the convolution layer and the pooling layer of the above-mentioned image acquisition model, the above-mentioned upsampling processing can be implemented by the deconvolution layer of the above-mentioned image acquisition model, and the above-mentioned convolution processing can be implemented by the convolution layer of the above-mentioned image acquisition model.

[0163] The above image acquisition model can be obtained according to the above Figure 1 The 2D video frame is processed in the manner shown to obtain an image with parallax with the 2D video frame, thereby achieving the above Figure 1 The embodiment shown.

[0164] See also Figure 2, is a flow chart of the first image acquisition model training method provided by an embodiment of the present invention. Specifically, the image acquisition model is obtained by training the preset neural network model through the following steps S201-S211.

[0165] S201: Inputting a first sample view and a second sample view in a 3D sample video frame into a preset neural network model respectively, obtaining a first downsampled image of the first sample view and a second downsampled image of the second sample view obtained by each downsampling process, and a sample convolution image of the first sample view obtained by each convolution process.

[0166] The first sample view and the second sample view are two views in the 3D sample video frame respectively.

[0167] Specifically, each 3D sample video frame includes two sample views, namely a left view and a right view. The two sample views are respectively input into a preset neural network model as a first sample view and a second sample view. The preset neural network model downsamples the input views to obtain a first downsampled image of the first sample view and a second downsampled image of the second sample view, respectively.

[0168] In addition, the image with the smallest size in the first down-sampled images may be up-sampled to obtain a first up-sampled image, and the up-sampled image may be convolved to obtain a sample convolution image with the same size as the first up-sampled image.

[0169] The first up-sampled image, the sample convolution image and the image of the first down-sampled image with the same size as the first up-sampled image are fused according to pixel positions to obtain a first fused image, and the first fused image is up-sampled to obtain a new first up-sampled image.

[0170] Repeat the steps of obtaining the first up-sampled image and performing convolution processing on the first up-sampled image to obtain the sample convolution image until the size of the first up-sampled image is the same as the size of the first sample view, thereby obtaining each sample convolution image corresponding to the first sample view.

[0171] Specifically, since the above-mentioned sample convolution image is obtained by processing the first sample view and does not involve processing of the second sample view, that is, the above-mentioned sample convolution image is obtained by processing only the view corresponding to a single eye of the viewer, the process of obtaining the sample convolution image by processing the single first sample view can be called monocular encoding.

[0172] The first sample view may be a left view or a right view in a 3D sample video frame.

[0173] In addition, in an embodiment of the present invention, the first sample view may be a sample view of a preset type.

[0174] Among them, the above preset type is a left view or a right view.

[0175] If the preset type is the left view, the left view in the sample view is always used as the first sample view during model training to obtain a sample convolution view of the left view. Therefore, the sample convolution views obtained according to the first sample view have similarity, and the neural network model trained based on similar sample convolution views is easier to be trained to convergence.

[0176] Similarly, if the above-mentioned preset type is the right view, the above-mentioned neural network model is also easier to be trained to convergence, which will not be repeated here.

[0177] S202: Fusing pixel values ​​of the first down-sampled image with the smallest size and the second down-sampled image with the smallest size according to pixel positions to obtain a sample fused image.

[0178] Specifically, the size of the left view or the right view in the 3D sample video frame is the same, that is, the size of the first sample view is the same as that of the second sample view, and the first sample view and the second sample view are subjected to the same cascade downsampling process, and the size of the first downsampled image and the second downsampled image obtained by each downsampling process is the same. The pixels in the first downsampled image and the second downsampled image of the same size correspond to each other according to the pixel position, that is, the pixels with the same pixel position correspond to each other. The pixel values ​​of the corresponding pixels can be fused according to the pixel position to obtain a sample fused image.

[0179] Since the first down-sampled image and the second down-sampled image are down-sampled images corresponding to the left view and the right view in the 3D sample video frame, respectively, the obtained sample fusion image can fuse the features of the down-sampled image of the left view and the down-sampled image of the right view, reflecting the disparity between the down-sampled image of the left view and the down-sampled image of the right view.

[0180] S203: performing convolution processing on the sample fusion image to obtain a binocular convolution image having the same size as the sample fusion image.

[0181] Specifically, the convolution processing may be a convolution processing that keeps the image size unchanged. Then the size of the convolution image obtained after the convolution processing is the same as the size of the sample fusion image.

[0182] Since the sample fusion image is obtained based on the downsampled image of the first sample image and the downsampled image of the second sample image, and the first sample image and the second sample image are respectively the left view and the right view in the 3D sample video frame, the sample fusion image fuses the downsampled image of the left view and the downsampled image of the right view, which are two types of features of downsampled images corresponding to the views of different eyes of the viewer. Therefore, the convolution image obtained by convolution processing the sample fusion image can be called a binocular convolution image.

[0183] S204: performing a size enlargement process on the obtained binocular convolution image to obtain an enlarged binocular convolution image having the same size as the target down-sampled image.

[0184] The size of the target down-sampled image is the smallest among the third down-sampled images, and the third down-sampled image is the first down-sampled image whose size is larger than the binocular convolution image before enlargement.

[0185] S205: fusing the enlarged binocular convolution image, the first down-sampled image having the same size as the enlarged binocular convolution image, and the second down-sampled image according to pixel positions to obtain a new sample fused image.

[0186] Specifically, the first down-sampled image whose size is the same as the size of the enlarged binocular convolution image is the target down-sampled image, and therefore the second down-sampled image whose size is the same as the size of the enlarged binocular convolution image is the second down-sampled image whose size is the same as the target down-sampled image.

[0187] Since the above-mentioned binocular convolution image is obtained based on the first down-sampled image and the second down-sampled image whose sizes are smaller than the target down-sampled image, the above-mentioned binocular convolution image can reflect the disparity between each pair of the first down-sampled image and the second down-sampled image whose sizes are smaller than the target down-sampled image.

[0188] On this basis, the target downsampled image and the second downsampled image having the same size as the target downsampled image are fused, so that the new sample fused image can reflect the disparity between each pair of the first downsampled image and the second downsampled image whose size is less than or equal to the target downsampled image.

[0189] S206: performing convolution processing on the sample fusion image to obtain a new binocular convolution image having the same size as that of the sample fusion image.

[0190] S207: According to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating the disparity offset between the binocular convolution image and the first sample convolution image and a disparity scaling value indicating the disparity scaling factor between the binocular convolution image and the first sample convolution image are calculated.

[0191] The first sample convolution image is a sample convolution image having the same size as the binocular convolution image.

[0192] In one embodiment of the present invention, the above step S207 can be implemented by the following steps E-H.

[0193] Step E: Calculate the product between the first sample convolution image and the transposed value of the first convolution image as a first result.

[0194] Step F: Invert the first result to obtain the second result.

[0195] Step G: Calculate the product of the first sample convolution image and the binocular convolution image as the third result.

[0196] Step H: Calculate the product of the second result and the third result to obtain the disparity offset value representing the disparity offset between the binocular convolution image and the first sample convolution image and the disparity scaling value representing the disparity scaling factor between the binocular convolution image and the first sample convolution image.

[0197] In one embodiment of the present invention, the above-mentioned steps E to H are implemented to calculate and obtain the above-mentioned parallax offset value and parallax scaling value according to the following formula:

[0198]

[0199] Where i is the sequence number of the first sample convolution image, h i =(s i ,t i ) T ,s i is the disparity scaling value corresponding to the i-th first sample convolution image, t i is the disparity offset value corresponding to the i-th first sample convolution image, d i is the i-th first sample convolution image, d i * is a binocular convolution image with the same size as the i-th first sample convolution image.

[0200] S208: adjusting pixel values ​​of pixels in the binocular convolution image according to the calculated disparity offset value, disparity scaling value, and the first sample convolution image to obtain an adjusted binocular convolution image.

[0201] In one embodiment of the present invention, the pixel value of the pixel in the first sample convolution image may be multiplied by the above-mentioned disparity scaling value, and the disparity offset value may be added to obtain a calculation result. The above-mentioned calculation result and the average value of the pixel value of the pixel in the above-mentioned binocular convolution image are calculated to obtain the pixel value of the pixel in the adjusted binocular convolution image, and the adjusted binocular convolution image is obtained.

[0202] Specifically, the pixel value of the pixel in the adjusted binocular convolution image can be calculated according to the following formula:

[0203] stereodisp i ′=(stereodisp i +samdisp i *s i +t i ) / 2

[0204] Among them, i is the sequence number of the first sample convolution image, stereodisp i is the pixel value of the pixel in the binocular convolution image with the same size as the i-th first sample convolution image, samdisp i is the pixel value of the pixel in the i-th first sample convolution image, s i is the disparity scaling value corresponding to the i-th first sample convolution image, t i is the disparity offset value corresponding to the i-th first sample convolution image, stereodisp i ' is the calculated new pixel value.

[0205] If the adjusted binocular convolution image is different in size from the fourth down-sampled image with the largest size, the process returns to step S204. Otherwise, the process proceeds to step S209.

[0206] S209: adjusting the pixel values ​​of the pixels in the sample convolution image with the largest size according to the binocular convolution image, and using the adjusted sample convolution image as an output image having a parallax with the first sample image.

[0207] In order to make 3D videos present different 3D stereoscopic effects, the disparity between views in different 3D video frames is often different, that is, the disparity distribution between pixel values ​​of pixels in views in different 3D video frames is often different. In other words, the similarity of disparity between views in different 3D video frames is low, and 3D video frames have 3D disparity ambiguity. Training the neural network model with views of different 3D video frames as sample views will have different effects on the above neural network model, making it difficult for the above neural network model to be trained to convergence.

[0208] Therefore, when the neural network model processes the sample view, the pixel values ​​of the pixels in the largest sample convolution image can be adjusted according to the binocular convolution image, so that the pixel values ​​of the pixels in the largest sample convolution image of the first sample view after adjustment are affected by the disparity distribution between the first sample view and the second sample view, and are close to the largest convolution image obtained by performing virtual image processing on the second sample view. Thus, the obtained output image is close to the second sample view.

[0209] In one embodiment of the present invention, the following steps I to J can be used to adjust the pixel values ​​of pixels in the largest sample convolution image according to the above binocular convolution image.

[0210] Step I: According to the above binocular convolution image and the first sample convolution image, a disparity offset value and a disparity scaling value between the above binocular convolution image and the first sample convolution image are calculated.

[0211] Step J: According to the calculated disparity offset value and disparity scaling value, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted.

[0212] In one embodiment of the present invention, the pixel value of a pixel in the largest sample convolution image may be multiplied by the disparity scaling value and then added with the disparity offset value to obtain an adjusted pixel value.

[0213] Specifically, the pixel values ​​of the pixels in the largest sample convolution image can be adjusted by the following formula:

[0214] samdisp′=samdisp*s+t

[0215] Among them, samdisp is the pixel value of the pixel in the sample convolution image with the largest size, s is the calculated disparity scaling value, t is the calculated disparity offset value, and samdisp′ is the pixel value obtained after adjusting samdisp.

[0216] In another embodiment of the present invention, the following steps K-L may be used to adjust the pixel values ​​of the pixels in the sample convolution image with the largest size according to the above binocular convolution image.

[0217] Step K: enlarge the size of the obtained binocular convolution image to obtain an enlarged binocular convolution image with the same size as the largest sample convolution image.

[0218] Step L: The enlarged binocular convolution image is fused with the sample convolution image of the largest size according to the pixel position to obtain the adjusted sample convolution image.

[0219] S210: Calculate the loss of the neural network model based on the output image and the second sample view.

[0220] In an embodiment of the present invention, the output image and the second sample view may be used as input values ​​of a preset algorithm to calculate the above loss. The above preset algorithm may be an L1 algorithm, an SSIM (Structural Similarity) algorithm, a second-order gradient algorithm of a disparity depth map, or other algorithms. The above algorithms are existing algorithms and will not be described in detail herein.

[0221] S211: Adjusting the model parameters of the neural network model according to the loss.

[0222] Specifically, after adjusting the model parameters, if the preset training termination condition is met, the trained neural network model is determined as the image acquisition model.

[0223] Specifically, the above-mentioned training termination condition may be that the model parameters are adjusted for a preset number of times, or that the calculated loss of the above-mentioned neural network model is less than a preset loss or other training termination conditions.

[0224] Otherwise, return to execute the above steps S201-S210.

[0225] As can be seen from the above, the solution provided by the embodiment of the present invention uses the view in the 3D video frame as the sample view, calculates the loss based on the output image of the first sample view and the second sample view, adjusts the model parameters, and makes the output image output by the neural network model gradually approach the second sample image through training. Thus, the trained image acquisition model can obtain an image with parallax from the input 2D video frame based on the input 2D video frame.

[0226] In addition, since the above binocular convolution image can reflect the disparity between the first sample image and the second sample image, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted according to the above binocular convolution image, and the pixel values ​​of the pixels in the obtained sample convolution image are close to the pixel values ​​of the pixels in the binocular convolution image. Therefore, the adjusted sample convolution image can reflect the disparity between the first sample view and the second sample view. The output image obtained according to the adjusted sample convolution image and having a disparity with the first sample image is affected by the disparity between the first sample view and the second sample view and approaches the second sample image. Thereby, the calculated loss can be reduced, and the above neural network model is easier to be trained to convergence.

[0227] See also Figure 3, is a flow chart of the second image acquisition model training method provided by an embodiment of the present invention, wherein L is the left view in the sample view, R is the right view in the sample view, and R' is the output image. The figure indicates that L is the first sample view and R is the second sample view. LeftEncoder is used as the first down-sampled image, RightEncoder is used as the second down-sampled image, d is the sample convolution image, and sd is the binocular convolution image.

[0228] Among them, a first down-sampled image is obtained according to the first sample view, a second down-sampled image is obtained according to the second sample view, a sample convolution image is obtained according to the first down-sampled image, and a binocular convolution image is obtained according to the first down-sampled image and the second down-sampled image. Since the above-mentioned first sample view is a left view, the above-mentioned output image R' can be considered as the generated right view. The model loss is calculated according to the output image and the second sample view, and the model parameters of the above-mentioned neural network model can be adjusted according to the loss.

[0229] As can be seen from the above, the first downsampled image of the first sample view can be obtained based on the first sample view, and the second downsampled image of the first sample view can be obtained based on the second sample view. The first sample image and the second sample image correspond to different eyes of the observer, so it can be considered that the first downsampled image and the second downsampled image correspond to different eyes of the observer. A binocular convolution image can be obtained based on the first downsampled image and the second downsampled image, and a monocular convolution image corresponding to a single eye can be obtained based on the single first downsampled image. Since the above monocular convolution image comes from the first sample image, the output result corresponding to the first sample image can be obtained based on the above monocular convolution image and the binocular convolution image, as an image with parallax with the first sample image. Based on the above output result and the second sample image, the loss of the neural network model can be calculated, and then the model parameters of the neural network model can be adjusted. Therefore, the process of training the neural network model shown in the above text can be completed through the embodiment of the present invention.

[0230] Corresponding to the above-mentioned method for generating disparity images of 2D video frames, see Figure 4 The embodiment of the present invention further provides a schematic structural diagram of a device for generating a disparity image of a 2D video frame, the device comprising:

[0231] A downsampling module 401 is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame, and obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as a processed image processed by the next downsampling process;

[0232] An up-sampling module 402 is used to perform up-sampling processing on the first image to obtain an up-sampled image, wherein the first image is: an image with the smallest size among the obtained down-sampled images;

[0233] A convolution module 403 is used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0234] The image fusion module 404 is used to fuse the up-sampled image, the convolution image and the second image of the same size as the down-sampled image according to the pixel position to obtain a fused image, and perform up-sampling processing on the fused image to obtain a new up-sampled image. If the size of the new up-sampled image is different from the size of the 2D video frame, the convolution module 403 and the image fusion module 404 are returned to trigger the execution. The second image is: the image of the down-sampled image other than the first image;

[0235] The image acquisition module 405 is used to obtain a convolution image with the same size as the upsampled image according to the first convolution image if the size of the new upsampled image is the same as the size of the 2D video frame, and to fuse the upsampled image, the obtained convolution image, and an image with the same size as the upsampled image in the downsampled image according to pixel positions to obtain an image with parallax with the 2D video frame, wherein the first convolution image is: the largest image in the second convolution image, and the second convolution image is: a convolution image with a size smaller than the upsampled image.

[0236] As can be seen from the above, the solution provided by the embodiment of the present invention can obtain an image with parallax to the 2D video frame, and in the process of obtaining the image with parallax to the 2D video frame, the down-sampled image obtained based on the original 2D video frame is used to adjust the up-sampled image before generating the image with parallax to the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the parallax between the two is smaller, thereby maintaining the parallax of the generated 3D video within a smaller range, and reducing the difference between the parallax of different 2D video frames and the generated image.

[0237] See also Figure 5 , is a schematic diagram of the structure of a second 2D video frame parallax image generation device provided by an embodiment of the present invention, which is similar to the aforementioned Figure 4 Compared with the embodiment shown, the image acquisition module 405 includes:

[0238] A convolution submodule 405A is used to perform convolution processing on the up-sampled image obtained according to the first convolution image to obtain a third convolution image having the same size as that of the up-sampled image;

[0239] A pixel value adjustment submodule 405B is used to adjust the pixel values ​​of the pixels in the obtained third convolution image based on preset parameters so that the mean and variance of the pixel values ​​of the pixels in the convolution image are changed, and a convolution image having the same size as the upsampled image is obtained;

[0240] The image acquisition submodule 405C is used to fuse the upsampled image, the obtained convolution image and the image of the downsampled image with the same size as the upsampled image according to the pixel position to obtain an image with parallax with the 2D video frame.

[0241] In one embodiment of the present invention, the pixel value adjustment submodule 405B is specifically used for:

[0242] A preset disparity scaling value is multiplied by the pixel value of the pixel in the obtained second convolution image, and a preset disparity offset value is added to obtain a new pixel value.

[0243] It can be seen from the above that by adjusting the above preset parameters, the adjustment result of the convolution image can be changed, thereby changing the generated fusion result, so that the parallax between the obtained image and the 2D video frame changes, and further the parallax between the obtained image and the 2D video frame changes, so the parallax between the obtained image and the 2D video frame is adjustable.

[0244] In one embodiment of the present invention, the image acquisition module 405 is specifically used for:

[0245] The first convolution image is enlarged to obtain a convolution image with the same size as the upsampled image, and the upsampled image, the obtained convolution image, and an image with the same size as the upsampled image in the downsampled image are fused according to pixel positions to obtain an image with parallax with the 2D video frame.

[0246] In one embodiment of the present invention, an image having parallax with the 2D video frame is obtained by the following image obtaining module, and the image determining module includes:

[0247] The 2D video frame is input into a pre-trained image acquisition model to obtain an image having parallax with the 2D video frame, wherein the image acquisition model is used to implement the downsampling module to the image acquisition module.

[0248] It can be seen from the above that the image acquisition model can process the 2D video frame in the manner shown in the above method for generating a disparity image of a 2D video frame to obtain an image having a disparity with the 2D video frame.

[0249] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model through the following model training module.

[0250] See also Figure 6 , is a schematic diagram of the structure of a model training module provided in an embodiment of the present invention, wherein the model training module comprises:

[0251] The convolution image acquisition submodule 601 is used to input the first sample view and the second sample view in the 3D sample video frame into a preset neural network model respectively, to obtain a first down-sampled image of the first sample view and a second down-sampled image of the second sample view obtained by each down-sampling process, and a sample convolution image of the first sample view obtained by each convolution process, wherein the first sample view and the second sample view are two views in the 3D sample video frame respectively;

[0252] The first image fusion submodule 602 is used to fuse the first down-sampled image with the smallest size and the second down-sampled image with the smallest size according to pixel positions to obtain a sample fused image;

[0253] The binocular convolution submodule 603 is used to perform convolution processing on the sample fusion image to obtain a binocular convolution image with the same size as the sample fusion image;

[0254] The size enlargement submodule 604 is used to enlarge the size of the obtained binocular convolution image to obtain an enlarged binocular convolution image with the same size as the target down-sampled image, wherein the size of the target down-sampled image is the smallest among the third down-sampled images, and the third down-sampled image is the first down-sampled image with a size larger than the binocular convolution image before enlargement;

[0255] The second image fusion submodule 605 is used to fuse the enlarged binocular convolution image, the first down-sampled image with the same size as the enlarged binocular convolution image, and the second down-sampled image according to pixel positions to obtain a new sample fusion image;

[0256] An image convolution submodule 606 is used to perform convolution processing on the sample fusion image to obtain a new binocular convolution image with the same size as the sample fusion image;

[0257] The disparity calculation submodule 607 is used to calculate, according to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating the disparity offset size between the binocular convolution image and the first sample convolution image and a disparity scaling value indicating the disparity scaling multiple between the binocular convolution image and the first sample convolution image, wherein the first sample convolution image is a sample convolution image having the same size as that of the binocular convolution image;

[0258] The pixel value adjustment submodule 608 is used to adjust the pixel values ​​of the pixels in the binocular convolution image according to the calculated disparity offset value, disparity scaling value and the first sample convolution image to obtain an adjusted binocular convolution image. If the size of the adjusted binocular convolution image is different from the size of the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as a new binocular convolution image, and the size enlargement submodule 604 is returned to execute the pixel value adjustment submodule 608, wherein the fourth down-sampled image is: the first down-sampled image except the first down-sampled image with the largest size;

[0259] The image output submodule 609 is configured to adjust the pixel values ​​of the pixels in the sample convolution image with the largest size according to the binocular convolution image if the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, and use the adjusted sample convolution image as an output image having a parallax with the first sample image;

[0260] A loss calculation submodule 610, configured to calculate the loss of the neural network model according to the output image and the second sample view;

[0261] The parameter adjustment submodule 611 is used to adjust the model parameters of the neural network model according to the loss.

[0262] As can be seen from the above, the solution provided by the embodiment of the present invention uses the view in the 3D video frame as the sample view, calculates the loss based on the output image of the first sample view and the second sample view, adjusts the model parameters, and makes the output image output by the neural network model gradually approach the second sample image through training. Thus, the trained image acquisition model can obtain an image with parallax from the input 2D video frame based on the input 2D video frame.

[0263] In addition, since the above binocular convolution image can reflect the disparity between the first sample image and the second sample image, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted according to the above binocular convolution image, and the pixel values ​​of the pixels in the obtained sample convolution image are close to the pixel values ​​of the pixels in the binocular convolution image. Therefore, the adjusted sample convolution image can reflect the disparity between the first sample view and the second sample view. The output image obtained according to the adjusted sample convolution image and having a disparity with the first sample image is affected by the disparity between the first sample view and the second sample view and approaches the second sample image. Thereby, the calculated loss can be reduced, and the above neural network model is easier to be trained to convergence.

[0264] In one embodiment of the present invention, the image output submodule 609 is specifically used for:

[0265] If the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, a disparity offset value and a disparity scaling value between the binocular convolution image and the first sample convolution image are calculated according to the binocular convolution image and the first sample convolution image;

[0266] According to the calculated disparity offset value and disparity scaling value, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted, and the adjusted sample convolution image is used as an output image having a disparity with the first sample image.

[0267] In one embodiment of the present invention, the image output submodule 609 is specifically used for:

[0268] If the size of the adjusted binocular convolution image is the same as the size of the fourth down-sampled image with the largest size, the obtained binocular convolution image is enlarged to obtain an enlarged binocular convolution image with the same size as the largest sample convolution image;

[0269] The enlarged binocular convolution image and the sample convolution image with the largest size are fused according to pixel positions to obtain an adjusted sample convolution image, and the adjusted sample convolution image is used as an output image with parallax with the first sample image.

[0270] In one embodiment of the present invention, the disparity calculation submodule 607 is specifically used for:

[0271] Calculate the product between the first sample convolution image and the transposed value of the first convolution image as a first result;

[0272] Inverting the first result to obtain a second result;

[0273] Calculating the product of the first sample convolution image and the binocular convolution image as a third result;

[0274] The product of the second result and the third result is calculated to obtain the disparity offset value representing the disparity offset size between the binocular convolution image and the first sample convolution image and the disparity scaling value representing the disparity scaling multiple between the binocular convolution image and the first sample convolution image.

[0275] In one embodiment of the present invention, the pixel value adjustment submodule 608 is specifically used to:

[0276] Multiplying a pixel value of a pixel in the first sample convolution image by the disparity scaling value, and adding the disparity offset value to obtain a calculation result;

[0277] Calculate the average value of the above calculation result and the pixel value of the pixel in the binocular convolution image to obtain the pixel value of the pixel in the adjusted binocular convolution image, and obtain the adjusted binocular convolution image;

[0278] If the size of the adjusted binocular convolution image is different from the size of the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as a new binocular convolution image, and the size enlargement submodule 604 to the pixel value adjustment submodule 608 are returned to execute.

[0279] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0280] Memory 703, used for storing computer programs;

[0281] The processor 701 is configured to implement any method step of the above-mentioned method for generating a parallax image of a 2D video frame when executing the program stored in the memory 703 .

[0282] When the electronic device provided by the embodiment of the present invention is applied to generate a disparity image of a 2D video frame, the scheme provided by the embodiment of the present invention can obtain an image that has a disparity with the 2D video frame, and in the process of obtaining the image that has a disparity with the 2D video frame, the down-sampled image obtained based on the original 2D video frame is used to adjust the up-sampled image before generating the image that has a disparity with the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the disparity between the two is smaller, thereby maintaining the disparity of the generated 3D video within a smaller range, and reducing the difference between the disparity of different 2D video frames and the generated image.

[0283] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0284] The communication interface is used for communication between the above terminal and other devices.

[0285] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0286] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0287] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method for generating a disparity image of a 2D video frame described in any of the above embodiments is implemented.

[0288] When the computer program stored in the computer-readable storage medium provided by the present embodiment is applied to generate the disparity image of the 2D video frame, the scheme provided by the embodiment of the present invention can obtain the image with disparity with the 2D video frame, and in the process of obtaining the image with disparity with the 2D video frame, the down-sampled image obtained based on the original 2D video frame is used to adjust the up-sampled image before generating the image with disparity with the 2D video frame, so that the obtained image is closer to the original 2D video frame, and the disparity between the two is smaller, thereby maintaining the disparity of the generated 3D video within a smaller range, and reducing the difference between the disparity of different 2D video frames and the generated image.

[0289] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for generating a disparity image of a 2D video frame described in any one of the above embodiments.

[0290] When the computer program product provided in this embodiment is executed to generate a disparity image of a 2D video frame, the scheme provided in the embodiment of the present invention can obtain an image with a disparity with the 2D video frame, and in the process of obtaining the image with a disparity with the 2D video frame, the up-sampled image before generating the image with a disparity with the 2D video frame is adjusted using a down-sampled image obtained based on the original 2D video frame, so that the obtained image is closer to the original 2D video frame, and the disparity between the two is smaller, thereby maintaining the disparity of the generated 3D video within a smaller range, and reducing the difference between the disparity of different 2D video frames and the generated image.

[0291] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0292] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0293] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0294] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a disparity image of a 2D video frame, characterized in that: The method comprises: Performing a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process; Performing up-sampling processing on the first image to obtain an up-sampled image, wherein the first image is: an image with the smallest size among the obtained down-sampled images; Performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image; The upsampled image, the convolution image and the second image with the same size as the upsampled image are subjected to pixel value fusion according to pixel positions to obtain a fused image, the fused image is subjected to upsampling processing to obtain a new upsampled image, if the size of the new upsampled image is different from the size of the 2D video frame, then the steps of performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image, and the upsampled image, the convolution image and the second image with the same size as the upsampled image are subjected to pixel value fusion according to pixel positions to obtain a fused image, and the fused image is subjected to upsampling processing to obtain a new upsampled image, wherein the second image is an image in the downsampled image other than the first image; Otherwise, according to the first convolution image, a convolution image having the same size as that of an upsampled image obtained after upsampling the fused image is obtained, and pixel values ​​of the upsampled image obtained after upsampling the fused image, the obtained convolution image, and an image having the same size as that of the upsampled image obtained after upsampling the fused image are fused according to pixel positions to obtain an image having parallax with the 2D video frame, wherein the first convolution image is: the image with the largest size among the second convolution images, and the second convolution image is: a convolution image having a size smaller than that of the upsampled image obtained after upsampling the fused image, and the size of the upsampled image obtained after upsampling the fused image is the same as that of the 2D video frame.

2. The method according to claim 1, characterized in that The step of obtaining, according to the first convolution image, a convolution image having the same size as that of an upsampled image obtained after upsampling the fused image comprises: Performing convolution processing on the up-sampled image obtained by up-sampling the fused image according to the first convolution image, to obtain a third convolution image having the same size as that of the up-sampled image obtained by up-sampling the fused image; Based on preset parameters, the pixel values ​​of the pixels in the obtained third convolution image are adjusted so that the mean and variance of the pixel values ​​of the pixels in the convolution image are changed, and a convolution image having the same size as the upsampled image obtained after upsampling the fused image is obtained.

3. The method according to claim 2, characterized in that The step of adjusting the pixel values ​​of pixels in the obtained third convolution image based on preset parameters includes: A preset disparity scaling value is multiplied by the pixel value of the pixel in the obtained third convolution image, and a preset disparity offset value is added to obtain a new pixel value.

4. The method according to claim 1, characterized in that: The step of obtaining, according to the first convolution image, a convolution image having the same size as that of an upsampled image obtained after upsampling the fused image comprises: The first convolution image is upscaled to obtain a convolution image having the same size as that of an upsampled image obtained by upsampling the fused image.

5. The method according to claim 1, characterized in that An image having a parallax with the 2D video frame is obtained in the following manner, including: The 2D video frame is input into a pre-trained image acquisition model to obtain an image with parallax to the 2D video frame, wherein the image acquisition model is used to: implement the cascade downsampling process of the 2D video frame including a preset number of downsampling processes, obtain the downsampled image obtained by each downsampling process, and perform pixel value fusion according to pixel position to obtain the image with parallax to the 2D video frame.

6. The method according to claim 5, characterized in that The image acquisition model is obtained by training the preset neural network model in the following manner: Inputting a first sample view and a second sample view in the 3D sample video frame into a preset neural network model respectively, obtaining a first downsampled image of the first sample view and a second downsampled image of the second sample view obtained by each downsampling process, and a sample convolution image of the first sample view obtained by each convolution process, wherein the first sample view and the second sample view are two views in the 3D sample video frame respectively; fusing pixel values ​​of the first down-sampled image with the smallest size and the second down-sampled image with the smallest size according to pixel positions to obtain a sample fused image; Perform convolution processing on the sample fusion image to obtain a binocular convolution image with the same size as the sample fusion image; The obtained binocular convolution image is enlarged to obtain an enlarged binocular convolution image having the same size as the target down-sampled image, wherein the size of the target down-sampled image is the smallest among the third down-sampled images, and the third down-sampled image is the first down-sampled image having a size larger than the binocular convolution image before enlargement; The enlarged binocular convolution image, the first down-sampled image and the second down-sampled image having the same size as the enlarged binocular convolution image are fused according to pixel values ​​at pixel positions to obtain a new sample fused image; Performing convolution processing on the sample fusion image to obtain a new binocular convolution image with the same size as the sample fusion image; According to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating the disparity offset size between the new binocular convolution image and the first sample convolution image and a disparity scaling value indicating the disparity scaling multiple between the new binocular convolution image and the first sample convolution image are calculated, wherein the first sample convolution image is a sample convolution image having the same size as the new binocular convolution image; According to the calculated disparity offset value, disparity scaling value and the first sample convolution image, pixel values ​​of pixels in the new binocular convolution image are adjusted to obtain an adjusted binocular convolution image; If the adjusted binocular convolution image is different in size from the fourth down-sampled image with the largest size, the adjusted binocular convolution image is used as the new binocular convolution image, and the step of performing the size enlargement processing on the obtained binocular convolution image to obtain an enlarged binocular convolution image with the same size as the target down-sampled image is returned to the step of adjusting the pixel values ​​of the pixels in the new binocular convolution image according to the calculated disparity offset value, the disparity scaling value and the first sample convolution image to obtain the adjusted binocular convolution image, wherein the fourth down-sampled image is: the first down-sampled image except the first down-sampled image with the largest size; Otherwise, adjusting the pixel values ​​of the pixels in the largest sample convolution image according to the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size, and using the adjusted sample convolution image as the output image having a parallax with the first sample image; Calculating the loss of the neural network model according to the output image and the second sample view; Model parameters of the neural network model are adjusted according to the loss.

7. The method according to claim 6, characterized in that The step of adjusting the pixel values ​​of pixels in the largest sample convolution image according to the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size comprises: According to the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size and the first sample convolution image, a disparity offset value and a disparity scaling value between the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size and the first sample convolution image are calculated; According to the calculated disparity offset value and disparity scaling value, the pixel values ​​of the pixels in the sample convolution image with the largest size are adjusted.

8. The method according to claim 6, characterized in that The step of adjusting the pixel values ​​of pixels in the largest sample convolution image according to the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size comprises: Performing a size enlargement process on the adjusted binocular convolution image having the same size as the fourth down-sampled image having the largest size, to obtain an enlarged binocular convolution image having the same size as the largest sample convolution image; The enlarged binocular convolution image is fused with the sample convolution image with the largest size according to the pixel position to obtain the adjusted sample convolution image.

9. The method according to claim 6, characterized in that The method of calculating, according to the new binocular convolution image and the first sample convolution image, a disparity offset value indicating a disparity offset size between the new binocular convolution image and the first sample convolution image and a disparity scaling value indicating a disparity scaling factor between the new binocular convolution image and the first sample convolution image comprises: Calculate the product of the first sample convolution image and the transposed value of the first sample convolution image as a first result; Inverting the first result to obtain a second result; Calculate the product of the first sample convolution image and the new binocular convolution image as a third result; The product of the second result and the third result is calculated to obtain the disparity offset value representing the disparity offset size between the new binocular convolution image and the first sample convolution image and the disparity scaling value representing the disparity scaling multiple between the new binocular convolution image and the first sample convolution image.

10. The method according to any one of claims 6 to 9, characterized in that: The step of adjusting the pixel values ​​of pixels in the new binocular convolution image according to the calculated disparity offset value, the disparity scaling value and the first sample convolution image to obtain the adjusted binocular convolution image includes: Multiplying a pixel value of a pixel in the first sample convolution image by the disparity scaling value, and adding the disparity offset value to obtain a calculation result; The average value of the calculation result and the pixel value of the pixel in the new binocular convolution image is calculated to obtain the pixel value of the pixel in the adjusted binocular convolution image, and the adjusted binocular convolution image is obtained.

11. A parallax image generation device for a 2D video frame, characterized in that: The device comprises: A downsampling module is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process; An up-sampling module is used to perform up-sampling processing on the first image to obtain an up-sampled image, wherein the first image is: an image with the smallest size among the obtained down-sampled images; A convolution module, used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image; an image fusion module, configured to fuse the upsampled image, the convolution image, and a second image of the same size as the upsampled image according to pixel positions to obtain a fused image, perform upsampling on the fused image to obtain a new upsampled image, and return to trigger the execution of the convolution module and the image fusion module if the size of the new upsampled image is different from the size of the 2D video frame, wherein the second image is: an image in the downsampled image other than the first image; An image acquisition module is used for obtaining, according to the first convolution image, a convolution image having the same size as that of an upsampled image obtained after upsampling the fused image if the size of the new upsampled image is the same as that of the 2D video frame, and fusing pixel values ​​of the upsampled image obtained after upsampling the fused image, the obtained convolution image, and an image having the same size as that of the upsampled image obtained after upsampling the fused image according to pixel positions to obtain an image having parallax with the 2D video frame, wherein the first convolution image is: an image with the largest size among the second convolution images, the second convolution image is: a convolution image having a size smaller than that of the upsampled image obtained after upsampling the fused image, and the size of the upsampled image obtained after upsampling the fused image is the same as that of the 2D video frame.

12. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 10 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 10 are implemented.

Images