A method and device for generating parallax images of 2D video frames

Abstract

The embodiment of the present invention provides a method and device for generating a parallax image of a 2D video frame, which relates to the field of video processing technology, wherein the method includes: performing cascade downsampling processing on the 2D video frame; adjusting the pixel value of each pixel in the second image based on a preset parameter with a fixed value; performing upsampling processing on the first image; performing convolution processing on the upsampled image; performing pixel value fusion on the upsampled image, the convolution image, and the second image with the same pixel value adjustment as the upsampled image; if the size of the fused image is different from the size of the 2D video frame, then upsampling the fused image, and returning to the step of performing convolution processing on the upsampled image; otherwise, determining the fused image as an image with parallax to the 2D video frame. Applying the solution provided by the embodiment of the present invention to obtain an image with parallax to the 2D video frame can reduce the difference between the parallax of different 2D video frames and the generated image.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for generating a parallax image of a 2D video frame. Background Art

[0002] From the perspective of visual perception, videos can be divided into two categories: 2D videos and 3D videos. In 2D videos, each video frame is an image, while in 3D videos, each video frame contains two views with parallax, where each view is an image. When users watch 3D videos, the left and right eyes see different views. Since there is parallax between the two views, users can see a 3D stereoscopic effect. 3D videos are currently widely used in many fields such as movies and television.

[0003] At present, in the process of converting 2D video frames into 3D videos, an image with parallax with the 2D video frame can be generated based on the 2D video frame in the 2D video, and then a 3D video frame can be obtained based on the 2D video frame and the generated image. However, in the process of generating an image with parallax with the 2D video frame based on the 2D video frame, there is a problem of parallax ambiguity, that is, images are generated based on similar 2D video frames, and there is parallax between the generated image and the 2D video frame, but the difference between the parallaxes corresponding to different 2D video frames is large. As a result, the 3D effect displayed by the 3D video generated based on the similar 2D video frame and the generated image is different, so the 3D effect of the 3D video obtained by the conversion of the prior art is unstable. In order to improve the stability of the 3D effect of the 3D video obtained by converting the 2D video, it is necessary to provide a parallax image generation method of a 2D video frame to reduce the difference between the parallax of different 2D video frames and the generated image. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and device for generating a disparity image of a 2D video frame, so as to reduce the difference between the disparity of different 2D video frames and the generated image. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a disparity image of a 2D video frame, the method comprising:

[0006] Performing a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process;

[0007] Based on preset parameters with fixed values, pixel values ​​of pixels in each second image other than the first image in each down-sampled image obtained by the down-sampling process are adjusted so that the mean value and variance of the pixel values ​​of the pixels in each second image are changed, wherein the first image is: an image with the smallest size in the obtained down-sampled images;

[0008] Performing upsampling processing on the first image to obtain an upsampled image;

[0009] Performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0010] The upsampled image, the convolution image, and the second image with the same size as the upsampled image and adjusted pixel values ​​are fused according to pixel positions to obtain a fused image;

[0011] If the size of the fused image is different from the size of the 2D video frame, upsampling the fused image to obtain a new upsampled image, and returning to the step of performing convolution on the upsampled image to obtain a convolution image of the same size as the upsampled image, and fusing the upsampled image, the convolution image, and the second image of the same size as the upsampled image by adjusting pixel values ​​according to pixel positions to obtain a fused image;

[0012] Otherwise, the fused image is determined as an image having parallax with the 2D video frame.

[0013] In one embodiment of the present invention, the step of adjusting the pixel value of each pixel in each second image other than the first image in the downsampled image obtained by each downsampling process based on the preset parameter with a fixed value includes:

[0014] The pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process is multiplied by a preset variance, and a preset average value is added to obtain an adjusted pixel value of each pixel.

[0015] In one embodiment of the present invention, the obtaining of the image having parallax with the 2D video frame in the following manner includes:

[0016] The 2D video frame is input into a pre-trained image acquisition model to obtain an image that has parallax with the 2D video frame, wherein the model parameters of the image acquisition model include the preset parameters, and the image acquisition model is used to: implement the cascade downsampling processing of the 2D video frame including a preset number of downsampling processings, obtain the downsampled image obtained by each downsampling processing, and determine the fused image as an image that has parallax with the 2D video frame.

[0017] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model in the following manner:

[0018] Determine a mean and a variance of pixel value differences between pixels in two sample views in a 3D sample video frame, use the determined mean as a sample mean, and use the determined variance as a sample variance;

[0019] Obtaining a preset number of disparity arrays, wherein the elements contained in the disparity arrays represent disparities;

[0020] Selecting a disparity from each disparity array to obtain a first array including the selected disparity;

[0021] According to the sample mean and the sample variance, adjusting the disparity in the first array so that the distribution of the disparity in the first array approaches the distribution of the pixel value difference between the pixels in the two sample views, to obtain a second array;

[0022] Calculate the disparity mean and disparity variance of the second array;

[0023] Setting the preset parameters in the model parameters of the preset neural network model to the disparity mean and the disparity variance;

[0024] Inputting a sample view into the neural network model to obtain an output image having a parallax with the sample view;

[0025] Calculating the loss of the neural network model according to the output image and another sample view;

[0026] Model parameters of the neural network model are adjusted according to the loss.

[0027] In one embodiment of the present invention, determining the mean and variance of the pixel value differences between pixels in two sample views in the 3D sample video frame includes:

[0028] Extracting a correlation feature representing a correlation relationship between the two sample views;

[0029] Performing convolution processing on the associated features;

[0030] The obtained convolution processing result is fully connected to obtain the mean and variance of the pixel value difference between the pixels in the two sample views.

[0031] In a second aspect, an embodiment of the present invention provides a device for generating a disparity image of a 2D video frame, the device comprising:

[0032] A downsampling module is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as a processed image processed by the next downsampling process;

[0033] A pixel value adjustment module, for adjusting the pixel values ​​of pixels in each second image other than the first image in each down-sampled image obtained by down-sampling processing based on preset parameters with fixed values, so that the mean value and variance of the pixel values ​​of the pixels in each second image are changed, wherein the first image is: an image with the smallest size in the obtained down-sampled images;

[0034] A first upsampling module, configured to perform upsampling processing on the first image to obtain an upsampled image;

[0035] A convolution module, used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0036] A pixel value fusion module, used for fusing the upsampled image, the convolution image and the second image with the same size as the upsampled image after the pixel value adjustment according to the pixel position to obtain a fused image;

[0037] A second upsampling module is used for upsampling the fused image to obtain a new upsampled image if the size of the fused image obtained by the pixel value fusion module is different from the size of the 2D video frame, and returning to trigger the execution of the convolution module and the pixel value fusion module;

[0038] The image determination module is configured to determine the fused image as an image having parallax with the 2D video frame if the size of the fused image obtained by the pixel value fusion module is the same as the size of the 2D video frame.

[0039] In one embodiment of the present invention, the pixel value adjustment module is specifically used to:

[0040] The pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process is multiplied by a preset variance, and a preset average value is added to obtain an adjusted pixel value of each pixel.

[0041] In one embodiment of the present invention, an image having parallax with the 2D video frame is obtained by the following image obtaining module:

[0042] An image acquisition module is used to input a 2D video frame into a pre-trained image acquisition model to obtain an image with parallax with the 2D video frame, wherein the model parameters of the image acquisition model include the preset parameters, and the image acquisition model is used to: implement the downsampling module to the image determination module.

[0043] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model through the following model training module, and the model training module includes:

[0044] A mean-variance determination submodule, used to determine the mean and variance of the pixel value difference between pixels in two sample views in the 3D sample video frame, and use the determined mean as the sample mean and the determined variance as the sample variance;

[0045] An array generation submodule, used to obtain a preset number of disparity arrays, wherein the elements contained in the disparity arrays represent disparities;

[0046] a disparity selection submodule, configured to select a disparity from each disparity array to obtain a first array including the selected disparity;

[0047] a disparity adjustment submodule, configured to adjust the disparity in the first array according to the sample mean and the sample variance so that the distribution of the disparity in the first array approaches the distribution of the pixel value difference between the pixels in the two sample views, thereby obtaining a second array;

[0048] A mean-variance calculation submodule, used for calculating the disparity mean and disparity variance of the second array;

[0049] A parameter setting submodule, used to set the preset parameters in the model parameters of the preset neural network model to the disparity mean and the disparity variance;

[0050] An image acquisition submodule, used for inputting a sample view into the neural network model to obtain an output image having a parallax with the sample view;

[0051] A loss calculation submodule, used to calculate the loss of the neural network model according to the output image and another sample view;

[0052] A parameter adjustment submodule is used to adjust the model parameters of the neural network model according to the loss.

[0053] In one embodiment of the present invention, the mean variance determination submodule is specifically used for:

[0054] Extracting a correlation feature representing a correlation relationship between the two sample views;

[0055] Performing convolution processing on the associated features;

[0056] The obtained convolution processing result is fully connected to obtain the mean and variance of the pixel value difference between the pixels in the two sample views.

[0057] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0058] Memory, used to store computer programs;

[0059] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0060] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0061] In a fifth aspect, in yet another aspect of the implementation of the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any method step described in the first aspect.

[0062] In the parallax image generation scheme of a 2D video frame provided in an embodiment of the present invention, a cascade downsampling process is performed on the 2D video frame to obtain a downsampled image obtained by each downsampling process, and the pixel values ​​of the pixels in the second image are adjusted based on preset parameters with fixed values. The first image is upsampled to obtain an upsampled image, and the upsampled image is convolved to obtain a convolved image of the same size as the upsampled image. The upsampled image, the convolved image, and the second image with the same pixel value adjustment as the upsampled image are fused to obtain a fused image. The above process is repeated until the size of the fused image is the same as the 2D video frame, and the finally obtained fused image is determined as an image with parallax from the 2D video frame.

[0063] As can be seen from the above, the scheme provided by the embodiment of the present invention can obtain an image with parallax with the 2D video frame. Moreover, in the process of obtaining an image with parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the scheme provided by the embodiment of the present invention can reduce the difference between the parallax between the 2D video frame and the generated image. Moreover, by adjusting the above preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame is adjustable. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0065] Figure 1 A schematic diagram of a process of generating a disparity image of a 2D video frame provided by an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of a flow chart of a first image acquisition model training method provided in an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of a flow chart of a second image acquisition model training method provided in an embodiment of the present invention;

[0068] Figure 4 A schematic structural diagram of a disparity image generating device for a 2D video frame provided by an embodiment of the present invention;

[0069] Figure 5 A schematic diagram of the structure of a model training module provided by an embodiment of the present invention;

[0070] Figure 6 The present invention is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0072] Since the difference between the disparity of different 2D video frames and the generated images in the prior art is relatively large, in order to solve this problem, an embodiment of the present invention provides a method and device for generating a disparity image of a 2D video frame.

[0073] In one embodiment of the present invention, a method for generating a disparity image of a 2D video frame is provided, the method comprising:

[0074] Performing a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process;

[0075] Based on preset parameters with fixed values, the pixel values ​​of pixels in each second image other than the first image in each down-sampled image obtained by the down-sampling process are adjusted so that the mean value and variance of the pixel values ​​of the pixels in each second image are changed, wherein the first image is: an image with the smallest size in the obtained down-sampled images;

[0076] Performing upsampling processing on the first image to obtain an upsampled image;

[0077] Performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0078] The upsampled image, the convolution image, and the second image with the same size as the upsampled image and adjusted pixel values ​​are fused according to pixel positions to obtain a fused image;

[0079] If the size of the fused image is different from the size of the 2D video frame, upsampling the fused image to obtain a new upsampled image, returning to perform the convolution process on the upsampled image to obtain a convolution image of the same size as the upsampled image, and fusing the upsampled image, the convolution image, and the second image of the same size as the upsampled image according to pixel positions to obtain a fused image;

[0080] Otherwise, the fused image is determined as an image having parallax with the 2D video frame.

[0081] As can be seen from the above, the scheme provided by the embodiment of the present invention can obtain an image with parallax with the 2D video frame. Moreover, in the process of obtaining an image with parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the scheme provided by the embodiment of the present invention can reduce the difference between the parallax between the 2D video frame and the generated image. Moreover, by adjusting the above preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame is adjustable.

[0082] The following describes a method and device for generating a disparity image of a 2D video frame provided by an embodiment of the present invention through specific embodiments.

[0083] Specifically, each 3D video frame in a 3D video contains two 2D images with parallax between them. The two 2D images can be called two views of the 3D video frame. When a user watches a 3D video, the left eye and the right eye see different views. Therefore, according to the different eyes corresponding to the views, the views of the 3D video frame can be divided into a left view and a right view. If the 2D video frame is used as the left view, the image with parallax between the 2D video frame and the 2D video frame is the right view. Conversely, if the 2D video frame is used as the right view, the image with parallax between the 2D video frame and the 2D video frame is the left view.

[0084] See also Figure 1 , an embodiment of the present invention provides a flow chart of a method for generating a disparity image of a 2D video frame. For each 2D video frame, an image having a disparity with the 2D video frame is obtained according to the following steps S101-S107.

[0085] S101: performing a cascade downsampling process including a preset number of downsampling processes on a 2D video frame to obtain a downsampled image obtained by each downsampling process.

[0086] The result of each downsampling process in the above cascade downsampling process is used as the image processed by the next downsampling process.

[0087] For example, the above preset number of times can be 5 times. After the 2D video frame is down-sampled for the first time, the result of the first down-sampling process is down-sampled for the second time, and the result of the second down-sampling process is down-sampled for the third time, and so on. A total of 5 down-sampling processes are performed to obtain 5 down-sampled images.

[0088] Since the downsampling process will reduce the size of the image, the sizes of the obtained downsampled images are different. Specifically, the downsampling process can be: reducing the size of the image after the downsampling process to the size of the original image. processing.

[0089] For example, the size of the above-mentioned 2D video frame is 1080 pixels × 720 pixels. The size of the downsampled image obtained after one downsampling process is 540 pixels × 360 pixels. The size of the downsampled image obtained after the second downsampling process is 270 pixels × 180 pixels, and so on. The size of the downsampled image obtained after each downsampling process is half of the size before the downsampling process.

[0090] In one embodiment of the present invention, the above-mentioned downsampling process can be implemented through convolution processing and pooling processing.

[0091] S102: Based on preset parameters with fixed values, pixel values ​​of pixels in each second image except the first image in the downsampled image obtained by each downsampling process are adjusted so that the mean and variance of the pixel values ​​of the pixels in each second image are changed.

[0092] The first image is the image with the smallest size among the obtained down-sampled images.

[0093] Specifically, since the size of the obtained down-sampled image is successively reduced after each down-sampling process, the above-mentioned first image is the image obtained by the last down-sampling process in the cascade down-sampling process.

[0094] In one embodiment of the present invention, the above step S102 can be implemented by the following step A.

[0095] Step A: multiply the pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process by a preset variance, and add a preset average value to obtain an adjusted pixel value of each pixel.

[0096] Specifically, the above-mentioned preset variance and preset average value are both the above-mentioned preset parameters. According to mathematical principles, multiplying each pixel value by the same preset variance can change the variance of the overall pixel value, and adding the same preset average value to each pixel value can change the average value of the overall pixel value.

[0097] The pixel value of each pixel in the second image except the first image in the obtained downsampled image may be adjusted according to the following formula:

[0098] encoder′ i =encoder i *sigma+mean

[0099] Among them, the above i is the sequence number of the second image, encoder i is the pixel value of the i-th pixel in the second image, sigma and mean are the above preset parameters, encoder′ i For encoder i The pixel value obtained after adjustment.

[0100] Specifically, by changing the above preset parameters, the calculated encoder′ can be adjusted i , thereby causing the adjusted second image to change.

[0101] S103: Perform up-sampling processing on the first image to obtain an up-sampled image.

[0102] Specifically, the upsampling process is the inverse transformation process of the downsampling process in the cascade downsampling process, that is, the size of the first image processed by the upsampling process is the same as the size of the image obtained by the downsampling process, and the size of the upsampled image obtained by the upsampling process is the same as the size of the downsampled image processed by the downsampling process.

[0103] For example, if the downsampling process reduces the size of the image after the downsampling process to the original image The above upsampling process can increase the size of the image after the upsampling process to twice the original image.

[0104] For example, if the size of the first image is 270 pixels×180 pixels, the size of the upsampled image obtained after upsampling the first image is 540 pixels×360 pixels.

[0105] The above-mentioned up-sampling process can be implemented by a bilinear interpolation method.

[0106] In addition, the above-mentioned downsampling process can be called an encoding process, and the above-mentioned upsampling process can be called a decoding process.

[0107] S104: Perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image.

[0108] Specifically, the convolution process may be a convolution process that keeps the image size unchanged. Then the size of the convolution image obtained after the convolution process is the same as the size of the upsampled image.

[0109] S105: fusing the up-sampled image, the convolution image, and the second image with the same size as the up-sampled image and after the pixel values ​​are adjusted according to pixel positions to obtain a fused image.

[0110] Since the up-sampling process and the down-sampling process are inverse transformation processes, there exists a down-sampled image having the same size as the up-sampled image obtained after the up-sampling process.

[0111] Specifically, since the upsampled image, the convolution image and the adjusted upsampled image have the same size, the pixels of the above images correspond to each other according to the pixel positions, that is, the pixels with the same pixel positions correspond to each other, for example, the pixels in the upper left corners of the three correspond to each other. The upsampled image, the convolution image and the adjusted downsampled image with the same size are fused, and the size of the obtained fused image is also the same as that of the upsampled image.

[0112] The pixel values ​​of the corresponding pixels may be averaged, weighted averaged, etc., and the result of the calculation may be used as the pixel value of the pixel at the corresponding pixel position of the fused image, thereby obtaining the fused image. For example, the pixel values ​​of the pixels at the upper left corner of the three pixels may be weighted averaged, and the result of the calculation may be used as the pixel value of the pixel at the upper left corner of the fused image.

[0113] In addition, the up-sampled image, the convolution image and the adjusted down-sampled image may also be fused by using methods such as a logical filter algorithm, a pyramid decomposition algorithm, and a high-pass filter algorithm, which will not be described in detail here.

[0114] S106: If the size of the fused image is different from the size of the 2D video frame, up-sampling the fused image to obtain a new up-sampled image.

[0115] Specifically, the sizes of the two views of the 3D video frame are the same, so if the size of the fused image is different from the size of the 2D video frame, it means that the obtained fused image is not an image with parallax with the 2D video frame, so it is necessary to continue processing the upsampled image and return to execute the above steps S104-S105. After returning to the above step S104, convolution processing will be performed on the new upsampled image, and fusion processing will be performed on the newly obtained convolution image, the upsampled image and the second image.

[0116] In addition, the process of performing cascade downsampling on the 2D video frame in steps S101 to S106 and then performing upsampling multiple times continuously until the size of the obtained fused image is the same as that of the 2D video frame can be called pyramid coding.

[0117] S107: If the size of the fused image is the same as the size of the 2D video frame, determine the fused image as an image having parallax with the 2D video frame.

[0118] As can be seen from the above, the scheme provided by the embodiment of the present invention can obtain an image with parallax with the 2D video frame. Moreover, in the process of obtaining an image with parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the scheme provided by the embodiment of the present invention can reduce the difference between the parallax between the 2D video frame and the generated image. Moreover, by adjusting the above preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame is adjustable.

[0119] In one embodiment of the present invention, the above steps S101-S107 can be implemented by the following step B.

[0120] Step B: Input the 2D video frame into a pre-trained image acquisition model to obtain an image with parallax to the 2D video frame.

[0121] The model parameters of the image acquisition model include the preset parameters. The image acquisition model is used to: implement the steps S101-S107.

[0122] Specifically, the above Figure 1 Compared with the embodiment shown, the above-mentioned downsampling processing can be implemented by the convolution layer and the pooling layer of the above-mentioned image acquisition model, the above-mentioned upsampling processing can be implemented by the deconvolution layer of the above-mentioned image acquisition model, and the above-mentioned convolution processing can be implemented by the convolution layer of the above-mentioned image acquisition model.

[0123] The above image acquisition model can be obtained according to the above Figure 1 The 2D video frame is processed in the manner shown to obtain an image with parallax with the 2D video frame, thereby achieving the above Figure 1 The embodiment shown.

[0124] See also Figure 2 , the embodiment of the present invention provides a flow chart of a first method for training an image acquisition model. Specifically, the preset neural network model is trained through the following steps S201-S209 to obtain the above-mentioned image acquisition model.

[0125] S201: Determine a mean and a variance of pixel value differences between pixels in two sample views in a 3D sample video frame, use the determined mean as a sample mean, and use the determined variance as a sample variance.

[0126] Specifically, the two sample views have the same size, and the pixel value differences between corresponding pixels in the two sample views may be determined according to pixel positions, thereby calculating the sample mean and sample variance of the sample disparity.

[0127] Since both the mean and the variance are statistical values ​​that can reflect the data distribution, the sample mean and the sample variance can represent the distribution of the pixel value difference between the pixels in the two sample views.

[0128] In the embodiment of the present invention, the sample mean and sample variance can be determined by following steps C-E.

[0129] Step C: Extract the association features representing the association relationship between the two sample views.

[0130] Specifically, the correlation features between the two sample views may be extracted through a correlation layer in the neural network model. The number of the correlation layer may be one.

[0131] The two sample views may be input into the association layer, and the output result of the association layer may be used as the association feature.

[0132] Step D: Perform convolution processing on the above-mentioned associated features.

[0133] Among them, the associated features can be convolved through the convolution layer in the neural network model.

[0134] Specifically, the associated features can be convolved through continuous convolution layers. For example, the associated features can be convolved through 5 convolution layers, that is, the first convolution layer performs convolution processing on the associated features to obtain a first convolution result; the second convolution layer performs convolution processing on the first convolution result to obtain a second convolution result; the third convolution layer performs convolution processing on the second convolution result to obtain a third convolution result, and so on. The fifth convolution result output by the fifth convolution layer is used as the final convolution processing result for convolution processing on the associated features.

[0135] Step E: Perform full connection processing on the obtained convolution processing result to obtain the mean and variance of the pixel value difference between the two sample views.

[0136] Among them, the convolution processing results can be respectively input into two different fully connected layers in the neural network model to perform full connection processing on the above convolution processing results to obtain the above sample mean and sample variance respectively.

[0137] S202: Obtain a preset number of disparity arrays.

[0138] The elements contained in the above disparity array represent disparity. The above disparity array may be randomly generated or pre-set, without any specific limitation.

[0139] Specifically, since the distribution of pixel value differences between pixels in the sample view is often close to a normal distribution, the disparity contained in the above disparity array may conform to a normal distribution, so that the distribution of the disparity in the disparity array is close to the distribution of the actual pixel value differences.

[0140] In addition, the disparities included in the above disparity array may also conform to other distribution forms such as average distribution.

[0141] The above-mentioned disparity array may be generated randomly.

[0142] In addition, the above preset number can be any power of 2, such as 128, 64, 32, 256, etc.

[0143] S203: Select a disparity from each disparity array to obtain a first array including the selected disparity.

[0144] Specifically, a disparity can be randomly selected from each disparity array, or a disparity at a preset position in each disparity array can be selected. The number of the above disparity arrays is different, and the number of dimensions of the obtained first array is also different. If the number of the above disparity arrays is 128, it can be considered that the dimension of the above first array is 128. The number of dimensions of the above first array is the same as the number of dimensions of the above sample variance and sample mean.

[0145] It can be considered that a disparity selected from each disparity array can represent the distribution of disparities in the disparity array. If the disparity array is randomly generated and has randomness, the first array obtained is also random. And because the number of the disparity arrays is large, the randomly generated first array can represent the distribution of pixel value differences between pixels in the view of the 3D video frame under common random conditions.

[0146] S204: According to the sample mean and the sample variance, adjust the disparity in the first array so that the distribution of the disparity in the first array approaches the distribution of the pixel value difference between the pixels in the two sample views, to obtain a second array.

[0147] Specifically, each disparity in the first array may be multiplied by a sample mean and then added with a sample variance, so as to adjust the disparity in the first array to obtain the second array.

[0148] If the first array is randomly generated, since the sample mean and the sample variance can represent the distribution of the pixel value difference between the pixels in the two sample views, the first array can represent the random distribution of the disparity between the views of the 3D video frame. Therefore, according to the sample mean and the sample variance, the second array obtained by adjusting the disparity of the first array can represent the distribution of the pixel value difference between the pixels in the sample views and reflect the random distribution of the disparity.

[0149] S205: Calculate the disparity mean and disparity variance of the second array.

[0150] Specifically, the second array may be fully connected to calculate the disparity mean and disparity variance. The second array may be input into a fully connected layer in a neural network, and the output result may be used as the disparity mean and disparity variance.

[0151] S206: setting the preset parameters in the model parameters of the preset neural network model to the above-mentioned disparity mean and disparity variance.

[0152] Specifically, since the distribution of pixel value differences between pixels in two sample views is different for different 3D video frames, the determined sample means and sample variances are different. For different 3D video frames, the calculated disparity means and disparity variances are different.

[0153] Before training the neural network model according to different sample views, it is necessary to set the preset parameters of the neural network model to the disparity mean and disparity variance corresponding to the sample view, so that the preset parameters match the sample view used in model training.

[0154] S207: Input a sample view into the neural network model to obtain an output image having parallax with the sample view.

[0155] In order to make 3D videos present different 3D stereoscopic effects, the disparity between views in different 3D video frames is often different, that is, the disparity distribution between views in different 3D video frames is often different. In other words, the similarity of the disparity between views in different 3D video frames is low, and the 3D video frames have 3D disparity ambiguity. Training the neural network model with views of different 3D video frames as sample views will have different effects on the above neural network model, making it difficult for the above neural network model to be trained to converge.

[0156] Therefore, when the neural network model processes the sample view, the pixel values ​​of the pixels in the fourth image excluding the third image in the sample downsampled image of the obtained sample view can be adjusted according to the disparity mean and disparity variance, so that the pixel values ​​of the pixels in the adjusted fourth image are affected by the distribution of pixel value differences between the pixels of the sample view, and approach the result obtained by virtual downsampling of another sample view. Thereby, the obtained output image approaches another sample view, making the neural network model easier to be trained to convergence. And if the first array is randomly generated, the disparity mean and disparity variance are random, so the obtained output image does not directly approach another sample view, ensuring that the neural network model can be effectively trained through the sample view.

[0157] The third image is the image with the smallest size among the obtained sample down-sampled images.

[0158] In addition, in one embodiment of the present invention, a sample view of a preset type may be input into the neural network model to obtain an output image having parallax with the sample view.

[0159] Among them, the above preset type is a left view or a right view.

[0160] If the preset type is a left view, the left view in the sample view is always input into the neural network model during model training, and the output image is used as the right view. Since the sample views input into the neural network model are all left views, the sample views input into the neural network model are similar, so the neural network model is easier to be trained to convergence.

[0161] Similarly, if the above-mentioned preset type is the right view, the above-mentioned neural network model is also easier to be trained to convergence, which will not be repeated here.

[0162] In addition, since the above sample view is a left view or a right view, that is, the sample view is a view corresponding to a single eye of the viewing user, the process of processing the sample view by the above neural network model can be called monocular encoding.

[0163] S208: Calculate the loss of the neural network model based on the output image and another sample view.

[0164] In an embodiment of the present invention, the output image and another sample view can be used as input data of the loss calculation algorithm, and the above loss can be calculated by the above loss calculation algorithm. Specifically, the above loss calculation algorithm can be an L1 algorithm, an SSIM (Structural Similarity) algorithm, a second-order gradient algorithm of a disparity depth map, or other algorithms. The above algorithms are existing algorithms and will not be described in detail here.

[0165] S209: Adjusting the model parameters of the neural network model according to the loss.

[0166] After adjusting the model parameters, if the preset training termination condition is met, the trained neural network model is determined as the image acquisition model.

[0167] Specifically, the above-mentioned training termination condition may be that the model parameters are adjusted for a preset number of times, or that the calculated loss of the above-mentioned neural network model is less than a preset loss or other training termination conditions.

[0168] Otherwise, return to execute the above steps S201-S208 and continue model training.

[0169] As can be seen from the above, the solution provided by the embodiment of the present invention uses the view in the 3D video frame as the sample view, inputs a sample view into the neural network model, calculates the loss based on the obtained output image and another sample view, adjusts the model parameters, and makes the output image output by the neural network model gradually approach another sample view with parallax with the input sample view through training. Thus, the trained image acquisition model can obtain an image with parallax with the 2D video frame based on the input 2D video frame.

[0170] Furthermore, the solution provided by the embodiment of the present invention adjusts the pixel values ​​of the pixels in each fourth image according to the calculated preset parameters during model training, so that the obtained output image approaches another sample view. This can reduce the calculated loss, and the above neural network model is easier to be trained to convergence.

[0171] See also Figure 3 , an embodiment of the present invention provides a flow chart of a second image acquisition model training method, wherein L is a left view in a sample view, R is a right view in a sample view, R' is an output image, and VAE is a model for outputting a disparity mean and a disparity variance.

[0172] Among them, the disparity mean and the disparity variance are calculated based on the sample view and the first array, and the neural network model is configured with the above disparity mean and disparity variance. The sample view of the input neural network model shown in the figure is the left view. A fused image is generated, and an output image is obtained based on the fused image. Since the sample view of the input neural network model is the left view, the output image R' can be considered to be the generated right view. The model loss is calculated based on the output image and the right view in the sample view, and the model parameters of the neural network model can be adjusted according to the loss.

[0173] As can be seen from the above, the VAE model can calculate the disparity mean and disparity variance based on the difference between the pixel values ​​of the sample view pixels. In the neural network model, the pixel values ​​of the pixels in the sample view are adjusted based on the above disparity mean and disparity variance to obtain the above fused image. The fused image is then processed to obtain an output image with disparity with the sample view. The loss of the neural network model can be calculated based on the output image and another sample image, and further the model parameters of the neural network model can be adjusted based on the above loss. Therefore, the solution provided by the embodiment of the present invention can realize the training process of the neural network model shown above.

[0174] Corresponding to the above-mentioned method for generating disparity images of 2D video frames, see Figure 4 , is a schematic diagram of the structure of a disparity image generating device for a 2D video frame provided by an embodiment of the present invention, the device comprising:

[0175] A downsampling module 401 is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame, and obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as a processed image processed by the next downsampling process;

[0176] The pixel value adjustment module 402 is used to adjust the pixel value of each pixel in each second image except the first image in the downsampled image obtained by each downsampling process based on a preset parameter with a fixed value, so that the pixel value mean and variance of the pixels in each second image are changed, wherein the first image is: the image with the smallest size in the obtained downsampled images;

[0177] A first upsampling module 403, configured to perform upsampling processing on the first image to obtain an upsampled image;

[0178] A convolution module 404 is used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image;

[0179] A pixel value fusion module 405 is used to fuse the upsampled image, the convolution image and the second image with the same size as the upsampled image after the pixel value adjustment according to the pixel position to obtain a fused image;

[0180] A second upsampling module 406 is configured to upsample the fused image to obtain a new upsampled image if the size of the fused image obtained by the pixel value fusion module 405 is different from the size of the 2D video frame, and return to trigger the execution of the convolution module 404 and the pixel value fusion module 405;

[0181] The image determination module 407 is configured to determine the fused image as an image having parallax with the 2D video frame if the size of the fused image obtained by the pixel value fusion module 405 is the same as the size of the 2D video frame.

[0182] As can be seen from the above, the scheme provided by the embodiment of the present invention can obtain an image with parallax with the 2D video frame. Moreover, in the process of obtaining an image with parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the scheme provided by the embodiment of the present invention can reduce the difference between the parallax between the 2D video frame and the generated image. Moreover, by adjusting the above preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame is adjustable.

[0183] In one embodiment of the present invention, the pixel value adjustment module 402 is specifically used to:

[0184] The pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process is multiplied by a preset variance, and a preset average value is added to obtain an adjusted pixel value of each pixel.

[0185] In one embodiment of the present invention, an image having parallax with the 2D video frame is obtained by the following image obtaining module:

[0186] An image acquisition module is used to input a 2D video frame into a pre-trained image acquisition model to obtain an image with parallax with the 2D video frame, wherein the model parameters of the image acquisition model include the preset parameters, and the image acquisition model is used to: implement the downsampling module to the image determination module.

[0187] It can be seen from the above that the image acquisition model can process the 2D video frame in the manner shown in the above method for generating a disparity image of a 2D video frame to obtain an image having a disparity with the 2D video frame.

[0188] In one embodiment of the present invention, the image acquisition model is obtained by training a preset neural network model through the following model training module. Figure 5 , an embodiment of the present invention provides a structural diagram of a model training module, the above-mentioned model training module includes:

[0189] The mean variance determination submodule 501 is used to determine the mean and variance of the pixel value difference between pixels in two sample views in the 3D sample video frame, and use the determined mean as the sample mean and the determined variance as the sample variance;

[0190] An array generation submodule 502 is used to obtain a preset number of disparity arrays, wherein the elements contained in the disparity arrays represent disparities;

[0191] A disparity selection submodule 503, configured to select a disparity from each disparity array to obtain a first array including the selected disparity;

[0192] a disparity adjustment submodule 504, configured to adjust the disparity in the first array according to the sample mean and the sample variance so that the distribution of the disparity in the first array approaches the distribution of the pixel value difference between the pixels in the two sample views, thereby obtaining a second array;

[0193] A mean-variance calculation submodule 505 is used to calculate the disparity mean and disparity variance of the second array;

[0194] A parameter setting submodule 506 is used to set the preset parameters in the model parameters of the preset neural network model to the disparity mean and the disparity variance;

[0195] An image acquisition submodule 507, configured to input a sample view into the neural network model to obtain an output image having a parallax with the sample view;

[0196] A loss calculation submodule 508, configured to calculate the loss of the neural network model according to the output image and another sample view;

[0197] The parameter adjustment submodule 509 is used to adjust the model parameters of the neural network model according to the loss.

[0198] As can be seen from the above, the solution provided by the embodiment of the present invention uses the view in the 3D video frame as the sample view, inputs a sample view into the neural network model, calculates the loss based on the obtained output image and another sample view, adjusts the model parameters, and makes the output image output by the neural network model gradually approach another sample view with parallax with the input sample view through training. Thus, the trained image acquisition model can obtain an image with parallax with the 2D video frame based on the input 2D video frame.

[0199] Furthermore, the solution provided by the embodiment of the present invention adjusts the pixel values ​​of the pixels in each fourth image according to the calculated preset parameters during model training, so that the obtained output image approaches another sample view. This can reduce the calculated loss, and the above neural network model is easier to be trained to convergence.

[0200] In one embodiment of the present invention, the mean variance determination submodule 501 is specifically used for:

[0201] Extracting a correlation feature representing a correlation relationship between the two sample views;

[0202] Performing convolution processing on the associated features;

[0203] The obtained convolution processing result is fully connected to obtain the mean and variance of the pixel value difference between the pixels in the two sample views.

[0204] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0205] Memory 603, used for storing computer programs;

[0206] The processor 601 is configured to implement any method step of the above-mentioned method for generating a parallax image of a 2D video frame when executing the program stored in the memory 603 .

[0207] When the electronic device provided by the embodiment of the present invention is applied to generate a parallax image of a 2D video frame, the scheme provided by the embodiment of the present invention can obtain an image with a parallax with the 2D video frame. Moreover, in the process of obtaining an image with a parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the difference between the parallax between the 2D video frame and the generated image can be reduced by the scheme provided by the embodiment of the present invention. Moreover, by adjusting the above-mentioned preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame can be adjusted.

[0208] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0209] The communication interface is used for communication between the above terminal and other devices.

[0210] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0211] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0212] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method for generating a disparity image of a 2D video frame described in any of the above embodiments is implemented.

[0213] When the computer program stored in the computer-readable storage medium provided by the present embodiment is applied to generate the parallax image of the 2D video frame, the scheme provided by the embodiment of the present invention can obtain an image with parallax with the 2D video frame. Moreover, in the process of obtaining an image with parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the difference between the parallax between the 2D video frame and the generated image can be reduced by the scheme provided by the embodiment of the present invention. Moreover, by adjusting the above-mentioned preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame can be adjusted.

[0214] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for generating a disparity image of a 2D video frame described in any one of the above embodiments.

[0215] When executing the computer program product provided in this embodiment to generate a parallax image of a 2D video frame, the scheme provided in the embodiment of the present invention can obtain an image with a parallax with the 2D video frame. Moreover, in the process of obtaining an image with a parallax with the 2D video frame, the second image is always adjusted using preset parameters with fixed values. Affected by the preset parameters with fixed values, the parallax between the generated image and the 2D video frame is relatively fixed, so the difference between the parallax between the 2D video frame and the generated image can be reduced by the scheme provided in the embodiment of the present invention. Moreover, by adjusting the above-mentioned preset parameters, the adjustment result of the second image can be changed, thereby changing the generated fused image, and further changing the parallax between the obtained image and the 2D video frame, so the parallax between the obtained image and the 2D video frame is adjustable.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0217] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0218] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0219] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a disparity image of a 2D video frame, characterized in that: The method comprises: Performing a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process; Based on preset parameters with fixed values, pixel values ​​of pixels in each second image other than the first image in each down-sampled image obtained by the down-sampling process are adjusted so that the mean value and variance of the pixel values ​​of the pixels in each second image are changed, wherein the first image is: an image with the smallest size in the obtained down-sampled images; Performing upsampling processing on the first image to obtain an upsampled image; Performing convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image; The upsampled image, the convolution image, and the second image with the same size as the upsampled image and adjusted pixel values ​​are fused according to pixel positions to obtain a fused image; If the size of the fused image is different from the size of the 2D video frame, upsampling the fused image to obtain a new upsampled image, returning to perform convolution processing on the new upsampled image to obtain a convolution image with the same size as the new upsampled image, and fusing the new upsampled image, the convolution image with the same size as the new upsampled image, and the second image with the same size as the new upsampled image after pixel value adjustment according to pixel positions to obtain a fused image; Otherwise, the fused image is determined as an image having parallax with the 2D video frame.

2. The method according to claim 1, characterized in that The step of adjusting the pixel value of each pixel in each second image except the first image in the downsampled image obtained by each downsampling process based on the preset parameter with a fixed value includes: The pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process is multiplied by a preset variance, and a preset average value is added to obtain an adjusted pixel value of each pixel.

3. The method according to claim 1, characterized in that The image having parallax with the 2D video frame is obtained in the following manner: The 2D video frame is input into a pre-trained image acquisition model to obtain an image that has parallax with the 2D video frame, wherein the model parameters of the image acquisition model include the preset parameters, and the image acquisition model is used to: implement the cascade downsampling processing of the 2D video frame including a preset number of downsampling processings, obtain the downsampled image obtained by each downsampling processing, and determine the fused image as an image that has parallax with the 2D video frame.

4. The method according to claim 3, characterized in that The image acquisition model is obtained by training the preset neural network model in the following manner: Determine a mean and a variance of pixel value differences between pixels in two sample views in a 3D sample video frame, use the determined mean as a sample mean, and use the determined variance as a sample variance; Obtaining a preset number of disparity arrays, wherein the elements contained in the disparity arrays represent disparities; Selecting a disparity from each disparity array to obtain a first array including the selected disparity; According to the sample mean and the sample variance, adjusting the disparity in the first array so that the distribution of the disparity in the first array approaches the distribution of the pixel value difference between the pixels in the two sample views, to obtain a second array; Calculate the disparity mean and disparity variance of the second array; Setting the preset parameters in the model parameters of the preset neural network model to the disparity mean and the disparity variance; Inputting a sample view into the neural network model to obtain an output image having a parallax with the sample view; Calculating the loss of the neural network model according to the output image and another sample view; Model parameters of the neural network model are adjusted according to the loss.

5. The method according to claim 4, characterized in that The determining the mean and variance of the pixel value differences between pixels in two sample views in the 3D sample video frame includes: Extracting a correlation feature representing a correlation relationship between the two sample views; Performing convolution processing on the associated features; The obtained convolution processing result is fully connected to obtain the mean and variance of the pixel value difference between the pixels in the two sample views.

6. A parallax image generation device for a 2D video frame, characterized in that: The device comprises: A downsampling module is used to perform a cascade downsampling process including a preset number of downsampling processes on the 2D video frame to obtain a downsampled image obtained by each downsampling process, wherein the result of each downsampling process in the cascade downsampling process is used as the image processed by the next downsampling process; A pixel value adjustment module, for adjusting the pixel values ​​of pixels in each second image other than the first image in each down-sampled image obtained by down-sampling processing based on preset parameters with fixed values, so that the mean value and variance of the pixel values ​​of the pixels in each second image are changed, wherein the first image is: an image with the smallest size in the obtained down-sampled images; A first upsampling module, configured to perform upsampling processing on the first image to obtain an upsampled image; A convolution module, used to perform convolution processing on the upsampled image to obtain a convolution image with the same size as the upsampled image; A pixel value fusion module, used for fusing the upsampled image, the convolution image and the second image with the same size as the upsampled image after the pixel value adjustment according to the pixel position to obtain a fused image; a second upsampling module, configured to perform upsampling processing on the fused image to obtain a new upsampling image if the size of the fused image obtained by the pixel value fusion module is different from the size of the 2D video frame, and return triggering execution of the convolution module and the pixel value fusion module for the new upsampling image, perform convolution processing on the new upsampling image to obtain a convolution image with the same size as the new upsampling image, and perform pixel value fusion on the new upsampling image, the convolution image with the same size as the new upsampling image, and the second image with the same size as the new upsampling image after pixel value adjustment, according to pixel positions, to obtain a fused image; The image determination module is configured to determine the fused image as an image having parallax with the 2D video frame if the size of the fused image obtained by the pixel value fusion module is the same as the size of the 2D video frame.

7. The device according to claim 6, characterized in that The pixel value adjustment module is specifically used for: The pixel value of each pixel in each second image except the first image in the down-sampled image obtained by each down-sampling process is multiplied by a preset variance, and a preset average value is added to obtain an adjusted pixel value of each pixel.

8. The device according to claim 6, characterized in that An image having parallax with the 2D video frame is obtained by the following image acquisition module: An image acquisition module is used to input a 2D video frame into a pre-trained image acquisition model to obtain an image with parallax with the 2D video frame, wherein the model parameters of the image acquisition model include the preset parameters, and the image acquisition model is used to: implement the downsampling module to the image determination module.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images