3D display image generation method and related equipment

Abstract

The invention discloses a 3D display image generation method and related equipment, and relates to the technical field of naked eye 3D display, and the method comprises the steps: obtaining a background layer image, a middle layer image and a foreground layer image to be superposed, processing the middle layer image based on an image processing mode of a 2D + Alpha channel, obtaining a first transparent color image corresponding to a middle layer, and obtaining a second transparent color image corresponding to the foreground layer; based on an image processing mode of 2D + Alpha channel + depth map, processing the foreground layer image to obtain a first image, executing an image matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image, and performing image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image. According to the invention, each layer of image is processed based on an image processing mode of 2D + Alpha channel + depth map, and the processed images are fused, so that the cost of generating the 3D display image is reduced.

Description

Technical Field

[0001] The present invention relates to the field of naked-eye 3D display technology, and in particular to a 3D display image generation method and related equipment. Background Art

[0002] In the information and digital age, with the development of society, 2D display can no longer meet human needs. 3D display has become a new research target for researchers and a new development trend in the display field.

[0003] In the related art, the multi-viewpoint grating naked-eye 3D display allows multiple viewers to simultaneously view stereoscopic images within a larger viewing angle, and can experience a shocking visual experience without the help of any visual aids, and therefore has attracted much attention. However, in the actual 3D display image generation process, it is necessary to collect stereoscopic images from multiple viewpoints in real time. The usual practice is to use N cameras arranged in parallel to simultaneously shoot the same scene, so as to obtain N images at different positions (viewpoints), which has a high production cost. In addition, in recent years, naked-eye 3D LED large screens based on visual 3D have been widely used, which usually adopts an arc corner design. This design avoids the folding of the picture caused by the corner, so that the picture can be seamlessly connected and smoother, ensuring the creation of a sense of space and atmosphere, and making the picture have a better sense of immersion. The cost of customizing this single-viewpoint 3D content is also very high. Therefore, how to generate 3D display images at a low cost and meet the requirements of single-viewpoint 3D and multi-viewpoint 3D display at the same time is a problem that needs to be solved urgently.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are related technologies. Summary of the invention

[0005] The main purpose of the present application is to provide a 3D display image generation method and related equipment, aiming to solve the technical problem of how to generate 3D display images at low cost and meet the requirements of single-viewpoint 3D and multi-viewpoint 3D display at the same time.

[0006] To achieve the above objectives, the present application proposes a method for generating a 3D display image, the method comprising:

[0007] Obtaining a background layer image, an intermediate layer image, and a foreground layer image to be superimposed;

[0008] Based on the image processing method of 2D+Alpha channel, the intermediate layer image is processed to obtain a first transparent color image corresponding to the intermediate layer;

[0009] Based on the image processing method of 2D+Alpha channel+depth map, the foreground layer image is processed to obtain a first image, and a matting operation and a virtual viewpoint generation operation are performed on the first image to obtain a virtual viewpoint color image;

[0010] An image fusion operation is performed on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0011] In one embodiment, before the step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image, the step includes:

[0012] Get the preset 3D display image resolution;

[0013] Based on the resolution, the background layer image is cropped to obtain a target background layer image.

[0014] In one embodiment, the step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image further includes:

[0015] An image fusion operation is performed on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0016] In one embodiment, the step of performing an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image further includes:

[0017] Performing 2D superposition on the target background layer image and the first transparent color image to obtain a first superimposed image;

[0018] Get the grating parameters to determine the number of RGB sub-pixels covered by a grating period in the horizontal direction. 1 ;

[0019] Determine the display value V of each sub-pixel based on l 1 sub-pixels, the display value V of each sub-pixel and the first superimposed image to obtain a 3D display image.

[0020] In one embodiment, the step of processing the intermediate layer image based on the image processing method of 2D+Alpha channel to obtain the first transparent color image corresponding to the intermediate layer further includes:

[0021] Based on the image processing method of 2D+Alpha channel, the intermediate layer image is subjected to dual-path processing to obtain a left-path image and a right-path image;

[0022] The foreground corresponding to the left image is set as the first color image, and the foreground corresponding to the right image is set as the first Alpha channel image;

[0023] A first transparent color image is obtained based on the first color image and the first Alpha channel image.

[0024] In one embodiment, the step of processing the foreground layer image based on the image processing method of 2D+Alpha channel+depth map to obtain a first image, performing a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image further includes:

[0025] Based on the image processing method of 2D+Alpha channel, the foreground layer image is processed in three ways to obtain an upper image, a middle image and a lower image;

[0026] Setting the foreground corresponding to the upper image as a second color image, setting the foreground corresponding to the middle image as a second alpha channel image, and setting the foreground corresponding to the lower image as a depth map image;

[0027] Obtaining a first image based on the second color image, the second alpha channel image, and the depth map image;

[0028] Obtaining a second transparent color image based on the second color image and the second Alpha channel image;

[0029] A virtual viewpoint color image is generated based on the second transparent color image and the depth map image.

[0030] In one embodiment, the step of generating a virtual viewpoint color image based on the second transparent color image and the depth map image further includes:

[0031] Dividing the second transparent color image into a plurality of groups of image blocks of size n×n, wherein the value of n is set in advance;

[0032] Obtaining a preset feature of the second transparent color image, and calculating the cosine similarity of the image blocks based on the preset feature to obtain an attention score of each image block;

[0033] Based on the attention score of each image block, reconstruct the preset feature to obtain the target feature;

[0034] A virtual viewpoint color image is obtained based on the target feature, the second transparent color image and the depth map image.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a 3D display image generation device, the 3D display image generation device comprising:

[0036] An acquisition module, the acquisition module is used to acquire the background layer image, the middle layer image and the foreground layer image to be superimposed;

[0037] A first image processing module, the first image processing module is used to process the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a first transparent color image corresponding to the intermediate layer;

[0038] A second image processing module, the second image processing module is used to process the foreground layer image based on the image processing method of 2D+Alpha channel+depth map to obtain a first image, perform a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image;

[0039] A fusion module is used to perform an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0040] In one embodiment, the 3D display image generating device further includes a cropping module, and the cropping module includes:

[0041] A first acquisition unit, used to acquire a preset 3D display image resolution;

[0042] A cropping unit is used to crop the background layer image based on the resolution to obtain a target background layer image.

[0043] In one embodiment, the fusion module includes:

[0044] A fusion unit is used to perform an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0045] In one embodiment, the fusion module further includes:

[0046] A superposition unit, used for performing 2D superposition on the target background layer image and the first transparent color image to obtain a first superimposed image;

[0047] A determination unit is used to obtain grating parameters to determine the number of RGB sub-pixels covered by a grating period in the horizontal direction. 1 ;

[0048] The first obtaining unit is used to determine the display value V of each sub-pixel based on l 1sub-pixels, the display value V of each sub-pixel and the first superimposed image to obtain a 3D display image.

[0049] In one embodiment, the first image processing module includes:

[0050] A first processing unit is used to perform dual-channel processing on the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a left-channel image and a right-channel image;

[0051] A first setting unit, used to set the foreground corresponding to the left image as a first color image, and set the foreground corresponding to the right image as a first Alpha channel image;

[0052] The second obtaining unit is used to obtain a first transparent color image based on the first color image and the first Alpha channel image.

[0053] In one embodiment, the second image processing module includes:

[0054] The second processing unit is used to perform three-way processing on the foreground layer image based on the image processing method of 2D+Alpha channel to obtain an upper image, a middle image and a lower image;

[0055] A second setting unit, configured to set the foreground corresponding to the upper image as a second color image, set the foreground corresponding to the middle image as a second alpha channel image, and set the foreground corresponding to the lower image as a depth map image;

[0056] A third obtaining unit, configured to obtain a first image based on the second color image, the second Alpha channel image, and the depth map image;

[0057] a fourth obtaining unit, configured to obtain a second transparent color image based on the second color image and the second Alpha channel image;

[0058] A generating unit is used to generate a virtual viewpoint color image based on the second transparent color image and the depth map image.

[0059] In one embodiment, the second image processing module further includes:

[0060] A block division unit, used for dividing the second transparent color image into a plurality of groups of image blocks of size n×n, wherein the value of n is set in advance;

[0061] A second acquisition unit, configured to acquire a preset feature of the second transparent color image, and calculate the cosine similarity of the image blocks based on the preset feature to obtain an attention score of each image block;

[0062] A reconstruction unit, used to reconstruct the preset features based on the attention score of each image block to obtain the target features;

[0063] The fifth obtaining unit is used to obtain a virtual viewpoint color image based on the target feature, the second transparent color image and the depth map image.

[0064] In addition, to achieve the above-mentioned purpose, the present application also proposes a 3D display image generating device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the 3D display image generating method as described above.

[0065] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by a processor, the steps of the 3D display image generation method described above are implemented.

[0066] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the 3D display image generating method as described above are implemented.

[0067] One or more technical solutions proposed in this application have at least the following technical effects:

[0068] The present application proposes a 3D display image generation method and related equipment, which relate to the field of naked-eye 3D display technology. Compared with the related technology, it is necessary to collect stereoscopic images from multiple viewpoints in real time. The usual practice is to use N cameras arranged in parallel to shoot the same scene at the same time, so as to obtain N images at different positions (viewpoints), which has a high production cost. In the present application, first, a background layer image, an intermediate layer image and a foreground layer image to be superimposed are obtained, and then, based on the image processing method of 2D+Alpha channel, the intermediate layer image is processed to obtain a first transparent color image corresponding to the intermediate layer, and further, based on the image processing method of 2D+Alpha channel+depth map, the foreground layer image is processed to obtain a first image, and a matting operation and a virtual viewpoint generation operation are performed on the first image to obtain a virtual viewpoint color image, and finally, an image fusion operation is performed on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the 3D display image generation method of the present application;

[0072] Figure 2 A schematic diagram of a naked-eye 3D display of multi-layer image fusion provided in Embodiment 1 of the 3D display image generation method of the present application;

[0073] Figure 3 A schematic diagram of an intermediate layer "2D+Alpha channel" video image provided in Example 1 of the 3D display image generation method of the present application;

[0074] Figure 4 A schematic diagram of a foreground layer "2D+Alpha channel+depth map" video image provided in Example 1 of the 3D display image generation method of the present application;

[0075] Figure 5 A schematic diagram of a flow chart provided for the second embodiment of the 3D display image generation method of the present application;

[0076] Figure 6 This is a schematic diagram of the module structure of the 3D display image generation device according to an embodiment of the present application;

[0077] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the 3D display image generation method in the embodiment of the present application.

[0078] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0079] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0080] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0081] The main solutions of the embodiments of this application are:

[0082] In this embodiment, for the convenience of description, the following description is made by taking a 3D display image generating device as the execution subject.

[0083] Due to the existing technology, in the actual 3D display image generation process, it is necessary to collect stereo images from multiple viewpoints in real time. The usual practice is to use N cameras arranged in parallel to shoot the same scene at the same time, so as to obtain N images from different positions (viewpoints), which is costly to produce. In addition, considering the widespread application of single-viewpoint 3D LED large screens based on visual 3D in recent years, the cost of content customization is also very high. Therefore, how to generate 3D display images at a low cost and meet the requirements of single-viewpoint 3D and multi-viewpoint 3D display at the same time is an urgent problem to be solved.

[0084] The present application provides a solution, which enables: obtaining a background layer image, an intermediate layer image, and a foreground layer image to be superimposed, processing the intermediate layer image based on an image processing method of 2D+Alpha channel to obtain a first transparent color image corresponding to the intermediate layer, processing the foreground layer image based on an image processing method of 2D+Alpha channel+depth map to obtain a first image, performing a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image, performing an image fusion operation on the background layer image, the first transparent color image, and the virtual viewpoint color image to obtain a 3D display image. The present application processes each layer of images based on an image processing method of 2D+Alpha channel+depth map, and fuses the processed images, thereby reducing the cost of generating 3D display images.

[0085] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a 3D display image generation device, etc. The following takes a 3D display image generation device as an example to illustrate this embodiment and the following embodiments.

[0086] Based on this, the present application embodiment provides a 3D display image generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the 3D display image generation method of the present application.

[0087] In this embodiment, the 3D display image generation method includes steps S10 to S40:

[0088] Step S10, obtaining a background layer image, an intermediate layer image and a foreground layer image to be superimposed;

[0089] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a 3D display image generation device, etc. The following takes a 3D display image generation device as an example to illustrate this embodiment and the following embodiments.

[0090] In this embodiment, the specific application scenario may be:

[0091] After the 3D display image generating device receives the task of generating a 3D display image, the 3D display image generating device obtains the background layer image, the intermediate layer image and the foreground layer image to be superimposed, and then processes the background layer image, the intermediate layer image and the foreground layer image to be superimposed to generate a 3D display image.

[0092] refer to Figure 2 , Figure 2 The present invention describes a schematic diagram of a multi-layer image fusion 3D display, wherein 101 is the background layer 2D camera captured video / local 2D video image (background layer image), 102 is the middle layer "2D+Alpha channel" video image, time, weather and other information (middle layer image), 103 is the foreground layer "2D+Alpha channel+depth map" video image, wherein the background layer and the middle layer fused image are 2D display, and the foreground layer fused image is 3D display.

[0093] Step S20, processing the intermediate layer image based on the image processing method of 2D+Alpha channel to obtain a first transparent color image corresponding to the intermediate layer;

[0094] Specifically, the step of processing the intermediate layer image based on the image processing method of 2D+Alpha channel to obtain the first transparent color image corresponding to the intermediate layer also includes steps S21 to S23:

[0095] Step S21, performing dual-channel processing on the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a left-channel image and a right-channel image;

[0096] Step S22, setting the foreground corresponding to the left image as the first color image, and setting the foreground corresponding to the right image as the first Alpha channel image;

[0097] Step S23: obtaining a first transparent color image based on the first color image and the first Alpha channel image.

[0098] It should be noted that, in this embodiment, in order to ensure the transparency of the middle layer, the middle layer is encoded using the "2D+Alpha channel" method. Considering the compatible playback of the video, a two-way splicing method can be used to compress the "2D+Alpha channel" video content. Figure 3 The present invention describes a schematic diagram of a naked-eye 3D display of multi-layer image fusion. Among them, 301 is the background of the left image, which is a black screen material, and 302 is a color image of the foreground of the left image; the right image is the same as 301, and the background is also a black screen material, and 303 is the Alpha channel image of the foreground object to be matted. Then the YUV format obtained by decoding the "2D+Alpha channel" video is converted into RGB format. For the time, weather and other information of the intermediate layer, text information or BMP or PNG image format with Alpha channel can be directly obtained. Assuming that C is the color value of the pixel, F and B represent the foreground color value and background color value of this pixel in their respective channels, respectively, and α is the opacity value of the pixel in the channel, then C=αF+(1-α)B, where the value range of α is [0, 1]. When α=1, it means that the pixel belongs to the foreground, when α=0, it means that the pixel belongs to the background, and when 0<α<1, it means that the value is a mixture of the foreground and the background. Under the RGB color model, it can be described as:

[0099]

[0100] Step S30, based on the image processing method of 2D+Alpha channel+depth map, the foreground layer image is processed to obtain a first image, and a matting operation and a virtual viewpoint generation operation are performed on the first image to obtain a virtual viewpoint color image;

[0101] Specifically, the image processing method based on 2D+Alpha channel+depth map processes the foreground layer image to obtain a first image, performs a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image, and further includes steps S31 to S35:

[0102] Step S31, performing three-way processing on the foreground layer image based on the image processing method of 2D+Alpha channel to obtain an upper image, a middle image and a lower image;

[0103] Step S32, setting the foreground corresponding to the upper image as a second color image, setting the foreground corresponding to the middle image as a second alpha channel image, and setting the foreground corresponding to the lower image as a depth map image;

[0104] Step S33, obtaining a first image based on the second color image, the second Alpha channel image and the depth map image;

[0105] Step S34, obtaining a second transparent color image based on the second color image and the second Alpha channel image;

[0106] Step S35: generating a virtual viewpoint color image based on the second transparent color image and the depth map image.

[0107] In this embodiment, in order to ensure the 3D transparent display effect of the foreground layer, the foreground layer is encoded using the "2D+Alpha channel+depth map" method. Considering the compatible playback of the video, the "2D+Alpha channel+depth map" video content can be compressed using a three-way splicing method. Figure 4 The schematic diagram of the foreground layer "2D+Alpha channel+depth map" video image in the present invention is described, wherein 401 is the upper image background, which is a black screen material, 402 is the color image of the foreground object to be matted; the middle image background is also a black screen material, 403 is the Alpha channel image of the foreground object to be matted; the lower image background is also a black screen material, and 404 is the depth map image of the foreground object to be matted. Then the YUV format obtained by decoding the "2D+Alpha channel+depth map" video is converted into RGB format.

[0108] In this embodiment, a virtual viewpoint color image is generated by real-time matting of the foreground layer "2D+Alpha channel+depth map" video image and virtual viewpoint generation. To ensure real-time matting of the foreground layer object, the "2D+Alpha channel" image is first processed, and then the transparent color image obtained by the "2D+Alpha channel" processing and the depth map are used to generate a virtual viewpoint, thereby obtaining a virtual viewpoint color image.

[0109] Step S40: performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0110] Specifically, before the step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image, steps S41 to S42 are included:

[0111] Step S41, obtaining a preset 3D display image resolution;

[0112] Step S42: cropping the background layer image based on the resolution to obtain a target background layer image.

[0113] It is understandable that the background layer image is a video captured by a 2D camera. In order to reduce the delay in displaying the image, the MJPEG compression format is usually selected, and the decoded image format is RGB format. If the video captured by the 2D camera only supports the output of video compression formats such as H.264 / H.265 or is a local 2D video, the YUV format obtained by decoding the 2D video needs to be converted to RGB format. The corresponding conversion relationship is:

[0114] R=Y+1.402(V-128)

[0115] G=Y-0.344(U-128)-0.174(V-128)

[0116] B=Y+1.772(U-128)

[0117] It is understandable that since the original aspect ratio of the video captured by the common 2D camera / local 2D video is inconsistent with the display aspect ratio of the 3D display terminal, if the picture is not cropped, it will often be deformed or have black edges. In order to match the background layer 2D camera captured video / local 2D video image with the naked-eye 3D display terminals such as LCD and LED, it needs to be cropped.

[0118] Assume that the resolution of the 3D display terminal is M 1 ×N 1 , where M 1 Horizontal resolution, N 1 The vertical resolution is M. The 2D camera captures the video / local 2D video resolution is 2 ×N 2 , where M 2 Horizontal resolution, M 2 is the vertical resolution, when When the 2D camera captures the video / local 2D video, the cropping method is: horizontal resolution M 2 Keep it unchanged, and cut the vertical resolution up and down; when When the 2D camera captures the video / local 2D video, the cropping method is: vertical resolution N 2 Keep it unchanged and cut the horizontal resolution in the left and right directions

[0119] Specifically, the step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image further includes step S43:

[0120] Step S43, performing an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0121] Specifically, the step of performing an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image further includes steps A01 to A03:

[0122] Step A01, performing 2D superposition on the target background layer image and the first transparent color image to obtain a first superimposed image;

[0123] Step A02, obtaining grating parameters to determine the number of RGB sub-pixels covered by a grating period in the horizontal direction. 1 ;

[0124] Step A03, determine the display value V of each sub-pixel based on l 1 sub-pixels, the display value V of each sub-pixel and the first superimposed image to obtain a 3D display image.

[0125] It should be noted that in this embodiment, the foreground layer "2D+Alpha channel+depth map" video image is matted in real time and virtual viewpoints are generated. When the transparent color image obtained by processing the "2D+Alpha channel" and the depth map are used to generate virtual viewpoints, although a virtual viewpoint color image can be obtained, the traditional image filling method often causes inconsistency between the filling of the hole area and the time domain information when processing the more complex texture structure of the hole area, resulting in poor quality of the virtual viewpoint image. In order to further improve the quality of image filling in the hole area, multi-scale feature aggregation is used to perform semantic encoding at multiple scales to balance accuracy and complexity in view of the correlation of local features of the image filled in the airspace.

[0126] In this embodiment, first, the background layer and the middle layer video images are superimposed in 2D in the RGB space, and then the virtual viewpoint transparent video layer generated in the foreground is synthesized in naked eye 3D. According to the grating parameters, i.e., the tilt angle, the horizontal intercept of the grating element, and the number of viewpoints, the number of RGB sub-pixels covered by one grating period in the horizontal direction is first determined. 1 , l 1 satisfy:

[0127]

[0128] in, is the horizontal angle of the grating axis relative to the vertical axis of the LCD display, l is the horizontal physical intercept of a grating period, and t is the horizontal physical intercept of the grating element.

[0129] Calculate which disparity map N the display value of each sub-pixel in the composite image comes from, where N satisfies the formula:

[0130]

[0131] Among them, % represents taking the remainder, x is the coordinate position of the RGB sub-pixel, and K off is the horizontal displacement amount between the upper left edge of the 2D display screen and the edge point of the grating unit, and n is the total number of viewpoints of the grating.

[0132] Finally, determine the display value V of each sub-pixel in the composite image. Preferably, in order to improve the display quality, V satisfies the formula:

[0133]

[0134] Among them, int(N) is the integer part of the value of N, fra(N) is the fractional part of the value of N, V[int(N)] is the display value of the sub-pixel at the position (x, y) of the int(N)-th parallax image, and V[int(N)+1] is the display value of the sub-pixel at the position (x, y) of the (int(N)+1)-th parallax image. When 1≤N<n, the display value of the sub-pixel is interpolated from the display values of the sub-pixels at the same coordinate of the int(N)-th and (int(N)+1)-th parallax images according to the ratio 1-fra(N), fra(N). When n≤N<(n+0.85), the display value of the sub-pixel is determined by the sub-pixel at the position (x, y) of the n-th parallax image. When (n+0.85)≤N<(n+1), the display value of the sub-pixel is determined by the sub-pixel at the position (x, y) of the 1st parallax image.

[0135] One or more technical solutions proposed in this application have at least the following technical effects:

[0136] This application proposes a 3D display image generation method and related devices, which relates to the field of naked-eye 3D display technology. Compared with the related technology, where it is necessary to collect stereo images of multiple viewpoints in real time, and the usual method is to use N cameras arranged in parallel to simultaneously shoot the same scene to obtain N images at different positions (viewpoints), with a high production cost. In this application, first, obtain the background layer image, the middle layer image, and the foreground layer image to be superimposed. Then, based on the image processing method of 2D+Alpha channel, process the middle layer image to obtain the first transparent color image corresponding to the middle layer. Further, based on the image processing method of 2D+Alpha channel+depth map, process the foreground layer image to obtain the first image, perform a keying operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image. Finally, perform an image fusion operation on the background layer image, the first transparent color image, and the virtual viewpoint color image to obtain a 3D display image.

[0137] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 The step of generating a virtual viewpoint color image based on the second transparent color image and the depth map image further includes steps B01 to B04:

[0138] Step B01, dividing the second transparent color image into a plurality of groups of image blocks of size n×n, wherein the value of n is set in advance;

[0139] Step B02, obtaining preset features of the second transparent color image, and calculating the cosine similarity of the image blocks based on the preset features to obtain an attention score of each image block;

[0140] Step B03, reconstructing the preset features based on the attention score of each image block to obtain the target features;

[0141] Step B04, obtaining a virtual viewpoint color image based on the target feature, the second transparent color image and the depth map image.

[0142] For example, in this embodiment, the cosine similarity is calculated according to the 6x6 size image block Assume f i is the i-th image block of the given feature F, and the attention score of each image block is obtained as Then reconstruct the features based on the attention map:

[0143]

[0144] Among them, f i is the reconstruction feature F rec The i-th image block.

[0145] When reconstructing features, holes with different expansion rates are used to obtain multi-scale features:

[0146]

[0147] Among them, Conv k (F rec ) represents the dilated convolution with different expansion rates, where k can be 1, 2, 4, or 8. Aggregate multiple features to obtain aggregated multi-scale features:

[0148]

[0149] In this application, skip connections are used to prevent input information loss. The input loss cost of a given feature F is:

[0150] F concact =F c -F

[0151] In this embodiment, in order to ensure the continuity of the image filled in the hole area, the loss cost of feature F and the temporal and spatial characteristics of the hole image are combined to ensure the feature consistency of the hole boundary and interior through spatial balance.

[0152] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the 3D display image generation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0153] This application also provides a 3D display image generation device, please refer to Figure 6 , the 3D display image generating device comprises:

[0154] An acquisition module 10, which is used to acquire a background layer image, an intermediate layer image and a foreground layer image to be superimposed;

[0155] A first image processing module 20, the first image processing module is used to process the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a first transparent color image corresponding to the intermediate layer;

[0156] A second image processing module 30, the second image processing module is used to process the foreground layer image based on the image processing method of 2D+Alpha channel+depth map to obtain a first image, and perform a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image;

[0157] A fusion module 40 is used to perform an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0158] In one embodiment, the 3D display image generating device further includes a cropping module, and the cropping module includes:

[0159] A first acquisition unit, used to acquire a preset 3D display image resolution;

[0160] A cropping unit is used to crop the background layer image based on the resolution to obtain a target background layer image.

[0161] In one embodiment, the fusion module includes:

[0162] A fusion unit is used to perform an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0163] In one embodiment, the fusion module further includes:

[0164] A superposition unit, used for performing 2D superposition on the target background layer image and the first transparent color image to obtain a first superimposed image;

[0165] A determination unit is used to obtain grating parameters to determine the number of RGB sub-pixels covered by a grating period in the horizontal direction. 1 ;

[0166] The first obtaining unit is used to determine the display value V of each sub-pixel based on l 1 sub-pixels, the display value V of each sub-pixel and the first superimposed image to obtain a 3D display image.

[0167] In one embodiment, the first image processing module includes:

[0168] A first processing unit is used to perform dual-channel processing on the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a left-channel image and a right-channel image;

[0169] A first setting unit, used to set the foreground corresponding to the left image as a first color image, and set the foreground corresponding to the right image as a first Alpha channel image;

[0170] The second obtaining unit is used to obtain a first transparent color image based on the first color image and the first Alpha channel image.

[0171] In one embodiment, the second image processing module includes:

[0172] The second processing unit is used to perform three-way processing on the foreground layer image based on the image processing method of 2D+Alpha channel to obtain an upper image, a middle image and a lower image;

[0173] A second setting unit, configured to set the foreground corresponding to the upper image as a second color image, set the foreground corresponding to the middle image as a second alpha channel image, and set the foreground corresponding to the lower image as a depth map image;

[0174] A third obtaining unit, configured to obtain a first image based on the second color image, the second Alpha channel image, and the depth map image;

[0175] a fourth obtaining unit, configured to obtain a second transparent color image based on the second color image and the second Alpha channel image;

[0176] A generating unit is used to generate a virtual viewpoint color image based on the second transparent color image and the depth map image.

[0177] In one embodiment, the second image processing module further includes:

[0178] A block division unit, used for dividing the second transparent color image into a plurality of groups of image blocks of size n×n, wherein the value of n is set in advance;

[0179] A second acquisition unit, configured to acquire a preset feature of the second transparent color image, and calculate the cosine similarity of the image blocks based on the preset feature to obtain an attention score of each image block;

[0180] A reconstruction unit, used to reconstruct the preset features based on the attention score of each image block to obtain the target features;

[0181] The fifth obtaining unit is used to obtain a virtual viewpoint color image based on the target feature, the second transparent color image and the depth map image.

[0182] The 3D display image generation device provided by the present application adopts the 3D display image generation method in the above embodiment, which can solve the technical problem of 3D display image generation. Compared with the prior art, the beneficial effects of the 3D display image generation device provided by the present application are the same as the beneficial effects of the 3D display image generation method provided by the above embodiment, and other technical features in the 3D display image generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0183] The present application provides a 3D display image generating device, the 3D display image generating device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the 3D display image generating method in the above-mentioned embodiment 1.

[0184] Reference below Figure 7 , which shows a schematic diagram of the structure of a 3D display image generating device suitable for implementing the embodiments of the present application. The 3D display image generating device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The 3D display image generating device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0185] like Figure 7 As shown, the 3D display image generation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the 3D display image generation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the 3D display image generation device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a 3D display image generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0186] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0187] The 3D display image generation device provided by the present application adopts the 3D display image generation method in the above embodiment, which can solve the technical problem of 3D display image generation. Compared with the prior art, the beneficial effects of the 3D display image generation device provided by the present application are the same as the beneficial effects of the 3D display image generation method provided by the above embodiment, and other technical features in the 3D display image generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0188] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0190] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the 3D display image generation method in the above-mentioned embodiment.

[0191] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0192] The computer-readable storage medium may be included in the 3D display image generating device; or may exist independently without being assembled into the 3D display image generating device.

[0193] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the 3D display image generating device, the 3D display image generating device:

[0194] Obtaining a background layer image, an intermediate layer image, and a foreground layer image to be superimposed;

[0195] Based on the image processing method of 2D+Alpha channel, the intermediate layer image is processed to obtain a first transparent color image corresponding to the intermediate layer;

[0196] Based on the image processing method of 2D+Alpha channel+depth map, the foreground layer image is processed to obtain a first image, and a matting operation and a virtual viewpoint generation operation are performed on the first image to obtain a virtual viewpoint color image;

[0197] An image fusion operation is performed on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

[0198] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0201] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned 3D display image generation method, and can solve the technical problem of 3D display image generation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the 3D display image generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0202] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned 3D display image generation method when executed by a processor.

[0203] The computer program product provided by this application can solve the technical problem of 3D display image generation. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the 3D display image generation method provided by the above embodiment, which will not be repeated here.

[0204] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a 3D display image, characterized in that: The 3D display image generation method comprises: Obtaining a background layer image, an intermediate layer image, and a foreground layer image to be superimposed; Based on the image processing method of 2D+Alpha channel, the intermediate layer image is processed to obtain a first transparent color image corresponding to the intermediate layer; Based on the image processing method of 2D+Alpha channel+depth map, the foreground layer image is processed to obtain a first image, and a matting operation and a virtual viewpoint generation operation are performed on the first image to obtain a virtual viewpoint color image; An image fusion operation is performed on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

2. The 3D display image generation method according to claim 1, characterized in that: Before the step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image, the method includes: Get the preset 3D display image resolution; Based on the resolution, the background layer image is cropped to obtain a target background layer image; The step of performing an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image also includes: An image fusion operation is performed on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

3. The 3D display image generation method according to claim 2, characterized in that: The step of performing an image fusion operation on the target background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image also includes: Performing 2D superposition on the target background layer image and the first transparent color image to obtain a first superimposed image; Obtain grating parameters to determine the number l1 of RGB sub-pixels covered by one grating period in the horizontal direction; A display value V of each sub-pixel is determined, and a 3D display image is obtained based on l1 sub-pixels, the display value V of each sub-pixel and the first superimposed image.

4. The 3D display image generation method according to claim 1, characterized in that: The step of processing the intermediate layer image based on the 2D+Alpha channel image processing method to obtain the first transparent color image corresponding to the intermediate layer also includes: Based on the image processing method of 2D+Alpha channel, the intermediate layer image is subjected to dual-path processing to obtain a left-path image and a right-path image; The foreground corresponding to the left image is set as the first color image, and the foreground corresponding to the right image is set as the first Alpha channel image; A first transparent color image is obtained based on the first color image and the first Alpha channel image.

5. The 3D display image generation method according to claim 1, characterized in that: The step of processing the foreground layer image based on the image processing method of 2D+Alpha channel+depth map to obtain a first image, performing a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image also includes: Based on the image processing method of 2D+Alpha channel, the foreground layer image is processed in three ways to obtain an upper image, a middle image and a lower image; Setting the foreground corresponding to the upper image as a second color image, setting the foreground corresponding to the middle image as a second alpha channel image, and setting the foreground corresponding to the lower image as a depth map image; Obtaining a first image based on the second color image, the second alpha channel image, and the depth map image; Obtaining a second transparent color image based on the second color image and the second Alpha channel image; A virtual viewpoint color image is generated based on the second transparent color image and the depth map image.

6. The 3D display image generation method according to claim 5, characterized in that: The step of generating a virtual viewpoint color image based on the second transparent color image and the depth map image further includes: Dividing the second transparent color image into a plurality of groups of image blocks of size n×n, wherein the value of n is set in advance; Obtaining a preset feature of the second transparent color image, and calculating the cosine similarity of the image blocks based on the preset feature to obtain an attention score of each image block; Based on the attention score of each image block, reconstruct the preset feature to obtain the target feature; A virtual viewpoint color image is obtained based on the target feature, the second transparent color image and the depth map image.

7. A 3D display image generating device, characterized in that: The 3D display image generating device comprises: An acquisition module, the acquisition module is used to acquire the background layer image, the middle layer image and the foreground layer image to be superimposed; A first image processing module, the first image processing module is used to process the intermediate layer image based on a 2D+Alpha channel image processing method to obtain a first transparent color image corresponding to the intermediate layer; A second image processing module, the second image processing module is used to process the foreground layer image based on the image processing method of 2D+Alpha channel+depth map to obtain a first image, perform a matting operation and a virtual viewpoint generation operation on the first image to obtain a virtual viewpoint color image; A fusion module is used to perform an image fusion operation on the background layer image, the first transparent color image and the virtual viewpoint color image to obtain a 3D display image.

8. A 3D display image generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the 3D display image generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the 3D display image generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the 3D display image generation method according to any one of claims 1 to 6 are implemented.

Images