Light field display method, device, electronic device and storage medium

Abstract

The present application proposes a light field display method, device, electronic device and storage medium, the method comprising: obtaining a light field image matrix; obtaining a light field depth information map through light field depth estimation according to the light field image matrix; determining an alternative reconstruction origin depth according to the light field depth information map; and reconstructing the light field image according to the alternative reconstruction origin depth. The present application performs light field depth estimation on the light field image to obtain multiple alternative reconstruction origin depths. The preset reconstruction origin depths of the light field display device are modified to the alternative reconstruction origin depths respectively, and the light field image is reconstructed based on the modified light field reconstruction origin. The light field image with the largest peak signal-to-noise ratio is selected from the multiple reconstructed light field images for display. The position of the light field reconstruction origin is changed multiple times according to the depth characteristics of the light field itself to obtain multiple light field reconstruction results, and the reconstruction result with the largest peak signal-to-noise ratio is selected for display, thereby improving the light field reconstruction quality and the display effect of the multi-layer light field display.

Description

Technical Field

[0001] The present application belongs to the field of display technology, and specifically relates to a light field display method, device, electronic device and storage medium. Background Art

[0002] Light field refers to the amount of light passing through each point in each direction. Light field display technology can reconstruct the three-dimensional information of objects in space and provide close-to-natural stereoscopic images.

[0003] At present, multi-layer light field display is a commonly used light field display technology. Multi-layer light field display is based on the principle of four-dimensional light field for light field acquisition and reconstruction. The light field reconstruction origin is preset in the light field display device. According to the light field reconstruction origin and the light field image obtained by light field acquisition, multiple display layers are used to construct the spatial light field of the three-dimensional scene, and the direction and color information of the light in the scene are determined by the pixels on multiple display planes. The brightness information of the pixels on each plane is obtained by optimizing and solving iteratively according to the brightness and spatial information of the three-dimensional scene.

[0004] However, the above-mentioned multi-layer light field display technology reconstructs the light field based on a preset light field reconstruction origin. During the reconstruction process, the influence of the characteristics of the light field image to be reconstructed on the light field reconstruction origin is not considered, resulting in a low peak signal-to-noise ratio of the reconstructed light field image and low reconstruction quality. Summary of the invention

[0005] The present application proposes a light field display method, device, electronic device and storage medium, which estimate the light field depth of a light field image and obtain multiple alternative reconstruction origin depths. The preset reconstruction origin depths of the light field display device are modified to the alternative reconstruction origin depths, and the light field image is reconstructed based on the modified light field reconstruction origin. The light field image with the largest peak signal-to-noise ratio is selected from the reconstructed multiple light field images for display. The light field reconstruction quality is improved, and the display effect of the multi-layer light field display is improved.

[0006] The first embodiment of the present application provides a light field display method, including:

[0007] Get the light field image matrix;

[0008] Obtaining a light field depth information map by light field depth estimation according to the light field image matrix;

[0009] Determining an alternative reconstruction origin depth according to the light field depth information map;

[0010] The light field image is reconstructed according to the alternative reconstruction origin depth.

[0011] In some embodiments of the present application, obtaining a light field depth information map by light field depth estimation according to the light field picture matrix includes:

[0012] Splicing a plurality of light field images included in the light field image matrix into a spliced ​​image;

[0013] Acquire a refocused image corresponding to the mosaic image according to the coordinates of each pixel point in the mosaic image and the number of pixels in the angular spot;

[0014] Performing light field depth estimation on the refocused image using a preset light field depth estimation algorithm to obtain a local depth map;

[0015] The local depth map is subjected to graph cut optimization and weighted median filtering to obtain a light field depth information map corresponding to the light field picture matrix.

[0016] In some embodiments of the present application, determining an alternative reconstruction origin depth according to the light field depth information map includes:

[0017] Counting the number of pixels corresponding to each depth value in the light field depth information map respectively;

[0018] Selecting a preset number of depth values ​​corresponding to the largest number of pixels from each of the depth values; or selecting a depth value corresponding to a ratio of the number of pixels to the total number of pixels in the light field depth information map greater than a preset threshold from each of the depth values;

[0019] According to the preset maximum depth value, the preset light field origin depth and each selected depth value, the candidate reconstruction origin depth corresponding to each selected depth value is calculated respectively.

[0020] In some embodiments of the present application, before respectively calculating the candidate reconstruction origin depth corresponding to each selected depth value according to the preset maximum depth value, the preset light field origin depth and each selected depth value, the method further includes:

[0021] If there are depth values ​​with adjacent values ​​among the selected depth values, the depth values ​​with adjacent values ​​are merged according to a preset merging principle.

[0022] In some embodiments of the present application, reconstructing the light field image according to the alternative reconstruction origin depth includes:

[0023] Obtaining a central viewing angle image in the light field image matrix;

[0024] Reconstructing the central viewing angle image according to each candidate reconstruction origin depth and a preset front and rear depth of field range corresponding to the light field display device;

[0025] According to the number of display layers included in the light field display device, performing tensor decomposition on the reconstructed central perspective image to obtain tensor values ​​of the number of display layers;

[0026] According to the tensor values ​​of the number of display layers, display images corresponding to each display layer are generated respectively through each display layer in the light field display device, and each display image is superimposed in space to obtain a reconstructed light field picture.

[0027] In some embodiments of the present application, reconstructing the central perspective image according to each alternative reconstruction origin depth and a preset front and back depth range corresponding to the light field display device includes:

[0028] Modifying a preset light field origin depth of the light field display device to a first reconstruction origin depth, where the first reconstruction origin depth is any reconstruction origin depth among the alternative reconstruction origin depths;

[0029] Adjusting the central viewing angle image according to the first reconstruction origin depth and a preset front and rear depth of field range corresponding to the light field display device;

[0030] Detect the peak signal-to-noise ratio of the adjusted central perspective image corresponding to each alternative reconstruction origin depth respectively;

[0031] Selecting a central perspective picture with the largest peak signal-to-noise ratio from the adjusted multiple central perspective pictures;

[0032] The central perspective image with the largest peak signal-to-noise ratio is determined as the final reconstructed central perspective image.

[0033] In some embodiments of the present application, after reconstructing the central perspective image, the method further includes:

[0034] According to the central perspective image before reconstruction and the central perspective image after reconstruction, the reconstructed central perspective image is corrected.

[0035] An embodiment of a second aspect of the present application provides a light field display device, including:

[0036] Image acquisition module, used to acquire light field image matrix;

[0037] A depth estimation module, configured to obtain a light field depth information map by light field depth estimation according to the light field image matrix;

[0038] An origin depth determination module, used to determine an alternative reconstruction origin depth according to the light field depth information map;

[0039] The light field reconstruction module is used to reconstruct the light field image according to the alternative reconstruction origin depth.

[0040] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0041] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0042] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0043] In an embodiment of the present application, a light field depth estimation algorithm is used to process a light field image to obtain multiple alternative reconstruction origin depths. The preset reconstruction origin depths of the light field display device are respectively modified to the alternative reconstruction origin depths, and the light field image is reconstructed based on the modified light field reconstruction origin. From the multiple reconstructed light field images, the one with the largest peak signal-to-noise ratio is selected for display. In this way, the influence of the depth characteristics of the light field itself on the light field reconstruction origin is taken into account, and the position of the light field reconstruction origin is changed multiple times according to the depth characteristics of the light field itself, and multiple light field reconstruction results are obtained. The reconstruction result with the largest peak signal-to-noise ratio is selected for display, which greatly improves the quality of light field reconstruction and improves the display effect of multi-layer light field display.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 A flow chart of a light field display method provided by an embodiment of the present application is shown;

[0047] Figure 2 A schematic diagram of a process of performing depth estimation processing on a light field image provided by an embodiment of the present application is shown;

[0048] Figure 3 A schematic diagram showing a process of light field reconstruction for a three-layer light field display provided by an embodiment of the present application is shown;

[0049] Figure 4 A schematic diagram of a multi-layer light field display attenuation model and a polarization model provided in an embodiment of the present application is shown;

[0050] Figure 5 A schematic structural diagram of a light field display device provided by an embodiment of the present application is shown;

[0051] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0052] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0054] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0055] A light field display method, device, electronic device, and storage medium proposed according to embodiments of the present application are described below in conjunction with the accompanying drawings.

[0056] The embodiment of the present application provides a light field display method, which uses a light field depth estimation algorithm to process a light field image to obtain its depth information. The depth value of the light field reconstruction origin of the light field display device is modified according to its depth information, and the light field image is reconstructed based on the modified light field reconstruction origin. The influence of the depth characteristics of the light field image itself on the light field reconstruction origin is taken into account, the peak signal-to-noise ratio of the reconstructed light field image is improved, and the quality of light field reconstruction is greatly improved.

[0057] See also Figure 1 , the method specifically comprises the following steps:

[0058] Step 101: Obtain a light field image matrix.

[0059] The light field image matrix may be an n*m image array, such as a 5*5 or 7*7 image array. In the embodiment of the present application, a light field image may be obtained by photographing an object or scene that needs to be displayed in stereoscopic form with a light field camera, and the light field image matrix may be obtained by analyzing light field images taken with a similar light field. In other embodiments of the present application, the light field image matrix may also be obtained from some existing light field image data sets.

[0060] Step 102: Obtain a light field depth information map by light field depth estimation according to the light field image matrix.

[0061] First, multiple light field images included in the light field image matrix are stitched together into a stitched image. The position of each light field image in the stitched image corresponds one-to-one to the number of each light field image in the light field image matrix. Figure 2 The “light field image array” shown in the figure is a mosaic image formed by stitching together 49 car images in the 7*7 light field image matrix.

[0062] The three attributes of the light field image measurement data cost are: EPI (Epipolar Plane Image), angular spot and refocused image. Among them, the refocused image is equivalent to the light field image of the central perspective in the light field image matrix. According to the coordinates of each pixel point in the spliced ​​image and the number of pixels in the angular spot, the refocused image corresponding to the spliced ​​image is obtained by the following formula (1).

[0063]

[0064] In formula (1), R α (P) is the refocused image, P is a pixel in the mosaic, (u, v) is the coordinate of the pixel P, and |A| is the number of pixels in the corner spot. The corner spot can be a corner point in the mosaic.

[0065] After obtaining the refocused image through the above formula (1), the light field depth estimation is performed on the refocused image through the preset light field depth estimation algorithm to obtain a local depth map. The preset light field depth estimation algorithm can be CAE (constrained angular entropy cost) or CAD (constrained adaptive defocus cost), etc. CAE and CAD are depth estimation models established on the MAP-MRF (Maximum A Posteriori-MarkovRandom Field) framework. The preset light field depth estimation algorithm specifically performs light field depth estimation on the refocused image through the following formula (2), and then uses the cost constraint of the preset light field depth estimation algorithm through formula (3) to perform cost constraints on the image obtained by light field depth estimation to obtain a local depth map corresponding to the refocused image.

[0066]

[0067] d i (p) = arg min d C(P, d)...(3)

[0068] In formulas (2) and (3), E is the local depth map, P and q are the pixels in the refocused image, α(p) is the depth label at the pixel point P, N(P) is the pixel area at the pixel point P, and Eunary It is the data cost that measures the suitability of the depth label α(p) of pixel P. binary is the smoothing cost that enforces consistency between adjacent pixels, λ is the weighting factor of the smoothing cost, d is the depth value, and C is the cost volume.

[0069] The cost C is a function of the image pixel spatial coordinates P and the depth value d, reflecting the cost of different pixels at different depth levels. The smaller the cost, the more reliable the depth value of the pixel. The depth value d corresponding to the minimum cost of each spatial point is extracted through the above formula (3), thereby obtaining a local depth map.

[0070] After obtaining the local depth map of the refocused image through the above formulas (2) and (3), the local depth map is subjected to graph cut optimization and weighted median filtering to obtain a light field depth information map corresponding to the light field image matrix, which is the global depth map.

[0071] The process of obtaining the light field depth information map in this step is as follows Figure 2 As shown, the light field image array (i.e., the above-mentioned stitched image) is refocused to obtain a focal stack image (i.e., the above-mentioned refocused image), and the focal stack image is layered by a preset light field depth estimation algorithm (i.e., the depth of the refocused image is estimated by the above-mentioned formula (2)), and then a cost minimization process is performed (i.e., the depth value d corresponding to the minimum cost of each spatial point is extracted by the above-mentioned formula (3)) to obtain a local depth map, and finally, the local depth map is subjected to graph cut optimization and smoothing constraint (i.e., weighted median filtering) to obtain a global depth map, that is, a light field depth information map corresponding to the light field image matrix is ​​obtained.

[0072] Step 103: Determine an alternative reconstruction origin depth according to the light field depth information map.

[0073] The commonly used depth classification is 0-255, with a total of 256 classifications, but the embodiments of the present application include but are not limited to 256 classifications. The number of pixels corresponding to each depth value in the light field depth information map is counted separately. If the depth classification used is a 0-255 classification, the number of pixels with a depth value of 0, the number of pixels with a depth value of 1, ..., the number of pixels with a depth value of 254, and the number of pixels with a depth value of 255 are counted. From each depth value, a depth value that meets the pre-selection condition is selected as an alternative reconstruction origin depth. The pre-selection condition may be a preset number of depth values ​​with the largest number of pixels corresponding to the depth value, or the pre-selection condition may be that the ratio of the number of pixels corresponding to the depth value to the total number of pixels in the light field depth information map is greater than a preset threshold.

[0074] That is, a preset number of depth values ​​with the largest number of corresponding pixels is selected from each depth value; or, from each depth value, a depth value whose ratio of the corresponding number of pixels to the total number of pixels in the light field depth information map is greater than a preset threshold is selected.

[0075] The preset threshold may be 1%, 1.5% or 2%, etc., and the preset number may be 3, 5, 8, etc. The present application embodiment does not limit the specific values ​​of the preset threshold and the preset number, which can be set according to requirements in actual applications.

[0076] In other embodiments of the present application, after selecting depth values ​​that meet the requirements in any of the above methods, it is further determined whether there are numerically adjacent depth values ​​in the selected depth values. If not, all selected depth values ​​are directly determined as candidate reconstruction origin depths. If there are numerically adjacent depth values ​​in the selected depth values, the numerically adjacent depth values ​​are merged according to a preset merging principle.

[0077] Among them, the preset merging principle can stipulate that if there are an odd number of numerically adjacent depth values, the median is selected; if the number of numerically adjacent depth values ​​is an even number, the two depth values ​​in the middle position are selected, or any depth value in the middle position is selected.

[0078] By merging numerically adjacent depth values, the number of final reconstructed origin depths can be reduced, and the amount of computation required for subsequent reconstruction of light field images can be reduced.

[0079] After one or more depth values ​​that meet the requirements are selected by any of the above methods, the candidate reconstruction origin depth corresponding to each selected depth value is calculated according to the preset maximum depth value, the preset light field origin depth and each selected depth value. The preset maximum depth value is the maximum depth value in the grading method used. For example, if a grading method of 0-255 is used, the preset maximum depth value is 255. The preset light field origin depth is the default light field origin depth pre-set in the light field display device, and the preset light field origin depth can be 16.7 or 16.8, etc.

[0080] Specifically, the preset light field origin depth is scaled proportionally according to the ratio of the selected depth value to the preset maximum depth value, that is, the ratio between the candidate reconstruction origin depth corresponding to the selected depth value and the preset light field origin depth is equal to the ratio between the selected depth value and the preset maximum depth value.

[0081] For example, assuming that the selected depth value is 200, the preset maximum depth value is 255, and the preset light field origin depth is 16.7, then 200 / 255=x / 16.7, where x is the calculated alternative reconstruction origin depth, x=13.

[0082] The candidate reconstruction origin depth corresponding to each selected depth value is calculated respectively in the above manner.

[0083] Step 104: Reconstruct the light field image according to the alternative reconstruction origin depth.

[0084] Specifically, a central viewing angle picture in the light field picture matrix is ​​obtained, and the central viewing angle picture is a light field picture located at the center of the light field picture matrix, such as the light field picture in the 4th row and the 4th column in the 7*7 light field picture matrix is ​​the central viewing angle picture. Before obtaining the central viewing angle picture in the light field picture matrix, the deviation of the position of each light field picture in the light field picture matrix can also be corrected according to the field of view of each light field picture in the light field picture matrix.

[0085] Reconstruct the central perspective image according to each alternative reconstruction origin depth and the preset front and back depth of field range corresponding to the light field display device. After obtaining multiple alternative reconstruction origin depths through the above step 103, reconstruct the central perspective image based on each reconstruction origin depth. The method for reconstructing the central perspective image for each reconstruction origin depth is the same. The embodiment of the present application takes the first reconstruction origin depth as an example to illustrate the specific reconstruction process. The reconstruction process based on other reconstruction origin depths can refer to the reconstruction process corresponding to the first reconstruction origin depth. The first reconstruction origin depth is any reconstruction origin depth among the alternative reconstruction origin depths.

[0086] Specifically, the preset light field origin depth of the light field display device is modified to the first reconstruction origin depth, where the first reconstruction origin depth is any reconstruction origin depth among the candidate reconstruction origin depths; and the central perspective image is adjusted according to the first reconstruction origin depth and the preset front and back depth range corresponding to the light field display device. That is, the central perspective image is adjusted according to the first reconstruction origin depth and the preset front and back depth range corresponding to the light field display device, so that the adjusted central perspective image is in the depth plane where the first reconstruction origin depth is located and is in the preset front and back depth range.

[0087] In the light field reconstruction process, the depth range displayed by the multi-layer light field is twice the minimum value of the maximum value that the light field origin depth can deviate from in the horizontal and vertical directions. However, the depth range of the light field may be much larger than this depth range, so changing the position of the light field origin depth so that the required target area is displayed within this depth range as much as possible will result in higher reconstruction quality.

[0088] Specifically, the reconstructed central viewing angle image can be obtained according to the Malus formula shown in formula (4).

[0089] I=I0sinθ...(4)

[0090] In formula (4), I is the reconstructed central viewing angle image, I0 is the image intensity passing through the first display layer, and θ is the deflection angle between the light and the display layer.

[0091] After reconstructing the central perspective image according to each alternative reconstruction origin depth in the above manner, the peak signal-to-noise ratio of the adjusted central perspective image corresponding to each alternative reconstruction origin depth is detected respectively; the central perspective image with the largest peak signal-to-noise ratio is selected from the multiple adjusted central perspective images; and the central perspective image with the largest peak signal-to-noise ratio is determined as the final reconstructed central perspective image.

[0092] The embodiment of the present application is based on light field depth estimation of a light field image matrix, and uses the depth characteristics of the light field itself to modify the reconstruction origin depth of the light field display device multiple times, and reconstructs multiple central perspective images respectively, and selects the central perspective image with the highest reconstruction quality, thereby greatly improving the quality of the stereoscopic image displayed by the multi-layer light field display technology.

[0093] In other embodiments of the present application, after reconstructing the central perspective image, in a multi-layer light field model, when the light field is decomposed into a tensor product form, it is not possible to form a ray by taking any number of points in multiple display layers. According to the Malus formula shown in formula (4) and the principle that light propagates in a straight line, a ray can only be formed when multiple points are on a straight line. Therefore, only part of the light is valid. Therefore, it is necessary to constrain the reconstructed light field. Specifically, based on the central perspective image before reconstruction and the central perspective image after reconstruction, the reconstructed central perspective image is corrected. Specifically, the correction can be made by the following formula (5).

[0094]

[0095] In formula (5), I′ is the corrected central view image, and I is the central view image before reconstruction. This is the reconstructed central perspective image.

[0096] By correcting the reconstructed central view image in the above manner, the quality of the reconstructed light field is further improved.

[0097] After obtaining the reconstructed central view image in the above manner, the reconstructed central view image is tensor-decomposed according to the number of display layers included in the light field display device through formula (6) to obtain tensor values ​​of the number of display layers. According to the tensor values ​​of the number of display layers, display images corresponding to each display layer are generated through each display layer in the light field display device, and each display image is superimposed in space to display the final reconstructed light field image.

[0098]

[0099] In formula (6), I is the reconstructed central view image, I (1) ,I (2) ,...,I(n) They are the tensor values ​​corresponding to each display layer.

[0100] like Figure 3 As shown, after obtaining a light field depth information map through light field depth estimation based on the light field picture array, the light field depth information map, which is a grayscale map, is converted into an RGB (red, green, and blue) map based on the light field picture array and the light field depth information map. Figure 3 The RGB image shown in is a mosaic of light field images in the light field image array corresponding to the light field depth information image (a mosaic with depth information), and the light field center view is reconstructed based on the light field image array and the light field depth information it possesses. Figure 3 The light field display device based on which the light field reconstruction is shown has three liquid crystal layers. Finally, the reconstructed light field central view is tensor-decomposed to obtain a display image corresponding to each of the three liquid crystal layers.

[0101] In the multi-layer light field display model, the display layers other than the first layer mainly play the role of adjusting the brightness of the light field. Since the four-dimensional light field is a high-dimensional data in the multi-layer light field display, the light field data cannot be represented by a matrix, but the light field data can be modeled by using a tensor. In the embodiment of the present application, a polarization display model or an attenuation display model can be used for multi-layer light field display. Figure 4 The multi-layer light field display layer shown has three liquid crystal layers, where (a) is an attenuation display model, in which three liquid crystal layers are arranged in sequence between two polarizer layers. (b) is a polarization display model, in which a polarizer layer is arranged between adjacent liquid crystal layers.

[0102] In an embodiment of the present application, a light field depth estimation algorithm is used to process a light field image to obtain multiple alternative reconstruction origin depths. The preset reconstruction origin depths of the light field display device are respectively modified to the alternative reconstruction origin depths, and the light field image is reconstructed based on the modified light field reconstruction origin. From the multiple reconstructed light field images, the one with the largest peak signal-to-noise ratio is selected for display. In this way, the influence of the depth characteristics of the light field itself on the light field reconstruction origin is taken into account, and the position of the light field reconstruction origin is changed multiple times according to the depth characteristics of the light field itself, and multiple light field reconstruction results are obtained. The reconstruction result with the largest peak signal-to-noise ratio is selected for display, which greatly improves the quality of light field reconstruction and improves the display effect of multi-layer light field display.

[0103] The present application embodiment provides a light field display device, which is used to execute the light field display method provided in any of the above embodiments, such as Figure 5 As shown, the device comprises:

[0104] Image acquisition module 501, used to acquire a light field image matrix;

[0105] A depth estimation module 502 is used to obtain a light field depth information map by light field depth estimation according to the light field image matrix;

[0106] An origin depth determination module 503 is used to determine an alternative reconstruction origin depth according to the light field depth information map;

[0107] The light field reconstruction module 504 is used to reconstruct the light field image according to the alternative reconstruction origin depth.

[0108] The depth estimation module 502 is used to stitch the multiple light field images included in the light field image matrix into a stitched image; obtain a refocused image corresponding to the stitched image according to the coordinates of each pixel point in the stitched image and the number of pixels in the angular spot; perform light field depth estimation on the refocused image using a preset light field depth estimation algorithm to obtain a local depth map; perform graph cut optimization and weighted median filtering on the local depth map to obtain a light field depth information map corresponding to the light field image matrix.

[0109] The origin depth determination module 503 is used to respectively count the number of pixels corresponding to each depth value in the light field depth information map; select a preset number of depth values ​​with the largest number of corresponding pixels from each depth value; or, from each depth value, select a depth value whose ratio of the corresponding number of pixels to the total number of pixels in the light field depth information map is greater than a preset threshold; and calculate the alternative reconstruction origin depth corresponding to each selected depth value according to the preset maximum depth value, the preset light field origin depth and each selected depth value.

[0110] The device also includes: a merging module, which is used to merge the depth values ​​adjacent to each other according to a preset merging principle if there are depth values ​​adjacent to each other in the selected depth values.

[0111] The light field reconstruction module 504 is used to obtain a central perspective image in the light field image matrix; reconstruct the central perspective image according to each selected reconstruction origin depth and a preset foreground and background depth range corresponding to the light field display device; perform tensor decomposition on the reconstructed central perspective image according to the number of display layers included in the light field display device to obtain tensor values ​​of the number of display layers; generate display images corresponding to each display layer through each display layer in the light field display device according to the tensor values ​​of the number of display layers, and superimpose each display image in space to obtain a reconstructed light field image.

[0112] The light field reconstruction module 504 is used to modify the preset light field origin depth of the light field display device to a first reconstruction origin depth, where the first reconstruction origin depth is any reconstruction origin depth among the alternative reconstruction origin depths; adjust the central perspective image according to the first reconstruction origin depth and the preset foreground and background depth range corresponding to the light field display device; respectively detect the peak signal-to-noise ratio of the adjusted central perspective image corresponding to each alternative reconstruction origin depth; select the central perspective image with the largest peak signal-to-noise ratio from the multiple adjusted central perspective images; and determine the central perspective image with the largest peak signal-to-noise ratio as the final reconstructed central perspective image.

[0113] The device also includes: a correction module, which is used to correct the reconstructed central perspective image according to the central perspective image before reconstruction and the reconstructed central perspective image.

[0114] The light field display device provided in the above-mentioned embodiment of the present application and the light field display method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0115] The present application also provides an electronic device to perform the above light field display method. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6 As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, wherein the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, the light field display method provided in any of the aforementioned embodiments of the present application is executed.

[0116] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0117] The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store a program, and the processor 600 executes the program after receiving an execution instruction. The light field display method disclosed in any implementation of the embodiment of the present application may be applied to the processor 600, or implemented by the processor 600.

[0118] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0119] The electronic device provided in the embodiment of the present application and the light field display method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0120] The present application also provides a computer-readable storage medium corresponding to the light field display method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is an optical disc 30 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the light field display method provided by any of the aforementioned embodiments will be executed.

[0121] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0122] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the light field display method provided in the embodiment of the present application are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0123] It should be noted that:

[0124] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0125] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0126] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0127] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A light field display method, characterized in that: include: Get the light field image matrix; Obtaining a light field depth information map by light field depth estimation according to the light field image matrix; Determining an alternative reconstruction origin depth according to the light field depth information map; Reconstructing a light field image according to the alternative reconstruction origin depth; The step of obtaining a light field depth information map by light field depth estimation according to the light field picture matrix includes: The plurality of light field images included in the light field image matrix are stitched together into a stitched image; a refocused image corresponding to the stitched image is obtained according to the coordinates of each pixel point in the stitched image and the number of pixels in the angular spot; a light field depth estimation is performed on the refocused image using a preset light field depth estimation algorithm to obtain a local depth map; and the local depth map is subjected to graph cut optimization and weighted median filtering to obtain a light field depth information map corresponding to the light field image matrix; The step of determining an alternative reconstruction origin depth according to the light field depth information map includes: The number of pixels corresponding to each depth value in the light field depth information map is counted respectively; a preset number of depth values ​​having the largest number of corresponding pixels are selected from each depth value; or, from each depth value, a depth value having a ratio of the number of corresponding pixels to the total number of pixels in the light field depth information map greater than a preset threshold is selected; and an alternative reconstruction origin depth corresponding to each selected depth value is calculated respectively according to a preset maximum depth value, a preset light field origin depth and each selected depth value; The step of reconstructing the light field image according to the alternative reconstruction origin depth includes: Acquire a central viewing angle image in the light field image matrix; reconstruct the central viewing angle image according to each selected reconstruction origin depth and a preset front and rear depth of field range corresponding to the light field display device; perform tensor decomposition on the reconstructed central viewing angle image according to the number of display layers included in the light field display device to obtain tensor values ​​of the number of display layers; generate display images corresponding to each display layer through each display layer in the light field display device according to the tensor values ​​of the number of display layers, and superimpose each display image in space to obtain a reconstructed light field image; The step of reconstructing the central viewing angle image according to each of the alternative reconstruction origin depths and the preset front and rear depth of field ranges corresponding to the light field display device includes: The preset light field origin depth of the light field display device is modified to a first reconstruction origin depth, where the first reconstruction origin depth is any reconstruction origin depth among the alternative reconstruction origin depths; the central perspective image is adjusted according to the first reconstruction origin depth and the preset front and back depth of field range corresponding to the light field display device; the peak signal-to-noise ratio of the adjusted central perspective image corresponding to each alternative reconstruction origin depth is detected respectively; the central perspective image with the largest peak signal-to-noise ratio is selected from the multiple adjusted central perspective images; and the central perspective image with the largest peak signal-to-noise ratio is determined as the final reconstructed central perspective image.

2. The method according to claim 1, characterized in that Before respectively calculating the candidate reconstruction origin depth corresponding to each selected depth value according to the preset maximum depth value, the preset light field origin depth and each selected depth value, the method further includes: If there are depth values ​​with adjacent values ​​among the selected depth values, the depth values ​​with adjacent values ​​are merged according to a preset merging principle.

3. The method according to claim 1, characterized in that After reconstructing the central viewing angle picture, the method further includes: According to the central perspective image before reconstruction and the central perspective image after reconstruction, the reconstructed central perspective image is corrected.

4. A light field display device, characterized in that: include: Image acquisition module, used to acquire light field image matrix; A depth estimation module, configured to obtain a light field depth information map by light field depth estimation according to the light field image matrix; An origin depth determination module, used to determine an alternative reconstruction origin depth according to the light field depth information map; A light field reconstruction module, used to reconstruct a light field image according to the alternative reconstruction origin depth; The step of obtaining a light field depth information map by light field depth estimation according to the light field picture matrix includes: The plurality of light field images included in the light field image matrix are stitched together into a stitched image; a refocused image corresponding to the stitched image is obtained according to the coordinates of each pixel point in the stitched image and the number of pixels in the angular spot; a light field depth estimation is performed on the refocused image using a preset light field depth estimation algorithm to obtain a local depth map; and the local depth map is subjected to graph cut optimization and weighted median filtering to obtain a light field depth information map corresponding to the light field image matrix; The step of determining an alternative reconstruction origin depth according to the light field depth information map includes: The number of pixels corresponding to each depth value in the light field depth information map is counted respectively; a preset number of depth values ​​having the largest number of corresponding pixels are selected from each depth value; or, from each depth value, a depth value having a ratio of the number of corresponding pixels to the total number of pixels in the light field depth information map greater than a preset threshold is selected; and an alternative reconstruction origin depth corresponding to each selected depth value is calculated respectively according to a preset maximum depth value, a preset light field origin depth and each selected depth value; The step of reconstructing the light field image according to the alternative reconstruction origin depth includes: Acquire a central viewing angle image in the light field image matrix; reconstruct the central viewing angle image according to each selected reconstruction origin depth and a preset front and rear depth of field range corresponding to the light field display device; perform tensor decomposition on the reconstructed central viewing angle image according to the number of display layers included in the light field display device to obtain tensor values ​​of the number of display layers; generate display images corresponding to each display layer through each display layer in the light field display device according to the tensor values ​​of the number of display layers, and superimpose each display image in space to obtain a reconstructed light field image; The step of reconstructing the central viewing angle image according to each of the alternative reconstruction origin depths and the preset front and rear depth of field ranges corresponding to the light field display device includes: The preset light field origin depth of the light field display device is modified to a first reconstruction origin depth, where the first reconstruction origin depth is any reconstruction origin depth among the alternative reconstruction origin depths; the central perspective image is adjusted according to the first reconstruction origin depth and the preset front and back depth of field range corresponding to the light field display device; the peak signal-to-noise ratio of the adjusted central perspective image corresponding to each alternative reconstruction origin depth is detected respectively; the central perspective image with the largest peak signal-to-noise ratio is selected from the multiple adjusted central perspective images; and the central perspective image with the largest peak signal-to-noise ratio is determined as the final reconstructed central perspective image.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 3.

Images