Method for generating integral-imaging micro-image array on basis of optimal voxel space distribution

Abstract

The present invention relates to a method for generating an integral-imaging micro-image array on the basis of optimal voxel space distribution. The method comprises the steps of: acquiring depth data and texture data of a 3D scene; on the basis of the relationship between a voxel space of an integral-imaging 3D display and 3D display performance, from the entire voxel space of the integral-imaging 3D display, selecting, as an optimal voxel space, part of the voxel space which satisfies the requirements of the 3D display performance; and by using the obtained depth data and texture data of the 3D scene as inputs, on the basis of the selected optimal voxel space, synthesizing a micro-image array which satisfies the display performance requirements of the integral-imaging 3D display. The method can obtain depth data and texture data by means of existing mature technology and product, without depending on the structure of an integral-imaging display system itself; and the method flexibly adjusts the resolution of the acquired depth data and texture data by means of an optimal voxel space, and thus is suitable for integral-imaging 3D displays having different specification parameters.

Description

Integrated imaging micro-image array generation method based on optimal voxel spatial distribution Technical Field

[0001] The present invention relates to integrated imaging technology, and in particular to an integrated imaging micro-image array generation method based on optimal voxel spatial distribution. Background Art

[0002] Integrated imaging 3D displays are a type of glasses-free 3D display. Compared to existing glasses- or head-mounted 3D displays, they require no additional equipment, allowing users to see 3D images that appear protruding or recessed from the display, much like watching a regular TV or monitor. Structurally, they are achieved by attaching a specially designed optical device (a microlens array) to a regular 2D display (such as a liquid crystal display (LCD) or projection screen), allowing the two to be precisely coupled.

[0003] Integrated imaging 3D displays require a unique 3D source, typically composed of multiple tiny images arranged in a staggered arrangement of pixels, also known as a micro-image array. Integrated imaging typically relies on complex equipment to capture the 3D scene, then synthesizes the micro-image array using algorithms. This process is also known as 3D source generation. Furthermore, the depth range of the 3D scene during capture is limited. This is because the 3D image reconstructed by integrated imaging can only project or recess within a very small range in front of or behind the display, typically a few centimeters to tens of centimeters. Images outside this range become blurry, and can even severely impact the viewing experience. Real-world 3D scenes can have a wide depth range, making it difficult for existing integrated imaging technologies to capture and display realistic 3D scenes. Consequently, integrated imaging faces the challenge of a shortage of 3D source material. Currently, integrated imaging 3D source material used for academic research or commercial presentations mostly uses computer software to create 3D models. This involves capturing images using a virtual micro-lens array or camera array within a computer, and then synthesizing the images using a compositing program. Technical issues

[0004] Existing methods for generating integrated imaging 3D source footage typically employ a camera array (either real or virtual) arranged in a specific pattern to capture a 3D scene from multiple angles. Each camera captures an image from a specific perspective, which is then processed pixel by pixel through a program to generate the 3D source footage. However, these methods are limited by the bulkiness of the camera array and the need for complex calibration and synchronization mechanisms, making commercialization difficult. Another approach involves capturing images using light-field cameras, but their low resolution results in poor quality reconstructed 3D images. In recent years, some researchers have proposed using depth cameras to capture texture and depth data of the 3D scene, known as RGB-D images. These images are then synthesized using computer image processing methods, mapping 3D pixels to voxels on the display. This approach eliminates the inherent structural dependence of integrated imaging 3D source generation and makes source acquisition more universal. However, it fails to integrate the capture and display processes and imposes certain requirements on the depth range of the captured 3D scene, which must not exceed the depth of field of the 3D display. Otherwise, the displayed 3D image will be blurry. Clearly, this approach is unlikely to be widely adopted. Technical Solutions

[0005] Therefore, the present invention proposes a multi-source and universal integrated imaging micro-image array generation method to solve the current problem of integrated imaging facing a shortage of micro-image arrays (3D chip sources), which affects its application and promotion.

[0006] The purpose of this invention is to propose a multi-source and universal method for generating integrated imaging micro-image arrays, addressing the current shortage of micro-image arrays (3D image sources) in integrated imaging, which hinders its widespread application. Multi-source means that the acquisition of depth and texture data no longer relies on the inherent structure of the integrated imaging display system. Instead, a common set of RGB-D images can be obtained using established technologies and products such as depth cameras and 3DS MAX software. Versatility is reflected in the flexible adjustment of the resolution of the acquired depth and texture data using an optimal voxel space to accommodate integrated imaging 3D displays of varying specifications.

[0007] The principle of integrated imaging 3D display is shown in Figure 1. The magic cube represents the virtual 3D image reconstructed by the integrated imaging display device, which is considered to be composed of individual voxel units. Based on the principle that the optical paths of the capture and display processes of integrated imaging are reversible, each voxel in the 3D image corresponds to a group of pixels in the micro-image array (called homonymous pixels). During the 3D display process, the light beams emitted by a group of homonymous pixels are refracted by the microlens units in front of them and converge to form an image point at a certain position in space. This image point is a voxel in the 3D image. If the light beam of each pixel is regarded as a principal ray and the lens unit is regarded as a pinhole imaging model, the spatial position of the voxel is the intersection of the principal rays emanating from multiple homonymous pixels in space.

[0008] The technical solutions of the present invention are as follows:

[0009] A method for generating an integrated imaging micro-image array based on optimal voxel spatial distribution, the method comprising the steps of:

[0010] S1. Obtain depth data and texture data of the 3D scene;

[0011] S2. Selecting an optimal voxel space: Based on the relationship between the voxel space of the integrated imaging 3D display and the 3D display performance, selecting a portion of the voxel space that meets the 3D display performance requirements from the entire voxel space of the integrated imaging 3D display as the optimal voxel space;

[0012] S3. Synthesize micro-image array: Using the obtained depth data and texture data of the 3D scene as input, synthesize a micro-image array that meets the display performance requirements of the integrated imaging 3D display based on the selected optimal voxel space.

[0013] Furthermore, the steps of selecting the optimal voxel space include:

[0014] S2-1, discarding the integrated image surface with missing voxels from the entire voxel space of the integrated imaging 3D display;

[0015] S2-2, roughly selecting an integrated image plane with relatively high voxel spatial resolution according to the position of the central depth plane;

[0016] S2-3. On the basis of step S2-2, according to the 3D depth of field requirement of the integrated imaging 3D display and the overlap between adjacent voxels, select a portion of the voxel space that meets the performance requirement as the optimal voxel space.

[0017] Furthermore, the step of synthesizing the micro-image array includes:

[0018] S3-1, transforming the obtained depth data of the 3D scene into a depth range corresponding to the optimal voxel space, and dividing the obtained depth data of the 3D scene into a plurality of integrated image planes corresponding to the optimal voxel space;

[0019] S3-2, segmenting the obtained texture data of the 3D scene and attaching it to each integrated image plane, and adjusting the spatial resolution of the texture data of the 3D scene on each integrated image plane to the voxel spatial resolution of the corresponding integrated image plane by upsampling or downsampling, thereby generating a texture slice map on the corresponding integrated image plane;

[0020] S3-3, mapping the texture slice image on each integrated image surface to the micro-image array plane point by point through the mapping relationship between voxels and pixels of the same name, thereby generating a sub-micro-image array corresponding to each integrated image surface;

[0021] S3-4. Based on the relationship between parallax (in pixels) and depth, the sub-micro-image array corresponding to each integrated image surface is extended in the direction of the hole (to eliminate the image quality degradation caused by the hole), and the extended sub-micro-image arrays are fused to generate a complete micro-image array.

[0022] Furthermore, the steps of selecting the optimal voxel space include:

[0023] S2-1. Discarding the integrated image surface with missing voxels from the entire voxel space of the integrated imaging 3D display, specifically including:

[0024] A world coordinate system is established with the geometric center of the microlens array plane as the origin, where the XY plane coincides with the microlens array plane, and the Z axis is perpendicular to the microlens array plane. Suppose each image element contains pixels (ie, R pixels in the horizontal and vertical directions respectively), each image element will emit The root reconstructed light propagates in the 3D image space at the same angular interval. The angular interval of adjacent reconstructed light is:

[0025]

[0026] Where g is the distance between the pixel plane and the microlens array plane, is the pixel size on a 2D display;

[0027] The reconstruction rays emitted by a pixel in any image element intersect with each reconstruction ray from its adjacent image elements, forming at most voxels, and the voxels are distributed in At different depth positions, the total number of integrated image surfaces carrying all voxels is:

[0028]

[0029] Let k( ) integrated image planes are , its distance to the microlens array plane Expressed as:

[0030]

[0031] Where p represents the pitch of the lens element;

[0032] Spacing between adjacent integrated image surfaces for:

[0033]

[0034] set up and Represents the integrated image surface The intervals between adjacent voxels in the horizontal and vertical directions are:

[0035]

[0036] The voxels formed by the intersection of the reconstruction rays of two adjacent image elements are distributed in a quadrangular pyramid shape. Some areas are unevenly distributed due to the lack of some voxels. The number of integrated image surfaces with uneven voxel distribution is obtained through geometric relationships:

[0037]

[0038] Among them, round(.) means rounding to the nearest integer;

[0039] The integrated image surface is cropped to the same size as the 2D display screen. The number of voxels on the cropped integrated image surface is expressed as:

[0040]

[0041] in, Represents the number of voxels in the horizontal direction on the cropped integrated image surface, represents the number of voxels in the vertical direction on the cropped integrated image surface, W represents the width of the 2D display screen, and H represents the height of the 2D display screen;

[0042] Equations (1-1) to (1-8) describe all voxel characteristic parameters of the integrated imaging 3D display. Obviously, the reverse extension of the reconstructed light will also form a similar voxel space behind the screen, and its characteristic parameters have the same expression as the characteristic parameters of the voxel space in front of the screen.

[0043] The present invention completely reveals the voxel spatial distribution law of the integrated imaging 3D display for the first time.

[0044] S2-2. Roughly select an integrated image plane with relatively high voxel spatial resolution based on the position of the central depth plane, specifically including:

[0045] set up and Represents integrated image surface The spatial resolution of the voxels in the horizontal and vertical directions is expressed as:

[0046]

[0047] From equations (1-9) and (1-10), it can be seen that the voxel spatial resolution decreases with the increase of the integrated image plane number, that is, the voxel spatial resolution on the integrated image plane closer to the microlens array plane is higher, and the voxel spatial resolution on the integrated image plane farther away from the microlens array plane gradually decreases;

[0048] From equations (1-5), (1-7) and (1-8), we can see that when the parameters of the 2D display (screen width W of the 2D display, screen height H of the 2D display and pixel size of the 2D display) are ) and the parameters of the lens element (the pitch p of the lens element) are determined, the number of voxels on each integrated image plane is determined, and the size of the voxels on each integrated image plane is related to the position of the central depth plane;

[0049] The position of the central depth plane is obtained by the Gaussian imaging formula:

[0050]

[0051] Where, l represents the distance from the central depth plane to the microlens array plane, and f is the focal length of the lens element;

[0052] Therefore, by adjusting the distance g between the microlens array plane and the pixel plane, the central depth plane is set near the integrated image plane with denser voxels, and then the integrated image plane with relatively high voxel spatial resolution is roughly selected according to the position of the central depth plane;

[0053] S2-3. Based on step S2-2, according to the 3D depth of field requirements of the integrated imaging 3D display and the overlap between adjacent voxels, select a portion of the voxel space that meets the performance requirements as the optimal voxel space, specifically including:

[0054] According to the geometric relationship, at a distance from the microlens array plane On the integrated image plane, the voxel size Expressed as:

[0055]

[0056] When adjacent voxels overlap, their overlap Expressed as:

[0057]

[0058] in, Represents the interval between adjacent voxels, and its value is given by and Given; given overlap threshold , the depth positions of the front and rear edge integrated image planes of the 3D depth of field range of the integrated imaging 3D display can be determined; the edge integrated image plane closest to the viewer is called the front edge integrated image plane, and the edge integrated image plane farthest from the viewer is called the rear edge integrated image plane; assuming that the distances between the front and rear edge integrated image planes and the microlens array plane are and , then the 3D depth of field of the integrated imaging 3D display is expressed as:

[0059]

[0060] Therefore, based on step S2-2, the front and rear edge integrated image planes are selected according to the 3D depth of field requirements of the integrated imaging 3D display and the overlap between adjacent voxels, so that the 3D image reconstructed between the front and rear edge integrated image planes always has high clarity.

[0061] Furthermore, the mapping relationship between the voxels and the pixels of the same name is:

[0062]

[0063] Where Vx represents a voxel set, HomoPx represents a pixel set, and F represents a mapping function. (For any element in the voxel set Vx, the number of corresponding elements in the pixel set HomoPx represents the number of reconstruction rays that constitute that voxel. Therefore, regions of interest in a 3D scene can be reconstructed on voxels with a larger number of reconstruction rays, resulting in denser viewing angles and smoother motion parallax.)

[0064] Furthermore, the method for obtaining depth data and texture data of a 3D scene includes: using a depth camera or 3D modeling software to obtain an RGB-D image of the 3D scene, wherein the RGB-D image includes depth data and texture data. Beneficial effects

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] 1. Acquisition of depth data and texture data: By simply adopting existing mature technologies and products, such as depth cameras and 3DS MAX software, more universal 3D information sources can be provided for integrated imaging 3D display technology, thereby freeing the 3D information acquisition process from dependence on the complex and delicate optical structure of integrated imaging itself.

[0067] 2. Selection of the optimal voxel space: Each integrated imaging 3D display with different structural parameters has its own unique voxel space. By selecting the part that best meets the system's 3D display performance requirements, the generated micro-image array (3D film source) can be matched to each specific integrated imaging 3D display, thus avoiding compatibility issues such as the inability of existing micro-image arrays (3D film sources) to display on 3D displays with different parameters.

[0068] 3. Synthesis of micro-image arrays: Arbitrary depth data can be flexibly transformed into the depth range of the optimal voxel space, and the spatial resolution of the texture data can be adjusted to be consistent with the voxel spatial resolution through upsampling or downsampling, so that each voxel can be accurately mapped to the corresponding pixel of the same name.

[0069] 4. The present invention fully reveals for the first time the spatial distribution law of voxels of an integrated imaging 3D display.

[0070] The present invention will be further described in detail below through specific implementation methods and drawings, but this does not mean to limit the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] FIG1 is a schematic diagram of the integrated imaging 3D display principle.

[0072] FIG2 is a schematic diagram of the spatial distribution of voxels according to an embodiment of the present invention.

[0073] FIG3 is a complete three-dimensional voxel spatial distribution diagram according to an embodiment of the present invention.

[0074] FIG4 is a schematic diagram of an actual voxel reconstruction process according to an embodiment of the present invention.

[0075] FIG5 is a schematic diagram of voxel size calculation according to an embodiment of the present invention.

[0076] FIG6 is a schematic diagram of 3D depth of field calculation according to an embodiment of the present invention.

[0077] FIG7 is a schematic diagram of a preferred solution of voxel space clipping and display performance according to an embodiment of the present invention.

[0078] FIG8 is a flow chart of a method for generating an integrated imaging micro-image array based on optimal voxel spatial distribution according to an embodiment of the present invention.

[0079] FIG9 is an RGB-D image pair obtained by a depth camera according to an embodiment of the present invention.

[0080] FIG10 is an RGB-D image pair of a virtual 3D scene created by 3DS MAX software according to an embodiment of the present invention.

[0081] FIG. 11 is a diagram showing the voxel spatial distribution and optimization results in the real mode according to an embodiment of the present invention.

[0082] FIG12 is a USAF 1951 resolution test target according to an embodiment of the present invention.

[0083] FIG13 is a diagram showing the reconstructed resolution test target image on each integrated image plane according to an embodiment of the present invention.

[0084] FIG14 is a comparison diagram of actual voxel values ​​and theoretical values ​​according to an embodiment of the present invention.

[0085] FIG15 is a comparison diagram of the image quality of high-frequency and low-frequency image reconstruction on different integrated image planes according to an embodiment of the present invention.

[0086] FIG16 is a diagram showing the reconstruction effect of a 3D image in real mode according to an embodiment of the present invention.

[0087] FIG17 is a diagram showing the reconstruction effect of a 3D image in a virtual mode according to an embodiment of the present invention. Modes for Carrying Out the Invention Example

[0088] This example provides a method for generating an integrated imaging micro-image array based on optimal voxel spatial distribution, the method comprising the steps of:

[0089] S1. Obtain depth data and texture data of a 3D scene.

[0090] In this example, a depth camera or 3D modeling software is used to obtain an RGB-D image of a 3D scene. The RGB-D image contains depth data and texture data.

[0091] S2. Selecting the optimal voxel space: Based on the relationship between the voxel space of the integrated imaging 3D display and the 3D display performance, a portion of the voxel space that meets the 3D display performance requirements is selected from the entire voxel space of the integrated imaging 3D display as the optimal voxel space.

[0092] The steps to select the optimal voxel space in this example include:

[0093] S2-1. Discarding the integrated image surface with missing voxels from the entire voxel space of the integrated imaging 3D display, specifically including:

[0094] A world coordinate system is established with the geometric center of the microlens array plane as the origin, where the XY plane coincides with the microlens array plane, and the Z axis is perpendicular to the microlens array plane. Suppose each image element contains pixels (ie, R pixels in the horizontal and vertical directions respectively), each image element will emit The root reconstructed light propagates in the 3D image space at the same angular interval. The angular interval of adjacent reconstructed light is:

[0095]

[0096] Where g is the distance between the pixel plane and the microlens array plane, is the pixel size on a 2D display. (Since most display devices use square pixels, i.e., a pixel aspect ratio of 1:1, the angular spacing of the reconstructed rays along the X and Y axes is equal. Equation (1-1) also applies to displays with non-square pixel structures and is not elaborated on here.)

[0097] The reconstruction rays emitted by a pixel in any image element intersect with each reconstruction ray from its adjacent image elements, forming at most voxels, and the voxels are distributed in At different depth positions, the total number of integrated image surfaces carrying all voxels is:

[0098]

[0099] Let k( ) integrated image planes are , its distance to the microlens array plane Expressed as:

[0100]

[0101] Where p represents the pitch of the lens elements. (In traditional integrated imaging light field acquisition, the pitch of the image elements is usually equal to the pitch of the lens elements to maximize the utilization of the pixel units in the micro-image array and avoid overlap of the boundary areas of adjacent image elements.)

[0102] Spacing between adjacent integrated image surfaces for:

[0103]

[0104] set up and Represents the integrated image surface The intervals between adjacent voxels in the horizontal and vertical directions are:

[0105]

[0106] The reconstructed light rays of two adjacent image elements intersect each other to form a set of voxels distributed in a pyramid shape. Some areas are unevenly distributed due to the lack of some voxels (as shown in Figure 2, the top area of ​​the pyramid shown by the dotted box in Figure 2 is unevenly distributed due to the lack of some voxels; while in the bottom area, some voxels in adjacent pyramids overlap, so that the voxels in these areas are evenly distributed on each integrated image surface.). The number of integrated image surfaces with uneven voxel distribution is calculated through geometric relationships:

[0107]

[0108] Among them, round(.) means rounding to the nearest integer;

[0109] The integrated image surface is cropped to the same size as the 2D display screen. The number of voxels on the cropped integrated image surface is expressed as:

[0110]

[0111] in, Represents the number of voxels in the horizontal direction on the cropped integrated image surface, represents the number of voxels in the vertical direction on the cropped integrated image surface, W represents the width of the 2D display screen, and H represents the height of the 2D display screen;

[0112] It should be noted that in the above process, only the intersection of pixel rays in two adjacent image elements is considered. In fact, the reconstructed rays of non-adjacent image elements will also form intersection points in the 3D image space, as shown by the dotted circles in Figure 2. From the perspective of the integrated imaging light field acquisition process, the 3D object points corresponding to these points are only recorded by two or more non-continuous lens elements. This is inconsistent with the continuous sampling of the 3D object light field by the microlens array in actual situations. Therefore, these intersection points should not be regarded as voxels of the 3D image. In addition, since each pixel on a real display has a certain size, the light beams of a group of pixels with the same name overlap in space to form a diffuse diffuse spot. The diffuse spots at these positions will be blocked by other diffuse spots on the integrated image surface in front of them. Therefore, the lack of these voxels will not have much impact on the quality of the reconstructed 3D image.

[0113] Equations (1-1) through (1-8) describe all the voxel characteristic parameters of an integrated imaging 3D display. It can be seen that these voxel characteristic parameters are solely dependent on the structural parameters of the integrated imaging 3D display, such as the lens element pitch, the number of lens units, the screen size and pixel size of the 2D display, and the spacing between the pixel plane and the microlens array plane. In other words, once the system's structural parameters are determined, all possible voxel positions and their spatial distribution patterns for the integrated imaging 3D display are uniquely determined.

[0114] Obviously, the reverse extension line of the reconstructed light will also form a similar voxel space behind the screen, and its characteristic parameters have the same expression as the characteristic parameters of the voxel space in front of the screen. The complete three-dimensional voxel space distribution is shown in Figure 3.

[0115] Figure 2 shows all possible voxel locations for 3D image reconstruction on an integrated imaging 3D display. is the first integrated image surface, For the An integrated image plane, For the An integrated image plane, is the distance from the kth integrated image plane to the microlens array plane, Integrated image surface The distance between adjacent voxels in the horizontal direction, is the angular interval between adjacent reconstructed rays. However, in practice, the first K integrated image planes shown in the dashed box in Figure 2 have a small number of voxels and are unevenly distributed. Therefore, they are not suitable for reconstructing continuous 3D scenes. These integrated image planes can be discarded when optimizing the voxel space.

[0116] S2-2. Roughly select an integrated image plane with relatively high voxel spatial resolution based on the position of the central depth plane, specifically including:

[0117] set up and Represents integrated image surface The spatial resolution of the voxels in the horizontal and vertical directions is expressed as:

[0118]

[0119] From equations (1-9) and (1-10), it can be seen that the voxel spatial resolution decreases with the increase of the integrated image plane number, that is, the voxel spatial resolution on the integrated image plane closer to the microlens array plane is higher, and the voxel spatial resolution on the integrated image plane farther away from the microlens array plane gradually decreases;

[0120] It should be noted that the voxel spatial distribution shown in Figure 2 is obtained based on the voxel reconstruction process under ideal conditions, that is, the pixels on the integrated imaging 3D display are simplified to ideal point light sources, and the influence of lens aberrations is not considered, resulting in ideal point voxels. In reality, each pixel on the integrated imaging 3D display is a planar light-emitting unit of a certain size. The light beam emitted by each pixel unit is modulated by the lens element to form a divergent light beam that propagates in the 3D image space. On the central depth plane that is conjugate with the pixel plane with respect to the microlens array plane, the light beam converges into a minimum light spot. At other depth positions before and after the central depth plane, the light beam diffuses into a larger diffuse spot, as shown in Figure 4 (Figure 4, is the pixel size on a 2D display, (where is the distance from the kth integrated image plane to the microlens array plane, and l is the distance from the central depth plane to the microlens array plane). Therefore, the voxels formed by the light beams from the same-name point pixels converging on the integrated image planes away from the central depth plane are actually diffuse spots, such as voxel A and voxel B shown in Figure 4. Without considering lens aberrations, there is no overlap between adjacent voxels at the central depth plane. However, voxels on other integrated image planes before and after the central depth plane will partially overlap due to voxel diffusion. The farther the integrated image plane is from the central depth plane, the greater the spread of voxel size, and accordingly, the higher the overlap ratio between voxels. This overlap will lead to a decrease in voxel spatial resolution. Therefore, the voxel spatial resolution of a 3D image is determined by both the number of voxels and the voxel size. Within a certain spatial range, the greater the number of voxels and the smaller the size, the higher the voxel spatial resolution.

[0121] From equations (1-5), (1-7) and (1-8), we can see that when the parameters of the 2D display (screen width W of the 2D display, screen height H of the 2D display and pixel size of the 2D display) are ) and the parameters of the lens element (the pitch p of the lens element) are determined, the number of voxels on each integrated image plane is determined, and the size of the voxels on each integrated image plane is related to the position of the central depth plane;

[0122] The position of the central depth plane is obtained by the Gaussian imaging formula:

[0123]

[0124] Where, l represents the distance from the central depth plane to the microlens array plane, and f is the focal length of the lens element;

[0125] Therefore, by adjusting the distance g between the microlens array plane and the pixel plane, the central depth plane is set near the integrated image plane with denser voxels, and then the integrated image plane with relatively high voxel spatial resolution is roughly selected according to the position of the central depth plane;

[0126] S2-3. Based on step S2-2, according to the 3D depth of field requirements of the integrated imaging 3D display and the overlap between adjacent voxels, select a portion of the voxel space that meets the performance requirements as the optimal voxel space, specifically including:

[0127] According to the geometric relationship, the size of the voxel on the integrated image plane at a distance z from the microlens array plane is Expressed as:

[0128]

[0129] Figure 5 is a schematic diagram of voxel size calculation. is the pixel size on a 2D display, is the pitch of the lens element, is the size of the voxel, l is the distance from the center depth plane to the microlens array plane, is the distance from the integrated image plane to the microlens array plane.

[0130] When adjacent voxels overlap, their overlap Expressed as:

[0131]

[0132] in, Represents the interval between adjacent voxels, whose value is given by Δx(k) and Δy(k) in formula (1-5); given the overlap threshold , the depth positions of the front and rear edge integrated image planes of the 3D depth of field range of the integrated imaging 3D display can be determined; the edge integrated image plane closest to the viewer is called the front edge integrated image plane, and the edge integrated image plane farthest from the viewer is called the rear edge integrated image plane; assuming that the distances between the front and rear edge integrated image planes and the microlens array plane are and , then the 3D depth of field of the integrated imaging 3D display is expressed as:

[0133]

[0134] Figure 6 is a schematic diagram of 3D depth of field calculation. is the distance between adjacent voxels, is the size of the voxel, is the distance between the front edge integrated image plane and the microlens array plane, is the distance between the rear edge integrated image plane and the microlens array plane, 3D depth of field.

[0135] It is worth noting that although this example uses the idea of ​​Rayleigh criterion to determine the edge plane where adjacent voxels can be just distinguished, in actual applications, the threshold value of voxel overlap is not very accurate. The choice of is far less strict than the Rayleigh criterion. This is because the Rayleigh criterion is a relatively strict objective judgment condition, usually used to measure the resolution limit of an optical system, while the human visual system has a much higher tolerance to image blur than a general optical system. In addition, the voxel overlap threshold It can also be calculated based on the actual value of the human eye's resolution limit, or measured through experimental methods based on the human eye's subjective perception of image clarity.

[0136] Therefore, based on step S2-2, the front and rear edge integrated image planes are selected according to the 3D depth of field requirements of the integrated imaging 3D display and the overlap between adjacent voxels, so that the 3D image reconstructed between the front and rear edge integrated image planes always has high clarity.

[0137] S3. Synthesize micro-image array: Using the obtained depth data and texture data of the 3D scene as input, synthesize a micro-image array that meets the display performance requirements of the integrated imaging 3D display based on the selected optimal voxel space.

[0138] The steps of synthesizing the micro-image array in this example include:

[0139] S3-1, transforming the obtained depth data of the 3D scene into a depth range corresponding to the optimal voxel space, and dividing the obtained depth data of the 3D scene into a plurality of integrated image planes corresponding to the optimal voxel space;

[0140] S3-2, segmenting the obtained texture data of the 3D scene and attaching it to each integrated image plane, and adjusting the spatial resolution of the texture data of the 3D scene on each integrated image plane to the voxel spatial resolution of the corresponding integrated image plane by upsampling or downsampling, thereby generating a texture slice map on the corresponding integrated image plane;

[0141] S3-3, mapping the texture slice image on each integrated image surface to the micro-image array plane point by point through the mapping relationship between voxels and pixels of the same name, thereby generating a sub-micro-image array corresponding to each integrated image surface;

[0142] S3-4. Based on the relationship between parallax (in pixels) and depth, the sub-micro-image array corresponding to each integrated image surface is extended in the direction of the hole (to eliminate the image quality degradation caused by the hole), and the extended sub-micro-image arrays are fused to generate a complete micro-image array.

[0143] The mapping relationship between the voxels and the pixels of the same name is:

[0144]

[0145] Where Vx represents a voxel set, HomoPx represents a pixel set, and F represents a mapping function. (For any element in the voxel set Vx, the number of corresponding elements in the pixel set HomoPx represents the number of reconstruction rays that constitute that voxel. Therefore, regions of interest in a 3D scene can be reconstructed on voxels with a larger number of reconstruction rays, resulting in denser viewing angles and smoother motion parallax.)

[0146] Limited by the pixel resolution of 2D display screens, the total amount of information that can be displayed by integrated imaging 3D displays is still very limited, which makes the performance indicators of 3D images such as spatial resolution, 3D depth of field, and parallax smoothness mutually restricted. Therefore, in practical applications, it is necessary to comprehensively consider the above performance indicators to select the optimal voxel space to reconstruct 3D images. Figure 7 shows a schematic diagram of voxel space cropping and the preferred solution based on display performance, in which each integrated image surface is cropped to the same size as the 2D display screen and the part of the integrated image surface with uneven voxel distribution is discarded, and the middle part (the darker part in the middle of Figure 7) represents the voxel space after optimization based on display performance, that is, the optimal voxel space.

[0147] The flow chart of the integrated imaging micro-image array generation method based on the optimal voxel spatial distribution in this example is shown in FIG8 .

[0148] In order to verify the effectiveness of the integrated imaging micro-image array generation method based on the optimal voxel spatial distribution proposed in this example, the corresponding 3D acquisition and 3D display devices are now built.

[0149] The 3D acquisition device consists of a depth camera and a computer. The depth camera used in this example is the Intel RealSense D435, a consumer-grade depth camera from Intel. It is powerful, compact, and lightweight, and is widely used in machine navigation, object recognition, and human-computer interaction.

[0150] The depth camera primarily consists of two infrared cameras, an RGB camera, and an infrared dot projector. The RealSense D435 uses a binocular vision solution for depth measurement. Images captured by the left and right infrared cameras are fed into a built-in depth processing module, which calculates the depth value for each pixel based on triangulation. The infrared dot projector projects an infrared dot pattern onto the target scene to improve depth calculation accuracy in scenes with fewer feature points, such as white walls. The RGB camera captures texture images of the target scene in real time. Due to the distance between the infrared and RGB cameras, the fields of view of the texture and depth images differ, necessitating origin alignment to ensure their consistency. The RealSense D435 supports dynamic depth measurement and can output an RGB video stream with a maximum resolution of 1920 × 1080 and a depth video stream with a maximum resolution of 1280 × 720. The depth video stream is similar to the RGB video stream, except that each pixel is represented by the depth distance between the camera and the target, rather than an RGB grayscale value. RealSense D435 supports depth measurement within 0.2 m to 10 m and can be used for real-time acquisition of RGB-D images in multiple scenarios, including indoor and outdoor scenes.

[0151] The hardware configuration and parameter specifications of the 3D acquisition device are shown in Table 1.

[0152] Table 1 Hardware configuration and parameter specifications of the integrated imaging 3D acquisition device built in this example

[0153]

[0154] In the experiment, the depth camera was connected to a computer via a USB cable. Using the RealSense development kit and OpenCV library functions, the corresponding code was written to read the real-time color and depth video streams from the depth camera and capture simultaneous frames. To prevent mismatching caused by asynchrony between the color and depth streams, the color and depth frames are typically aligned. Directly acquired depth images often contain black holes caused by computational errors, which can be repaired by calling the corresponding hole-filling function. The repaired depth data frame was normalized so that its depth values ​​corresponded to the grayscale range of 0–255. Finally, the depth data frame and the corresponding color frame were saved as images to obtain a pair of RGB-D images. Figure 9 shows the pair of RGB-D images obtained by the depth camera in this example. Both images have a resolution of 1280 × 720. Among them, two dinosaur toys placed front and back in a staggered manner represent the 3D target being photographed. The texture map shown in Figure 9 (a) represents the surface texture of the 3D target. The depth map shown in Figure 9 (b) represents the distance between each target point and the depth camera. The lower the grayscale value, the closer the distance between the target point and the camera.

[0155] To verify the applicability of the proposed method to different depth data acquisition methods, this example also used 3DS MAX software to create a virtual 3D scene. Two cartoon characters, "Mario" and "Luigi," represented the 3D objects being photographed. A pair of RGB-D images with a resolution of 3840 × 2160 were rendered using 3DS MAX's virtual camera, as shown in Figure 10. Figure 10(a) shows the texture map, representing the surface texture of the 3D object; Figure 10(b) shows the depth map, indicating the distance between each object point and the depth camera. Higher grayscale values ​​indicate closer distances to the camera.

[0156] The 3D display device consists of an integrated imaging 3D display and a high-definition digital camera. The 3D display is a smartphone with a 3840 × 2160 resolution LCD screen and a microlens array precisely coupled together. The high-definition digital camera records 3D images from different viewing angles. Table 2 lists the hardware configuration and detailed parameters of the 3D display device.

[0157] Table 2 Hardware configuration and detailed parameters of the integrated imaging 3D display device built in this example

[0158]

[0159] When the distance g between the microlens array and the 2D display is 4 mm, g>f. At this time, the integrated imaging 3D display operates in real mode. The depth value and number of voxels of each integrated image plane are calculated according to the voxel space characteristic formula, as shown in Figure 11(a). The 3D display has a total of 29 integrated image planes, among which there are missing voxels on the integrated image planes P(1)-P(14) and should be discarded. The voxels on the integrated image planes P(28) and P(29) are too few, and their corresponding depth values ​​are multiplied compared with the previous integrated image plane, causing the voxels located there to diffuse sharply, which will seriously reduce the contrast of the reconstructed 3D image. Therefore, these two integrated image planes also need to be discarded.

[0160] From equations (1-11), it can be calculated that the depth value of the central depth plane is approximately 18.85 mm, which is located between the integrated image planes P(23) and P(24). After discarding the above integrated image planes, the depth values ​​of P(15)-P(27) are between 8 mm and 30 mm, roughly symmetrically distributed on both sides of the central depth plane. They can be regarded as the primary optimal voxel space considering the depth value and the number of voxels, as shown in the dotted box in Figure 11(a).

[0161] On the basis of the above-mentioned preferred voxel space, a secondary optimization is performed considering the influence of factors such as voxel overlap and the number of reconstruction rays. The voxel space distribution at this time is shown in Figure 11(b), where the stacked bar graph represents the voxel distribution on each integrated image surface, and each block in the stacked bar represents the number of voxels with a specific number of reconstruction rays, which are distinguished by different grayscales. Taking the integrated image surface P(20) as an example, there are 1280 voxels at this depth, of which 1260 voxels contain 3 reconstruction rays, and the other 20 voxels contain only 2 reconstruction rays. As the number of the integrated image surface increases, the proportion of voxels with more reconstruction rays increases significantly, making the motion parallax of the reconstructed 3D image smoother.

[0162] The curve in Figure 11(b) represents the overlap of adjacent voxels on each integrated image plane. It can be seen that the voxel overlap on the integrated image plane P(24) closest to the central depth plane is the lowest, while on the two edge integrated image planes P(15) and P(27), the voxel overlap is roughly equal and is within the range that the human eye can distinguish. Therefore, the voxels on these integrated image planes can all be retained. The final optimized voxel space range in this example is P(15)-P(27), which is the optimal voxel space.

[0163] In order to verify the spatial resolution of the 3D image reconstructed on each integrated image plane, this example uses the U.S. Air Force USAF 1951 resolution test target for testing. The USAF 1951 test target is usually used to test the resolution capability of an optical system. Its test pattern consists of multiple groups of horizontally and vertically arranged black and white line pairs, as shown in Figure 12, where the vertical line pairs are used to test the horizontal resolution, while the horizontal line pairs are used to test the vertical resolution. The USAF 1951 test target uses the number of resolvable line pairs per millimeter to measure the resolution of the optical system, with the unit being lp / mm. Different resolution test units are marked with group numbers and element numbers. Each element has a specific line length and line width, and the element size gradually decreases from the periphery to the center. The smallest resolvable element on the resolution test target is the resolution limit of the optical system under test. The resolution corresponding to each element can be calculated using the following formula:

[0164]

[0165] In the experiment, first, based on the USAF 1951 resolution test target image and the depth value of each integrated image surface, the micro-image array generation method proposed in this example is used to sequentially generate sub-micro-image arrays corresponding to the integrated image surfaces P(15)-P(27), and they are displayed one by one on the integrated imaging 3D display. The resolution target image reconstructed on each integrated image surface is recorded and reconstructed using a camera, as shown in Figure 13. For the convenience of discussion, only the images reconstructed on some integrated image surfaces are shown here. Figure 13 (a) is the resolution target image reconstructed on the integrated image surface P(16), Figure 13 (b) is the resolution target image reconstructed on the integrated image surface P(21), Figure 13 (c) is the resolution target image reconstructed on the integrated image surface P(24), and Figure 13 (d) is the resolution target image reconstructed on the integrated image surface P(27).

[0166] The line width corresponding to the minimum resolvable element in the reconstructed image represents the actual voxel size of the image. This is compared with the theoretical voxel size calculated using equation (1-12), as shown in Figure 14. It can be seen that, except for the voxels on the integrated image plane P (13) with the largest depth value, which have a large human eye tolerance error due to excessive diffusion, the measured voxel sizes on the remaining integrated image planes are in very good agreement with the theoretical values. This experiment demonstrates that the voxel spatial distribution model proposed in this example is applicable.

[0167] In order to verify the difference in the image quality of high-frequency and low-frequency images reconstructed by the integrated imaging 3D display, the USAF 1951 resolution test target image was replaced with a real image containing more high-frequency and low-frequency image information at the same time, and the above experiment was repeated. The image quality comparison of high-frequency and low-frequency images reconstructed on different integrated image planes is shown in Figure 15. Among them, Figure 15 (a) is a high-frequency and low-frequency image quality comparison on the integrated image plane P (16), Figure 15 (b) is a high-frequency and low-frequency image quality comparison on the integrated image plane P (21), Figure 15 (c) is a high-frequency and low-frequency image quality comparison on the integrated image plane P (24), and Figure 15 (d) is a high-frequency and low-frequency image quality comparison on the integrated image plane P (27). It can be seen from the reconstructed images shown in Figure 15 that on the integrated image planes closer to the central depth plane (such as P (21) and P (24)), more high-frequency image information can be distinguished, such as the two groups of Chinese characters "technology" and "people" in the picture. On the integrated image planes that are far from the central depth plane (such as P(16) and P(27)), the reconstructed image gradually becomes blurred, and only a few low-frequency image contents (such as the English characters "UX" in Figure 15) can be clearly distinguished. This experiment proves that for integrated imaging 3D displays with specific structural parameters, high-frequency images have higher requirements on the 3D reconstruction capabilities of the display system than low-frequency images. In practical applications, the reconstruction quality of each integrated imaging 3D display for images of different frequencies can be qualitatively or quantitatively tested through experimental methods, so that in the process of generating the micro-image array, the most appropriate voxel space can be flexibly selected according to the richness of the high-frequency information in the 3D image to be reconstructed, so as to optimize the image quality of the reconstructed 3D image.

[0168] The above experiments verified the voxel features on each integrated image surface. Next, the two sets of 3D scenes shown in Figures 9 and 10 were used to verify the reconstruction effect of the integrated imaging 3D display on the complete 3D image. First, the integrated image surface P(15)-P(27) was used as the optimal voxel space to generate and display the micro-image arrays of virtual and real 3D scenes respectively. The reconstructed images of different perspectives were recorded by a camera from the left, middle and right directions, as shown in Figure 16. In order to clearly show the motion parallax between images from different perspectives, a transparent thin ruler was placed on the surface of the microlens array. From the partial enlargement of Figure 16, it can be seen that when the viewpoint moves from left to right, the feature points in the 3D image, such as the "M" logo on the hat of the cartoon character "Mario" in the virtual scene and the eyes of the green dinosaur in the real scene, will move to the left accordingly. This shows that the reconstructed 3D image has a stereoscopic depth protruding from the screen and continuous motion parallax. At the same time, the image details at each perspective can be clearly distinguished, and there are almost no obvious cracks. The experimental results show that the micro-image array with optimal voxel spatial distribution generated by the proposed method can achieve high-performance 3D image reconstruction.

[0169] The above experiments were all conducted on a real-mode integrated imaging 3D display. When the distance g between the microlens array and the pixel plane is 2 mm, g

Claims

1. An integrated imaging micro-image array generation method based on the optimal voxel spatial distribution, characterized in that The method includes the steps of: S1. Obtain the depth data and texture data of the 3D scene; S2. Select the optimal voxel space: According to the relationship between the voxel space of the integral imaging 3D display and the 3D display performance, select a part of the voxel space that meets the 3D display performance requirements from the entire voxel space of the integral imaging 3D display as the optimal voxel space; S3. Synthesize the micro-image array: Using the obtained depth data and texture data of the 3D scene as inputs, synthesize a micro-image array that meets the display performance requirements of the integral imaging 3D display according to the selected optimal voxel space; The steps of selecting the optimal voxel space include: S2-1. Discard the integral image planes with voxel missing from the entire voxel space of the integral imaging 3D display; S2-2. Coarsely select the integral image planes with relatively high voxel space resolution according to the position of the central depth plane; S2-3. On the basis of step S2-2, select a part of the voxel space that meets the performance requirements as the optimal voxel space according to the 3D depth of field requirements of the integral imaging 3D display and the coincidence degree between adjacent voxels; The steps of synthesizing the micro-image array include: S3-1. Transform the obtained depth data of the 3D scene into the depth range corresponding to the optimal voxel space, and divide the obtained depth data of the 3D scene into multiple integral image planes corresponding to the optimal voxel space; S3-2. Divide the obtained texture data of the 3D scene and paste it onto each integral image plane, and adjust the spatial resolution of the texture data of the 3D scene on each integral image plane to the voxel space resolution of the corresponding integral image plane by upsampling or downsampling, that is, generate the texture slice map on the corresponding integral image plane; S3-3. Through the mapping relationship between the voxel and the corresponding point pixel, map the texture slice map on each integral image plane point by point onto the micro-image array plane to generate the sub-micro-image array corresponding to each integral image plane; S3-4. According to the relationship between the parallax and the depth, extend the sub-micro-image array corresponding to each integral image plane in the direction of the hole, and fuse the extended sub-micro-image arrays to generate a complete micro-image array.

2. The method for generating an integrated imaging micro-image array based on the optimal voxel spatial distribution according to claim 1, wherein The steps of selecting the optimal voxel space include: S2-1. Discard the integral image planes with voxel missing from the entire voxel space of the integral imaging 3D display, specifically including: A world coordinate system is established with the geometric center of the micro-lens array plane as the origin, where the X-Y plane coincides with the micro-lens array plane and the Z-axis is perpendicular to the micro-lens array plane. Assume that each image element contains a pixel, each image element will emit Reconstructed rays, and the reconstructed rays propagate in the 3D image space at the same angular interval, and the angular interval between adjacent reconstructed rays is: where g is the distance between the pixel plane and the microlens array plane Is the pixel size on the 2D display screen; The reconstructed rays emitted from a pixel in any image element intersect pairwise with each reconstructed ray from its adjacent image elements, forming at most individual voxels, and the voxels are distributed in At different depth positions, so the total number of integral image planes carrying all voxels is: Let the k-th ( ) The integrated image plane is , the distance to the micro-lens array plane Expressed as: Where, p represents the pitch of the lens element; Spacing between adjacent integrated image planes Is: Set And Respectively represent the integrated image plane The adjacent voxel intervals in the horizontal and vertical directions, then there is: The set of voxels formed by the pairwise intersection of the reconstructed rays of adjacent two image elements is distributed in a pyramid shape, and there are some regions with uneven distribution due to the lack of some voxels. The number of integral image planes with non-uniform voxel distribution is obtained through geometric relationship as: Where, round(.) represents rounding to the nearest integer; Crop the format of the integral image plane to the same size as the 2D display screen, and the number of voxels on the cropped integral image plane is expressed as: Among them, Indicates the number of voxels in the horizontal direction on the integrated image surface after cropping, represents the number of voxels in the vertical direction on the integrated image plane after cropping, W represents the width of the 2D display screen, and H represents the height of the 2D display screen; Equations (1-1) - (1-8) describe all the voxel characteristic parameters of the integral imaging 3D display; obviously, the reverse extension lines of the reconstructed light rays will also form a similar voxel space behind the screen, and its characteristic parameters have the same expressions as those of the voxel space in front of the screen; S2-2. Coarsely select the integrated image plane with relatively high voxel space resolution according to the position of the central depth plane, specifically including: Set and respectively represent the integrated image plane The spatial resolution of the voxels on in the horizontal and vertical directions is expressed as: It can be seen from Equations (1-9) and (1-10) that the voxel space resolution decreases with the increase of the integrated image plane number, that is, the voxel space resolution on the integrated image plane closer to the microlens array plane is higher, while the voxel space resolution on the integrated image plane farther from the microlens array plane gradually decreases; It can be seen from Equations (1-5), (1-7) and (1-8) that when the parameters of the 2D display screen and the lens element are determined, the number of voxels on each integrated image plane is determined, and the voxel size on each integrated image plane is related to the position of the central depth plane; The position of the central depth plane is obtained by the Gaussian imaging formula: where, l represents the distance from the central depth plane to the microlens array plane, and f is the focal length of the lens element; Therefore, by adjusting the distance g between the microlens array plane and the pixel plane, the central depth plane is set near the integrated image plane with relatively dense voxels, and then the integrated image plane with relatively high voxel space resolution is coarsely selected according to the position of the central depth plane; S2-3. On the basis of step S2-2, select a part of the voxel space that meets the performance requirements as the optimal voxel space according to the 3D depth of field requirements of the integral imaging 3D display and the coincidence degree between adjacent voxels, specifically including: According to the geometric relationship, at a distance from the plane of the microlens array On the integrated image plane, the size of the voxel It is expressed as: When adjacent voxels overlap, the degree of overlap It is expressed as: Among them, Indicates the interval between adjacent voxels, and its value is given by and Given; given overlap threshold , the depth positions of the front and rear edge integrated image planes that determine the 3D depth of field range of the integral imaging 3D display can be determined; the edge integrated image plane closest to the viewer is called the front edge integrated image plane, and the edge integrated image plane farthest from the viewer is called the rear edge integrated image plane; assuming that the distances between the front and rear edge integrated image planes and the microlens array plane are respectively And Then, the 3D depth of field of the integral imaging 3D display is expressed as: Therefore, on the basis of step S2-2, select the front and rear edge integrated image planes according to the 3D depth of field requirements of the integral imaging 3D display and the coincidence degree between adjacent voxels, so that the 3D image reconstructed between the front and rear edge integrated image planes always has relatively high clarity.

3. The integrated imaging micro-image array generation method based on the optimal voxel spatial distribution according to claim 1, wherein The mapping relationship between the voxel and the corresponding point pixel is: where, Vx represents the voxel set, HomoPx represents the pixel set, and F represents the mapping function.

4. The method for generating an integrated imaging micro-image array based on the optimal voxel spatial distribution according to claim 1, wherein The methods for obtaining the depth data and texture data of the 3D scene include: using a depth camera or 3D modeling software to obtain the RGB-D image of the 3D scene, and the RGB-D image contains depth data and texture data.

Images