A method for generating image array for integrated imaging display based on local depth template matching

Abstract

The present invention discloses a method for generating an integrated imaging display image array based on local depth template matching using a two-dimensional color image and a depth image of an original scene, which mainly includes three parts: local depth template construction, template matching, and global template application. The present invention utilizes the spatial geometric relationship between a microlens array and a display image, and repeatedly iterates and compares in the original light field to find the problem of corresponding mapping points of pixels in the display image array, and converts it into a local depth template matching problem of each pixel in a micro-unit image in the mapping range of the display light field and the original light field. To further optimize the computational efficiency, the present invention proposes an innovative idea of ​​batch calculating the same points of the micro-image array using each point in the template, and global depth matching can be completed by template displacement. The method used in the present invention increases the generation speed by an average of more than 72% compared to the comparative method.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional image processing, in particular to the field of integrated imaging display content generation, and in particular provides a method for generating an integrated imaging display image array based on local depth template matching. Background Art

[0002] Integrated imaging light field display technology has attracted extensive attention from domestic and foreign scholars and enterprises due to its simple system structure, no need to wear auxiliary equipment, and full-color, continuous viewing angle display images. Integrated imaging light field display image generation is a key link in the system and is particularly important for the stereoscopic display of three-dimensional scenes. At present, generating display image arrays based on the original light field color map and its registered depth map has become a mainstream generation method, but there are problems such as large amount of calculation and long generation time. Based on this, the present invention "a method for generating integrated imaging display image arrays based on local depth template matching" is proposed. Summary of the invention

[0003] The purpose of the present invention is to provide an integrated imaging display image array generation method based on local depth template matching, firstly establish a "one-to-one" mapping relationship between the display light field and the original light field to achieve efficient mapping without redundancy, and then based on the spatial geometric relationship between the microlens array and the display image, repeatedly iteratively compare in the original light field to find the problem of corresponding mapping points of pixels in the display image array, and convert it into a local depth template matching problem of each pixel in the micro-unit image in the display light field and the original light field mapping range, so as to achieve accurate depth information matching. Effectively solve the problem of large amount of calculation and long generation time.

[0004] The technical solution of the present invention is as follows:

[0005] The first step is to establish the parallel relationship between the lens arrays: the light field coordinate system is as follows Figure 1 As shown, the Y axis is omitted, and the zero point of the scene depth Z is set at the optical center of the lens. Assuming that the lens array is composed of M×N square lenses with a width of P and is closely arranged under ideal conditions, a display image array composed of M×N micro-unit images is generated accordingly, as shown in the attached figure. Figure 1 As shown in EIA. Assuming that the resolution of each micro-unit image is i×j, the pixel position relationship between a point A(i,j) in the i-th row and j-th column of the micro-unit image in the M-th row and N-th column of the display image array and a point B(i',j') in the i-th row and j-th column of the micro-unit image in the M'th row and N'th column is as follows:

[0006]

[0007] round(P / P D) is the number of pixels in each row or column in a single micro-unit image. The round function indicates rounding to the nearest integer. The abs function indicates taking the absolute value. D is the display pixel diameter of the display device. According to the principle of linear propagation of light paths, it can be deduced that the intersection of A(i,j) and B(i',j') with the same depth plane in the original light field should also have the above spatial mathematical relationship. The specific relationship will be given in the subsequent technical steps.

[0008] The second step is to establish the reverse mapping relationship from the three-dimensional light field to the original light field: According to the principle of reversibility of the optical path, the "one-to-many" mapping relationship from the original light field to the display image in the traditional method is converted into a "one-to-one" mapping relationship from the display image to the original light field. Light is emitted from a point A(i,j) in the display image array, passes through the optical center of the lens L(x0,0) and enters the original light field, intersecting with a depth plane Z in the original light field at point A(x,Z), then this point is the mapping point of A(i,j). The following relationship is satisfied:

[0009]

[0010] Among them, P I is the pixel diameter of the two-dimensional image in the original light field, which can be obtained by the following formula:

[0011]

[0012] d c is the central depth of the integrated imaging system, calculated according to formula (4):

[0013]

[0014] Where f is the focal length of the lens and g is the distance between the lens array and the display image array.

[0015] The third step is to establish the search range of "one-to-one": Since Z in equation (2) is unknown, it is impossible to directly find x, so it is necessary to build a search model for the mapping point. Figure 1 As shown, taking a lens as the master lens, according to the given display system parameters, the maximum display depth plane Z of the display system is calculated. max and minimum display depth plane Z min , a ray of light is emitted from the pixel point A(i,j) in the i-th row and j-th column of the master lens, passes through the optical center of the lens, and intersects with the maximum display depth plane Z max and minimum display depth plane Z min The intersection points A(x max ,Z max )、A(x min ,Z min ), then the line segment between the two points is the search range of the mapping point A(x,Z).

[0016] The fourth step is to establish a depth matrix template for the search range: According to image processing knowledge, pixel positions can only be rounded to integers, as shown in the following figure. Figure 1 As shown by the thick dashed line, in the present invention, the pixel coordinates are calculated in [x max ,x min ] range. Then, according to the geometric relationship of three-dimensional space, it is deduced that when the slope is fixed, if the pixel position change step is determined, the depth change step step z It can be expressed as:

[0017]

[0018] Then the first depth range matrix Z can be established. max、 x min Only displayed at the maximum display depth plane Z max and minimum display depth plane Z min The x at the time does not represent the size of the specific value. According to the coordinate pair matrix composed of the pixel position changes, a second depth range matrix H with the same shape as Z can be obtained in the two-dimensional depth map of the original light field (the depth data conversion step is omitted here and is completed by default).

[0019] Step 5: Confirm the unique mapping point: Compare the same positions of matrix Z and matrix H, and find the position where the difference is greater than or equal to 0. If there are multiple positions, select the position close to the observer and extract the corresponding pixel position, which is the unique mapping point A' of pixel A.

[0020] Step 6: Translate the depth matrix template: as shown in the attached Figure 1 As shown, the intersection of a point A(i, j) in the i-th row and j-th column of a micro-unit image in the display image array and a point B(i', j') in the i-th row and j-th column of the M-row and N-column micro-unit image at the maximum display depth plane and the minimum display depth plane has the following relationship:

[0021]

[0022] According to the above formula, the search range depth matrix of the pixels in the i-th row and j-th column in all micro-unit images can be calculated to obtain the corresponding mapping points.

[0023] Similarly, the depth matrix of other pixels in the master lens is calculated, and then translated according to equation (6) to obtain the mapping points of all pixels in the display image array.

[0024] The beneficial effect of the present invention is: assuming that the average depth of each point in the original scene is Z A, the number of pixels of the original scene is U, the number of pixels of the generated display image array is V, and the number of pixels of the micro-unit image is u. Then the time complexity of the comparison algorithm is O(U). The method proposed in the present invention only consumes more time in finding the depth template mapping range of a micro-unit image. The other micro-unit images only need to translate the template, and the time complexity is The calculation time is related to the pixel sum of the display image array, especially the pixel sum of the micro-unit image, and has nothing to do with the complexity of the original scene, which greatly improves the calculation speed.

[0025] In the process of searching for mapping points, the present invention derives an effective change step length of the depth within a determined search range according to the spatial relationship, and the search step length becomes larger, thereby saving search time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram showing the generation of an array of display images in the present invention.

[0027] Figure 2 The two-dimensional color image and depth image of the original scene used in the specific implementation of the present invention are

[0028] Figure 3 In the specific implementation of the present invention, the display image array generated

[0029] Figure 4 This is the actual three-dimensional display effect diagram produced by different viewing angles in the specific implementation of the present invention.

[0030] The comparison method is the method disclosed in the invention patent "Piao Yan, Gu Yuejianan, & Wang Yu. (2019). A method for rapid generation of element image arrays of an integrated imaging system. CN110418125A." DETAILED DESCRIPTION

[0031] The following is a detailed description of the implementation method of "a method for generating an integrated imaging display image array based on local depth template matching" of the present invention in conjunction with the accompanying drawings.

[0032] In the present invention, the two-dimensional color image and depth image of the original scene used are as shown in the attached figure. Figure 2 The integrated imaging display system consists of a lens array consisting of 100×100 square lenses with a focal length of f=3mm and a width of P=1mm, which are closely arranged. The distance from the lens array to the display image array is g=3.5mm, and the display pixel size is P D =0.06mm, Redmi Note 11 mobile phone with a resolution of 1920×1080.

[0033] Based on the above parameters, the center depth and depth range of the system can be calculated:

[0034] According to formula (4),

[0035] Depth range

[0036] Then the maximum display depth surface and the minimum display depth surface are

[0037]

[0038] In this example we use the Figure 2 The color map and depth map are used as the information of the original scene.

[0039] In this example, the lens in the first row and first column of the lens array is used as the master lens. The micro-unit display image corresponding to each lens can be calculated by the given parameters to contain 17×17 pixels. To facilitate calculation, the search range of the micro-unit display image is set to [-8,8]. Then, according to formula (2), the intersection of the pixel point in the first row and first column of the master lens and the maximum display depth plane and the minimum display depth plane is obtained:

[0040]

[0041] The depth change step can be calculated according to formula (5) for the x range passed, and the matrices Z and H can be determined. The two matrices are compared to find the mapping point. The depth data conversion in the depth map is omitted here and is completed by default.

[0042] By translating the range matrices Z and H obtained by equation (8) according to equation (6) and then comparing them, the mapping points of the first row and first column pixels in all lens arrays can be obtained. Similarly, the global mapping point search can be completed.

[0043] Attached Figure 3 In the specific implementation of the present invention, the display image array generated is attached Figure 4 It is an actual stereoscopic display effect diagram of different viewing angles produced by the display image array, which can clearly show the left and right viewing angle changes of the original scene.

[0044] The generation speed of the display image array under different original scenes, different micro-unit sizes, different lens arrays and different configurations is compared and analyzed with the comparison method. The evaluation results are shown in Table 1:

[0045] Table 1 Comparison of the rate of generating display image arrays by two methods

[0046]

[0047]

[0048] The data in the table are all achieved on the PC side using Python 3.9. They are all test results using buffer screen writing in single-threaded mode. If multi-threaded parallel mode is used, the overall speed can be accelerated.

[0049] Matters not covered by the present invention are known technologies.

[0050] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating an integrated imaging display image array based on local depth template matching, characterized in that: The method comprises the following steps: S1, establish the parallel relationship between lens arrays: Assuming that the lens array is closely arranged in an ideal state, the specific spatial position relationship of the pixel points in the micro-unit image corresponding to the same position of adjacent lenses can be derived according to the principle of linear propagation of light paths, which is specifically expressed as follows: i'=i+abs((M-M'))×round(P / P D ) j'=j+abs((N-N'))×round(P / P D ) (1) Among them, the round function represents rounding, the abs function represents taking the absolute value, P is the physical width of the lens, and P D is the display pixel diameter of the display device, M and N are the number of rows and columns of the lens array respectively, i and j are the pixel position coordinates in the micro-unit image corresponding to a certain lens, and i' and j' represent the pixel coordinates at the same position as i and j in the micro-unit image corresponding to the adjacent lens; S2, establishing a reverse mapping relationship from the display image array to the original light field: based on the principle of reversibility of the optical path, establishing a "one-to-one" mapping relationship from the pixel points of the display image array to the pixel points of the original light field image; S3, establish a "one-to-one" search range: take the lens in the Mth row and the Nth column as the master lens, and according to the given display system parameters, including but not limited to the focal length f of the lens, the distance g between the lens array and the display image array, and the display pixel size P D , calculate the maximum depth plane and the minimum depth plane of the display system, emit light from the pixel point A in the mth row and nth column of the master lens, pass through the optical center of the lens, and intersect the maximum depth plane and the minimum depth plane at two points respectively. The line segment between the two points is the search range of the mapping point A' of the pixel point A; S4, establishing a search range depth array template: Since the pixel position can only be changed by integers, based on the three-dimensional space geometric relationship, a reasonable depth change step size is determined with the change of the pixel position when the slope is fixed, and a depth range array Z of the pixel point A in the display light field is established, and a depth range array H with the same shape as Z is obtained in the depth map of the original light field according to the pixel position change; S5, confirm the unique mapping point: compare and differ the same positions of the two matrices in S4, and find the position where the difference is greater than or equal to 0. If there are multiple positions, select the position close to the observer, and extract the corresponding pixel position as the unique mapping point A' of pixel point A; S6, translate the depth matrix template: the spatial position relationship of the pixels at the same position of adjacent lenses is derived from S1. Similarly, the mapping points of the pixels at the same position of adjacent lenses also have a corresponding parallel relationship. According to the determined parallel formula, the depth matrix template obtained by translating S4 can obtain the unique mapping point of the pixel at the mth row and nth column of all lenses; S7, and so on, calculate the depth matrix of other pixel points of the master lens, and then translate them to generate a complete display image array.

Images