A reconstruction method of fourier hologram based on sparse sampling

Abstract

The present application relates to a kind of based on sparse collection's Fourier hologram generation reconstruction method, belong to Fourier hologram generation reconstruction method.Integrated imaging method based on sparse collection is used to generate the stereoscopic meta image of object;According to the stereoscopic meta image of object, multi-view projection incoherent Fourier calculation hologram generation and simulation reconstruction are carried out.The present application has the advantages that compared with the traditional image recognition gray integral projection method, the block matching integral projection method based on SAD search method is used, the parallax error rate obtained by calculation is reduced from 3.125% to 0.2%, the structural similarity SSIM of the generated stereoscopic meta image and the original object view is increased from 0.71 to 0.75, the accuracy of the generated stereoscopic meta image is improved, so that the object Fourier hologram storage information and the reproduced image are more accurate, the block matching integral projection method carries out image correction when running code for block matching, and only searches for matching block in horizontal direction, compared with other algorithms, the steps are more simple.

Description

Technical Field

[0001] This invention relates to a Fourier hologram generation and reconstruction method that uses the principle of sparse acquisition integrated imaging to generate stereo images, and particularly to a Fourier hologram generation and reconstruction method based on sparse acquisition. Background Technology

[0002] Holograms can effectively record information such as the size, depth, brightness, and shape of an object, and efficiently preserve this information for data storage and reproduction. Compared to single-angle imaging, using microlens arrays or camera arrays to capture images from multiple angles can obtain a greater degree of overall object information, thus enabling better simulation and reconstruction. Fourier holograms generated based on the Fourier algorithm are characterized by tear resistance, simple reconstruction algorithms, and small errors, and are therefore often used to generate high-quality digital holograms. However, existing error-free Fourier hologram generation techniques typically involve reading a scene from a single two-dimensional image of the object and then performing multi-view simulation numerical calculations. This results in a single image acquisition perspective, which prevents the hologram from preserving the overall object's appearance to a great extent, leading to unsatisfactory reconstruction quality.

[0003] To address this issue, some scholars have attempted to find solutions using ensemble imaging technology. Ensemble imaging technology can generate stereoscopic images that preserve the overall appearance of an object more accurately for hologram generation. However, traditional ensemble imaging techniques require significant human and material resources to deploy large camera arrays for image capture and rendering, resulting in long rendering times and frequent discrepancies between the generated images and the original object scene. To address this, a sparse acquisition stereoscopic image array mapping system can be established to achieve sparse acquisition. Currently, ensemble imaging technology based on the principle of sparse acquisition can acquire stereoscopic images of objects with fewer cameras, significantly improving image generation efficiency and reducing errors. During sparse acquisition, the disparity values ​​between projected images from different viewpoints need to be calculated. These disparity values ​​determine the parameters for cropping multi-view projected images. The calculated disparity values ​​usually contain errors compared to standard disparity values, directly affecting the quality of the generated stereoscopic images. A higher disparity error rate results in a greater difference between the generated stereoscopic image and the actual object. The imaging quality of the stereoscopic image can be measured using the Structural Similarity Index (SSIM). The different methods used in the parallax calculation process can lead to a large error rate, which in turn affects the quality of the generated stereo image and the accuracy of the hologram in preserving object scene information.

[0004] Many studies have proposed different methods for disparity calculation to try to reduce the disparity error rate. Some have proposed obtaining the disparity value by calculating the integral projection of the entire object in adjacent viewpoint images. However, this method ignores the occlusion problem between different objects, resulting in a large disparity error rate. Others have calculated the mean square error function of each pixel in the left viewpoint image relative to the right viewpoint image, and used the one-dimensional window translation distance at the minimum value of the mean square error function as the disparity value of the corresponding point. At the same time, a threshold is set to determine the reliability of the obtained disparity value due to errors in disparity value calculation caused by factors such as occlusion and mismatch in the disparity map. However, if the threshold range is set too large, it will lead to a decrease in the accuracy of disparity calculation. To address the occlusion problem caused by different parts of the object, some have proposed a color segmentation algorithm based on HSV space to segment the object into different color parts and calculate them separately to solve the occlusion problem. Summary of the Invention

[0005] This invention provides a Fourier hologram generation and reconstruction method based on sparse acquisition, with the aim of reducing the parallax error rate and improving the quality of the generated stereo image.

[0006] The technical solution adopted by this invention includes the following steps:

[0007] (i) Generate stereo image of an object using an integrated imaging method based on sparse acquisition;

[0008] (II) Generation and simulation reconstruction of holograms based on multi-view projection incoherent Fourier calculations of the object's stereoscopic image.

[0009] The method for generating the object's 3D element image in step (one) of this invention is as follows:

[0010] (1) Obtain the multi-view projection view array of the object

[0011] In computer simulation software, a virtual camera array is used to photograph objects, resulting in several projected views of the objects from multiple perspectives. These views are then arranged according to the relative positions of the cameras to form a view array.

[0012] Using the virtual computer modeling software 3DS Max, create an object scene, build a virtual camera array, perform preliminary multi-view acquisition and rendering of the objects to obtain a projected view, complete the work of building the acquisition platform and rendering the object projection view array, create an object scene and build a camera array, establish a virtual target scene in the virtual computer modeling software 3DS Max and place a combination of target objects at a specified location, and use the software's built-in free camera to create an n*n equidistant camera array in 3DS MAX software. This camera array has a total of n*n cameras, and the distance between any two adjacent cameras is equal.

[0013] (2) Calculating disparity based on SAD-based block matching integral projection

[0014] A segmentation algorithm is used to segment the projected view to obtain different shaped parts of the object. Then, a block matching integral projection method based on the SAD (Sum of absolute differences) method is used to obtain the pixel coordinates of each shaped part of the object. The disparity value of each shaped part is calculated based on the pixel coordinates of the object. After discarding invalid disparity values, the average of the valid disparity values ​​of each shaped part is taken as the final disparity value. The quality of the final disparity values ​​obtained by different disparity calculation methods is evaluated. A standardized object is set to obtain the standard disparity. The disparity error and the disparity error rate are calculated based on the relationship between the final disparity value and the standard disparity value.

[0015] Because occlusion between different shaped parts of an object can compromise the accuracy of parallax calculations, and directly calculating the parallax of the entire object can lead to significant errors, a segmentation algorithm is first used to segment the object into different shaped parts before performing the subsequent parallax calculations.

[0016] After segmenting the object into its various shapes, disparity is calculated. Here, a block matching integral projection method based on the SAD search method is used to analyze the pixels. During the process, block comparisons are performed, and the process of finding matching blocks uses the sum of the absolute values ​​of the differences. After block matching, the grayscale images of various target images are processed by integral projection in the horizontal and vertical directions. The total number and distribution of target pixels in each row and column in the horizontal and vertical directions are counted, which can intuitively display the specific position of the target object in the corresponding sub-image. After obtaining the pixel position coordinates of each shape of the object, the disparity value of each shape can be calculated based on the pixel position coordinates of the object. After discarding invalid disparity values, the average of the valid disparity values ​​of each shape is taken as the final disparity value.

[0017] After obtaining the final disparity value, the advantages and disadvantages of each disparity calculation method are judged. A standardized object is set up to calculate the standard disparity. The disparity error and the disparity error rate are calculated based on the relationship between the final disparity value and the standard disparity value. The advantages and disadvantages are analyzed by comparison.

[0018] (3) Based on the parallax, the images are cropped. Using MATLAB software, each view in the original multi-view object view array is cropped according to the formula based on the final parallax value to obtain the cropped sub-image images. The numerous sub-image images constitute the sub-image array.

[0019] (4) After obtaining the cropped sub-image array, the cropped images are stitched together in order according to the relative position of the camera in the camera array to generate the cropped sub-image array.

[0020] (5) Mapping to generate stereo image: The position of the pixels of the cropped sub-image array obtained by stitching is transformed according to the mapping relationship to reconstruct the stereo image of the object. The stereo image that would have required more cameras to be captured is generated with fewer cameras, realizing sparse acquisition integrated imaging. After obtaining the stereo image, the SSIM (Structural Similarity) index is introduced to measure the similarity between the stereo image generated by imaging and the original object view, and to evaluate the quality of the stereo image generated after calculating the disparity using different disparity calculation methods.

[0021] The generated stereo image based on the sparse acquisition integrated imaging principle can efficiently and conveniently save relevant information about objects.

[0022] In step (ii) of this invention, the process of generating and simulating the Fourier hologram of the object is as follows:

[0023] (1) Generate an array of orthographic projection views of objects

[0024] The object's 3D element image is read as the scene. Then, the loop interval is set, and the shooting process of the equidistant camera array is simulated by numerical calculation in MATLAB software. The acquired image information is orthogonalized according to the orthogonal projection formula to generate orthogonal view information. The obtained orthogonal projection view information is arranged according to the relative position relationship to obtain the orthogonal projection view array information.

[0025] (2) After multiplying the orthogonal projection view with the plane wave, the integral is used to generate a Fourier hologram. Then, to generate a high-precision Fourier hologram, the information of each orthogonal projection view is multiplied by the corresponding plane wave and the product is integrated to obtain a complex value in the Fourier hologram. The complex values ​​calculated from the orthogonal projection view information at different angles and positions are arranged in relative position order to generate the Fourier hologram of the object. This method uses a coordinate system transformation method to connect the complex field of the three-dimensional object on the Fourier plane of the Fourier transform lens with the three-dimensional coordinate system of the object. It can be proven to be accurate and without error.

[0026] (3) The image was reconstructed based on the Fourier hologram simulation. Finally, the Fourier hologram was reconstructed based on the formula to obtain the reconstructed image of the object.

[0027] The beneficial effects of this invention are:

[0028] This invention addresses the need to reduce disparity error rates by replacing direct grayscale integral projection with a block matching integral projection method based on the SAD (Sum of Absolute Differences) search approach in the disparity calculation step of generating stereoscopic images. This reduces the disparity error rate and improves the quality of the generated stereoscopic images. This method corrects the image during the search for matching blocks; features from the left image are placed in the same pixel row as those in the right image, meaning that matching blocks are searched only in the horizontal direction, thus reducing workload.

[0029] This invention employs an ensemble imaging method based on sparse acquisition to generate stereoscopic images of objects, thus solving the problem that holograms cannot preserve the overall appearance of objects to a large extent due to a single image acquisition perspective. Ensemble imaging technology can acquire stereoscopic images that preserve the overall appearance of objects to a greater extent for hologram generation, and using ensemble imaging technology based on the principle of sparse acquisition allows for the acquisition of stereoscopic images with fewer cameras.

[0030] This invention combines an integrated imaging method based on the sparse acquisition principle with a Fourier hologram calculation, generation, and reconstruction method. Using the sparse acquisition principle-based integrated imaging method, a sub-image array is obtained by calculating the parallax of a multi-view projection view array of an object, cropping and stitching the images together. The pixels of this sub-image array are then rearranged according to mapping relationships to generate a stereo image of the object. Next, the scene is read based on the stereo image, and a camera array is simulated in software using numerical simulation to capture the scene, obtaining perspective projection view array information. This information is then orthogonalized to generate a multi-view orthogonal projection view array. Each orthogonal projection view is multiplied with a plane wave and then integrated to generate each pixel in the Fourier hologram. Each pixel is then arranged sequentially according to the relative positions of the simulated cameras used to generate the projection views, thus generating the Fourier hologram of the object. This Fourier hologram can then be used for simulation reconstruction to reproduce the original object image. In the disparity assessment step of generating stereo images, a block matching integral projection method based on the SAD (Sum of absolute differences) search approach is used instead of the gray-scale integral projection method. This reduces the disparity error rate and improves the quality of the generated stereo images. Simultaneously, two metrics, disparity error rate and structural similarity (SSIM), are introduced to evaluate the imaging quality of the stereo images.

[0031] The feature of this invention is that after obtaining the various shape parts of an object using a segmentation algorithm, a block matching integral projection method based on the SAD (Sum of absolute differences) search method is used to correct the image and then search for matching blocks in the horizontal direction. After that, integral projection is performed to calculate the disparity value, and the final disparity value is obtained. This disparity value is used to crop the multi-view image array and stitch it together to map it into a stereo image of the object for the generation and reconstruction of Fourier holograms.

[0032] This invention employs a block-matching integral projection method based on the SAD (Sum of Absolute Differences) search approach. Compared to the traditional image recognition grayscale integral projection method, this reduces the disparity error rate from 3.125% to 0.2%, and increases the structural similarity (SSIM) between the generated stereo image and the original object view from 0.71 to 0.75, thus improving the accuracy of the generated stereo image. This results in more accurate storage of information and reconstruction of the object's Fourier hologram. Furthermore, this block-matching integral projection method performs image correction during block matching, searching for matching blocks only in the horizontal direction, making the process simpler compared to other algorithms. Attached Figure Description

[0033] Figure 1 This is a flowchart of the present invention;

[0034] Figure 2 This is a diagram showing the relative positions of the cameras;

[0035] Figure 3 This is a scene image after the setup is complete;

[0036] Figure 4 It is an array of multi-view projection views of an object;

[0037] Figure 5 This is the result of the view segmentation from camera number 5;

[0038] Figure 6 It is the horizontal integral projection diagram and the vertical integral projection diagram of the entire object;

[0039] Figure 7 These are the horizontal and vertical integral projection diagrams of the small ball portion;

[0040] Figure 8 These are the horizontal and vertical integral projection diagrams of the annular portion;

[0041] Figure 9 These are the horizontal and vertical integral projection diagrams of the white sphere.

[0042] Figure 10 It is a cropped sub-image array;

[0043] Figure 11 It is a sub-image array diagram;

[0044] Figure 12 It is a stereo image generated by calculating disparity based on the SAD (Short Aspect Divergence) method, which uses block matching integral projection to calculate disparity.

[0045] Figure 13 It is a stereo image generated by calculating disparity using the gray-scale integral projection method;

[0046] Figure 14 It is an amplitude Fourier hologram;

[0047] Figure 15 It is a phase Fourier hologram;

[0048] Figure 16 This is the reconstructed image. Detailed Implementation

[0049] Includes the following steps:

[0050] (i) Generate stereo image of an object using an integrated imaging method based on sparse acquisition;

[0051] (1) Obtain the multi-view projection view array of the object

[0052] This step uses a virtual camera array in computer simulation software to photograph the object, obtaining several projected views of the object from multiple perspectives. These views are then arranged according to the relative positions of the cameras to form a multi-view projection view array.

[0053] The virtual computer modeling software 3DS Max can be used to create object scenes, build virtual camera arrays, perform preliminary multi-view acquisition of objects, and render projected views. This completes the work of building the acquisition platform and rendering the object projection view array. First, the object scene and camera array must be created. In 3DS Max, a virtual target scene is established, and a combination of target objects is placed at designated locations. Using the software's built-in free camera, an n*n equidistant camera array is created in 3DS Max. This camera array contains n*n cameras, with equal distances between adjacent cameras.

[0054] To illustrate the above process, create an object in 3DS MAX software. This object consists of a small ball and a ring, with the ring surrounding the ball. Then, create a camera array. The camera can be the built-in camera in the software, and the camera parameters can be freely set. Here, we choose the free camera built into 3DS MAX software, with the following parameter settings: lens 43.456mm, field of view 45.0 degrees, close range 0.0cm, and far range 2540.0cm.

[0055] For simplicity, the example provided uses a 3x3 camera array, with a 5cm spacing between every two adjacent cameras. A diagram illustrating the relative positions of the cameras in the array is shown below. Figure 2 As shown.

[0056] This completes the work of creating the scene and setting up the camera array. Once the setup is complete, the scene in 3DS MAX software will look like this: Figure 3 As shown.

[0057] Figure 3 The image on the left is the object, and the image on the right is a camera array with adjacent cameras arranged at equal intervals. The virtual cameras used above can render images, and their imaging principle is as follows:

[0058] If we define g as the object distance of the virtual camera's lens, f j Let h be the focal length and object distance, respectively; L be the widest range of the target scene; c be the length of the image on the virtual recording medium; and θ be the viewpoint from the camera's focal length. Then, we can obtain the following equations (1) and (2) using the Gaussian formula:

[0059]

[0060]

[0061] Set the maximum range of the target scene to be placed to L. max Given the maximum observation range as T, n as the number of cameras in each row and column, and p as the distance between cameras, we can obtain the following formulas (3) and (4):

[0062]

[0063]

[0064] Based on the imaging principle described above, each camera can generate views of objects from different directions according to its own perspective. Then, the V-RAY 5 rendering plugin is used to set the size and format of the output image to render the view of each camera, resulting in a total of n*n rendered output images. Each image contains all the objects as a whole. These images have the same content, only different perspectives. After rendering, a multi-view projection view array of the objects is obtained. The images formed by the virtual cameras in 3DS MAX software can be rendered and output using the V-RAY 5 rendering plugin to generate images whose size and format can be freely set.

[0065] In the example given, the rendered output image size is set to 640 pixels * 640 pixels, and the format is JPG. In this case, n = 3. After rendering with the V-RAY 5 plugin, a total of 9 grayscale views (3x3) of the object's projection, each 640 pixels * 640 pixels, are obtained at different angles. Figure 4 As shown.

[0066] (2) Based on SAD, find the method block matching integral projection to calculate disparity.

[0067] A segmentation algorithm is used to segment the projected view to obtain different shaped parts of the object. Then, a block matching integral projection method based on the SAD (Sum of Absolute Differences) method is used to calculate the pixel coordinates of each shaped part of the object. Based on the pixel coordinates, the disparity value of each shaped part can be calculated. After discarding invalid disparity values, the average of the valid disparity values ​​of each shaped part is calculated as the final disparity value. A standardized object is set to calculate the standard disparity value. Based on the relationship between the final disparity value and the standard disparity value, the disparity error and the disparity error rate are calculated.

[0068] Because occlusion between different parts of an object can compromise the accuracy of disparity calculations, and directly calculating disparity for the entire object can lead to significant errors, a segmentation algorithm can be used to first segment the object into its different shapes before performing the subsequent disparity calculations. The example given uses a threshold-based segmentation method, and the segmentation process is performed in MATLAB software.

[0069] After reading the image to be processed in MATLAB, the image is converted to HSV space. Then, a black image is created. The `ind2sub` function is used to extract specific shapes from the image to the black image by setting a threshold. The extracted shapes are then converted to RGB space. Different shapes have significantly different pixel values ​​in HSV space due to color differences. Within a single shape, pixel values ​​are very similar in HSV space due to their similar colors. Therefore, various specific shapes can be extracted by setting a threshold to define a parameter range. When extracting a specific shape, the parameters for both S and V spaces in the `ind2sub` function are set to positive numbers. The key to segmentation lies in setting the parameters for H space; defining different ranges allows for the extraction of different shape parts.

[0070] In this example, when extracting the ball part, the parameters in H space are set to 0.4-0.6, and when extracting the ring part, the parameters in H space are set to 0.1-0.3. The projection view obtained in the previous step is segmented to obtain different shaped parts of the object in an image. Figure 5 The grayscale images of the projected view output by camera number 5 in the middle position are obtained by color segmentation.

[0071] After resolving the occlusion problem by segmenting the object into its various shapes, the next step is to calculate disparity. A block-matching integral projection method based on the Sum of Absolute Differences (SAD) is used to analyze pixels. During the process, block comparisons are performed, and the SAD method is used to find matching blocks. After block matching, horizontal and vertical integral projection processing is applied to the grayscale images of the various target images. The total number and distribution of target pixels in each row and column in the horizontal and vertical directions are statistically analyzed, visually displaying the specific position of the target object in the corresponding sub-image. After obtaining the pixel coordinates of each shape of the object, the disparity value of each shape can be calculated based on the pixel coordinates. Invalid disparity values ​​are discarded, and the average of the valid disparity values ​​of each shape is taken as the final disparity value. The principle of the block-matching integral projection method based on the SAD is as follows:

[0072] The first step is to match the same feature location between two adjacent images. Here, we choose to divide the image into blocks for matching the same feature location. Since block comparison cannot directly process RGB data, the two adjacent images must be converted from RGB data space to grayscale image space before searching for matching blocks. The grayscale value range is from 0 to 255. After that, the images are corrected. The features of the left image will be placed on the same pixel row of the right image, that is, matching blocks are searched only in the horizontal direction, which reduces the workload. For the search area and direction, the search mode and range of the block matching algorithm on the template need to be set. The basis for determining the range is the disparity we want to find in corresponding images with different shapes. During the block matching process, the shape on the image boundary needs to be considered. The usual approach is to crop the template to the maximum value.

[0073] Next, to obtain the coordinates of the target pixel, during sub-pixel estimation, the block matching algorithm yields the integer disparity value corresponding to the image, which is generally the offset value of pixels with the same feature point in neighboring images. Then, an attempt is made to fine-tune the disparity value to the sub-pixel position by interpolating between the nearest matching block and adjacent pixels. At this point, only the minimum cost and the cost of two adjacent targets are considered, and these are used to fit and output the corresponding parabola. By analyzing the minimum point of the corresponding parabola, the coordinates of the target pixel can be obtained. That is, if the horizontal axis of the parabola is specified as the horizontal coordinate of the pixel, then the vertical coordinate is the minimum cost and the cost of two adjacent targets, and the extreme points of the three parabolas are the points with the minimum cost. Let C... p C is the pixel offset value. g F1 is the subpixel estimate of the actual disparity, F2 is the output value of the nearest matching block, F1 is the output value of one image, and F3 is the output value of another image. The relationship between them is shown in Equation (5):

[0074] C g =C p -(F3-F1) / (2*(F1-2F2+F3)) (5)

[0075] Once the coordinates of the corresponding pixels in two adjacent images are obtained, the disparity value can be obtained by calculating the difference between the corresponding pixel coordinates.

[0076] The disparity calculation for integral projection can be performed using MATLAB software. First, two adjacent images are read in MATLAB, then combined to obtain a composite image. Disparity is then calculated by estimating sub-pixel block matching. The block matching process begins by converting the RGB image of the target object to a grayscale image by setting an average of three color channel values. Next, a disparity range is defined, specifying the pixel distance from the block position in the first image to search for matching blocks in other images. For the 640*640 pixel image used in the example, a disparity range of 50 is suitable. Then, the block size for block matching is defined. A loop structure is then used to set minimum and maximum block boundaries for each row and column of pixels in the image as templates and blocks. The template position is defined as the search boundary, limiting the search to prevent it from exceeding the image boundaries. The rightmost image block is selected as the template. After obtaining the number of image blocks searched, a vector is created to store the block bias. Finally, the bias between the template and each block is calculated. Since the rightmost image block is chosen as the template, the distance from the leftmost image block is used. The 1-based index of the block at position "w" is calculated and placed into a vector containing the block offset. Here, "w" is a loop counter parameter, ranging from the maximum number of pixels that can be searched to the left to the maximum number of pixels that can be searched to the right. The value of "w" is incremented by 1 after each loop. The sum of the absolute values ​​of the differences between the template and the blocks (SAD) is calculated as the result. Then, the SAD values ​​are sorted to find the nearest match (minimum offset). The sub-pixel position of the best matching position is estimated, which yields an overall disparity comparison map between two adjacent images. The horizontal and vertical integral projection processes are performed on each image. The total number and distribution of target pixels in each row and column in the horizontal and vertical directions are statistically analyzed, which can intuitively display the specific position of the target object in the corresponding sub-image, resulting in a horizontal and vertical projection comparison map between two adjacent images. After obtaining the position of the corresponding object shape in the image by integral projection of the different shape parts, the disparity value of each shape part is obtained by subtracting the position coordinates of the corresponding object shape parts in adjacent images. Since the imaging uses an equidistant camera array, the horizontal and vertical disparities of each shape part of the object should be the same. If the calculated horizontal and vertical disparities are not equal, they should be discarded as invalid disparity values. The results with equal horizontal and vertical disparities are retained as the valid disparity values ​​of that shape part. Finally, the average of the valid disparity values ​​of each shape part is calculated to obtain the final disparity value.

[0077] In this example, block matching is performed on each shape and the object as a whole to calculate the disparity, resulting in a disparity comparison map and the positions of the start and end pixels for any two adjacent images. Taking the images output by a pair of adjacent cameras, camera 1 and camera 2, as an example... Figure 6 It is the horizontal integral projection diagram and the vertical integral projection diagram of the entire object. Figure 7These are the horizontal and vertical integral projection diagrams of the small ball portion. Figure 8 These are the horizontal integral projection diagram and the vertical integral projection diagram of the annular portion.

[0078] The position of the corresponding object shape in the image is obtained by integral projection of the different shape parts of the object. The difference between this position and the position coordinates of the corresponding object shape in the adjacent image is used to obtain the final disparity value of 40.08 (pixels). If no segmentation is performed, the final disparity value of the whole is calculated using the same method, which is 39.83 (pixels).

[0079] When evaluating the merits of different disparity calculation methods, a standardized object can be used to calculate the standard disparity value. Based on the relationship between the standard disparity value and the final disparity value, the disparity error and disparity error rate can be calculated and compared to analyze their respective merits. The principle of disparity error rate calculation is as follows:

[0080] The final disparity value is recorded as M0. The disparity error rate E is defined as the ratio of the absolute value of the difference between the final disparity value M0 and the standard disparity value M to the standard disparity value M. Its formula is as follows (6):

[0081]

[0082] In the example given, to obtain the standard disparity value, we replace the target object with a small white sphere with a radius of 7.394 cm, keeping all other conditions unchanged, and repeat the same steps. We use a camera array in the virtual simulation software 3DS MAX to photograph the white sphere, rendering nine projected views of the same size (640 pixels x 640 pixels). Then, we calculate the disparity based on these projected images. Again, using the example of cameras 1 and 2, the integrated projection images of the horizontal and vertical directions of the images output by this pair of adjacent cameras are shown below. Figure 9 As shown.

[0083] The position of the white ball in the image is obtained by calculating its integral projection. The difference between this position and the corresponding coordinates of the white ball in adjacent images yields a standard disparity value of 40 pixels. It can be observed that after segmenting the object, the disparity error rate obtained by calculating the disparity of each shape is 0.2%, while the disparity error rate obtained by calculating the disparity of the entire object without segmentation is 0.4925%. Segmenting the object effectively reduces the disparity error caused by occlusion.

[0084] The disparity error rate obtained by the proposed SAD-based integral projection disparity calculation method is 0.2% based on the final disparity value and the standard disparity value. Using the traditional image recognition grayscale integral projection method to calculate the disparity of the object projection array obtained in the previous step, the final disparity value is 38.75 (pixels), and the standard disparity value remains 40 (pixels), resulting in a disparity error rate of 3.125%. The comparison shows that the disparity error rate calculated by the former method is smaller.

[0085] (3) Extract the image based on the final parallax.

[0086] Based on the calculated final disparity value, each view in the multi-view projection view array of the rendered object is cropped to obtain a cropped sub-image. The principle is as follows:

[0087] The final disparity value is M0. Using a rectangular window of size B (pixels) * D (pixels), each view in the original multi-view object projection view array is cropped to obtain individual cropped sub-images. First, the position (x, y) of the camera in the first row and first column of the cropped sub-image is determined. (1，1) y (1，1) Then, based on the final disparity value obtained in the previous step, the rectangular window is translated along the horizontal and vertical directions to sequentially crop each view, resulting in a cropped sub-image array. Let the coordinates of the upper left corner of the rectangular window in the first row and first column of the camera image be (x...). (1，1) y (1，1) The coordinates of the top-left corner of the rectangular window in the image captured by the camera in row a0 and column b0 are... The two are related by the following formulas (7) and (8), where the values ​​of a0 and b0 are 1≤a0≤n and 1≤b0≤n.

[0088]

[0089]

[0090] In the example given, the `imcrop` function in MATLAB is used to set a rectangular window for image cropping based on parallax. To ensure that the target object is centered in the sub-image captured by the 5th camera (located in the center), the top-left corner coordinates of the cropping rectangular window (row 2, column 2, i.e., the image rendered by the 5th camera in the center) are set to (210, 210), and a 220*220 pixel rectangular window is used to crop each view. Since the final parallax value calculated in the previous step is 40.08 pixels, images from different cameras need to be cropped every 40.08 pixels from left to right and top to bottom. Finally, nine sub-images of size 220*220 pixels are cropped from nine 640*640 pixel projected views. The grayscale images of these sub-images are as follows: Figure 10 As shown.

[0091] (4) The sub-image array is obtained by splicing the images;

[0092] The cropped sub-images are reassembled and stitched together according to their relative positions when the original images were taken by the camera, resulting in a stitched sub-image array.

[0093] In the example given, the reshape function in MATLAB software is used to stitch together a total of 9 sub-images (3*3) according to the relative positions of the cameras to form a sub-image array.

[0094] When processing images in MATLAB, converting RGB color images to grayscale using the `rgb2gray` function before further processing is a common method. Grayscale images are more convenient than color images. A grayscale image is an image without color, represented in MATLAB as a two-dimensional matrix array. An RGB color image consists of red, green, and blue components, represented in MATLAB as a three-dimensional matrix array. Processing a grayscale image requires considering all three color dimensions simultaneously, making it more complex than processing a two-dimensional matrix array of a grayscale image.

[0095] Therefore, the stitching yields a grayscale image of the sub-image array, as shown below. Figure 11 As shown.

[0096] (5) Mapping to generate a stereo image.

[0097] The pixel positions of the stitched sub-image array are transformed according to the mapping relationship to reconstruct a two-dimensional stereo image. This generates a stereo image that would otherwise require more cameras with fewer cameras, thus achieving sparse acquisition integrated imaging.

[0098] To achieve sparse acquisition, an inverse mapping method is used to obtain the stereo image from the sub-image array. The sub-image array, composed of e*r sub-images of size i (pixels)*q (pixels), is subjected to inverse mapping. Pixels at the same position in each sub-image are extracted and combined according to their corresponding positions in the sub-image array to form the mapped unit image, i.e., the stereo image. The correspondence between pixels in the images before and after mapping is shown in equation (9):

[0099]

[0100] In the formula, the pixel position in the image before mapping is f. q The pixel position in the mapped image is f h x and y are the pixel coordinates before mapping, x = 0, 1, 2, ... (e*i - 1), y = 0, 1, 2, ... (r*q - 1), and % is the modulo operation. It is a floor operation, so the mapped image consists of i*q mapped units, and the size of each mapped unit image is e (pixels)*r (pixels).

[0101] In the example given, MATLAB software is used for mapping processing. First, a zero matrix of the same size as the sub-image array is created to store the translated pixel information later. Then, the sub-image array is converted into a grayscale image, and pixels with the same coordinates in each sub-image are extracted. At this time, given e=3, r=3, i=220, q=220, any coordinate can be calculated using equation (9) to obtain its new position coordinate in the new matrix. The pixel information in the original grayscale image is then assigned to the corresponding position in the new matrix according to the new position coordinate. Using a loop statement, rows are processed first and then columns. After all the pixels in the sub-image array are processed, the stereo image is obtained. The stereo image is as follows: Figure 12 As shown; the disparity value obtained by calculating the disparity using the gray-scale integral projection method is repeated in steps 3, 4, and 5, and the resulting stereo image is shown. Figure 13 As shown.

[0102] This completes the process of generating a stereo image of the object using sparse acquisition and integrated imaging. Since a 220*220 pixel rectangle was used for cropping, and the object's multi-view projection array is generated and output using a 3*3 virtual camera array in the 3DS MAX virtual simulation software, the above process captured the content that would normally require a 220*220 camera array using only a 3*3 camera array. This achieves sparse acquisition and more conveniently and accurately reproduces the object's information.

[0103] After obtaining the stereo image, the SSIM (Structural Similarity) index can be introduced to measure the similarity between the generated stereo image and the object view, thereby evaluating the quality of the generation method. In MATLAB software, the ssim function can be used to compare the structural similarity of two images and obtain the SSIM index value. The closer this value is to 1, the more similar the two images are.

[0104] In the example given, the SSIM index of the stereo image obtained by calculating disparity using block matching integral projection based on SAD is 0.75, while the SSIM index of the stereo image obtained by calculating disparity using grayscale integral projection is 0.71. Therefore, the stereo image generated by the former method has better imaging quality.

[0105] The generated stereo image based on the sparse acquisition integrated imaging principle can efficiently and conveniently save relevant information about objects.

[0106] (II) Generation and simulation reconstruction of holograms based on multi-view projection incoherent Fourier calculations of the object's stereoscopic image.

[0107] (1) Generate an array of orthogonal projection views of objects.

[0108] An error-free Fourier hologram of the object is generated using an incoherent multi-view orthogonal projection method based on the generated stereoscopic image of the object. First, an array of orthogonal projection views of the object needs to be generated. After reading the stereoscopic image, numerical simulation is used to obtain orthogonal projection views of the object at different angles, i.e., the array of orthogonal projection views of the object. The principle is as follows:

[0109] Set the projection angle to θ x and θ y The target point coordinates of the object are (x, y, z). The perspective projection view in the object's spatial coordinate system is orthogonalized. The coordinates (x, y, z) in the projection view are... p ,y p The relationship between the coordinates (x, y, z) in the object space and the coordinates in the object space can be expressed by equations (10) and (11):

[0110] x p =x + ztanθ x =x + zs / l (10)

[0111] y p =y+ztanθ y =y+zt / l (11)

[0112] From the above formula, we can know the projection angle θ x and θ yIt can be replaced by s, t, and l, where s and t are variable parameters and l is a fixed value. After the above changes, an orthographic projection view can be obtained for further processing.

[0113] MATLAB software can be used to generate an array of orthogonal projection views of an object. MATLAB's workflow involves matrix processing. The image is read using the `imread` function to generate a matrix containing relevant information, followed by matrix operations to complete the image processing. This simplifies the process. After reading the stereo image as the scene, the steps of generating the object's projection views are simulated by setting parameters and calculating numerical values. The information from each viewpoint is then stored in a matrix for further processing, eliminating the need to generate individual images, thus simplifying the process. After generating the object's projection view array, a loop is used in MATLAB to orthogonalize the projection view matrices for different values ​​of `s` and `t`, according to the formula. This results in an orthogonal projection view matrix, which stores the information of the orthogonal projection view at that location. Performing this operation on each projection view yields multiple matrices storing the orthogonal projection view array, completing the generation of the orthogonal projection view array.

[0114] In the example given, MATLAB software is used to generate the orthogonal projection view array. This step employs a numerical simulation method. First, a stereo image is read into MATLAB as the scene. Then, the viewing angle range and the cyclic interval step size of s and t are set to simulate the process of equally spaced cameras acquiring projected views of an object from different perspectives. The viewing angle range is used to determine the maximum and minimum values ​​of s and t, and the cyclic interval step size of s and t is used to determine the distance between the simulated cameras. In this example, the viewing angle range is set to 0.001m, and the cyclic interval step size of s and t is 0.00001m. This represents the process of using numerical calculation to simulate an equally spaced camera array with adjacent cameras spaced 0.00001m apart within a range of 0.001(m) * 0.001(m) to acquire a multi-view projection view array of an object. It is easy to see that this simulated camera array has 201*201 cameras. In MATLAB software, for ease of processing, the matrix containing image information can be directly orthogonalized. Based on the orthogonalization principle formula above, the projection view information matrix is ​​orthogonalized to obtain the orthogonal projection view information of the object. The information contained in each view is saved as a matrix in MATLAB. All the matrices obtained after orthogonalization store the information of the orthogonal projection view array of the object, which is used for the next step of processing.

[0115] (2) Multiply the orthogonal projection view with the plane wave and then integrate to generate a Fourier hologram.

[0116] For each view in the orthogonal projection view array, a pixel in the Fourier hologram is generated by multiplying it with a plane wave and then integrating the result. The Fourier hologram is then generated by arranging these pixels sequentially according to their relative positions based on the acquired angles. The generation principle is as follows:

[0117] Each orthographic projection view is multiplied by the plane wave propagating in the corresponding viewing direction, and then double-integrated into a single complex value, as shown in equation (12) below:

[0118] H(s,t)=∫∫P s,t (x p ,y p )*exp[-2jπb(x p *s+y p *t)]dx p dy p (12)

[0119] A plane wave is the power-law part of the above equation: exp[-2jπb(x)] p *s+y p *t), while the orthographic projection view is P in the above formula. s,t (x p ,y p b is a constant.

[0120] Orthogonal projection point P s,t (x p ,y p ) can be replaced with the corresponding object point Therefore, by conducting With x and The substitution between y and y can transform the above equation (12) into the following equation (13):

[0121] H(s,t)=∫∫∫O(x,y,z)*exp[-2jπb{(x*s+y*t)+z*(s 2 +t 2 ) / l}]dxdydz (13)

[0122] If the scale coordinates on the Fourier hologram plane are represented as (u,v), the above equation (13) can be transformed into the following equation (14):

[0123] H(u,v)=∫∫∫O(x,y,z)*exp[-2jπb{(x*u+y*v)+z*(u 2 +v 2 ) / lM} / M]dxdydz (14)

[0124] Where M = u / s = v / t, M is a coordinate magnification factor. Therefore, the complex field g(u,v) of the object on the Fourier plane of the Fourier transform lens with focal length f can be given by the following equation (15):

[0125] g(u,v)=A∫∫∫O(x,y,z)*exp[-2jπ{(x*u+y*v)+z*(u 2 +v 2 ) / 2f} / λf]dxdydz (15)

[0126] Where A is a constant. From equations (14) and (15), we can see that if the following conditions are met... and The hologram generated by the method is a precise Fourier hologram.

[0127] An orthogonal projection view generates a complex value in the Fourier hologram according to the above formula (12), which constitutes a pixel in the Fourier hologram. By repeating this process for all orthogonal view images, we can obtain each pixel in the Fourier hologram. Then, by arranging each pixel in order according to the relative position of the original projection view, we can generate the Fourier hologram of the object. Since the value of each pixel is a complex value, the Fourier hologram contains complex information. Therefore, we can obtain the amplitude and phase of the Fourier hologram to generate two representations: amplitude Fourier hologram and phase Fourier hologram.

[0128] When using MATLAB to process an array of orthogonal projection views to generate Fourier holograms, since the information of each image in the array is in matrix form, the complex value of a pixel in the Fourier hologram can be obtained directly in MATLAB by using the orthogonal projection view information matrix according to equation (12). By using a loop statement to perform the above operation on the information matrix corresponding to each orthogonal projection view in sequence, the complex value of each pixel in the Fourier hologram is obtained. Arranging the complex values ​​in order according to their relative positions yields the Fourier hologram. The Fourier hologram generated in MATLAB exists in the form of a complex matrix. The amplitude Fourier hologram can be obtained by using the abs function to calculate the amplitude of the Fourier hologram matrix, and the phase Fourier hologram can be obtained by using the angle function to calculate the phase of the Fourier hologram matrix.

[0129] In the example given, MATLAB software is used to perform a cyclic operation of multiplication followed by integration on the orthogonal projection view information matrix to obtain the Fourier hologram complex matrix. Since the previous numerical simulation used a 201*201 camera array, resulting in an array of 201*201 orthogonal projection views, the Fourier hologram generated in this step is a complex matrix of size 201 (pixels) * 201 (pixels). Then, the amplitude of the Fourier hologram complex matrix is ​​calculated to obtain the amplitude Fourier hologram; the phase of the Fourier hologram complex matrix is ​​calculated to obtain the phase Fourier hologram. The amplitude Fourier hologram and phase Fourier hologram obtained from the stereo image are shown below. Figure 14 and Figure 15 As shown.

[0130] (3) Reconstruct and restore the image based on Fourier hologram simulation.

[0131] The Fourier hologram is reconstructed by setting the focal length f of the simulated Fourier lens, the reconstruction depth d, and the wavelength λ of the light wave, and then calculating and reconstructing the read stereoscopic image. The principle is as follows:

[0132] The Fourier hologram obtained after synthesis is reconstructed. After the reconstruction operation is performed using the following formula (16), the reconstructed image of the original object can be obtained from the Fourier hologram.

[0133]

[0134] When using MATLAB software to simulate and reconstruct an image from a Fourier hologram, the reconstructed image information matrix can be obtained by calculating the complex matrix of the Fourier hologram and the values ​​of each parameter according to the formula above. The reconstructed image can then be reproduced from the matrix information using the `imshow` function. It should be noted that the formula (16) above is for the complex field g(u,v) of the object on the Fourier plane of a Fourier transform lens with focal length f. A transformation from orthogonal projection based on the s, t coordinate system to the complex field based on the u, v coordinate system needs to be performed in MATLAB. Define the focal length. The values ​​of the cyclic interval step size of u and v are the products of the corresponding cyclic interval step size of s and t and M, respectively. Then, the u and v system was implemented to replace the s and t system, so that simulation and reconstruction could be performed according to the formula.

[0135] In the example given, MATLAB software is used to simulate and reconstruct the image from a hologram. By setting the values ​​of each parameter, the reconstructed image can be obtained through Fourier holographic complex matrix simulation. Here, red light with a wavelength λ of 700nm is used, the focal length f is 250nm, and the reconstruction depth d is 800nm. The resulting reconstructed image is shown below. Figure 16 As shown.

Claims

1. A method for generating and reconstructing Fourier holograms based on sparse acquisition, characterized in that, Includes the following steps: (i) Generate stereo image of an object using an integrated imaging method based on sparse acquisition; (1) Obtain the array of multi-view projection views of the object In computer simulation software, a virtual camera array is used to photograph objects, resulting in several projected views of the objects from multiple perspectives. These views are then arranged according to the relative positions of the cameras to form a view array. (2) Calculating disparity based on SAD-based block matching integral projection method (3) Based on the parallax, use MATLAB software to crop each view in the original multi-view object view array according to the formula based on the final parallax value to obtain the cropped sub-image images. The numerous sub-image images constitute the sub-image array. (4) After obtaining the cropped sub-image array, the cropped images are stitched together in order according to the relative position of the camera in the camera array to generate the cropped sub-image array. (5) Mapping to generate stereo image: The position of the pixels of the cropped sub-image array obtained by stitching is transformed according to the mapping relationship to reconstruct the stereo image of the object. This generates a stereo image that would otherwise require more cameras to capture with fewer cameras, thus achieving sparse acquisition integrated imaging. (ii) Generation and simulation reconstruction of holograms based on multi-view projection incoherent Fourier transform calculations of the object's stereoscopic image; (1) Generate an array of orthographic projection views of objects The object's 3D element image is read as the scene. Then, the loop interval is set, and the shooting process of the equidistant camera array is simulated by numerical calculation in MATLAB software. The acquired image information is orthogonalized according to the orthogonal projection formula to generate orthogonal view information. The obtained orthogonal projection view information is arranged according to the relative position relationship to obtain the orthogonal projection view array information. (2) After multiplying the orthogonal projection view with the plane wave, the integral is used to generate a Fourier hologram. Then, to generate a high-precision Fourier hologram, the information of each orthogonal projection view is multiplied by the corresponding plane wave and the product is integrated to obtain a complex value in the Fourier hologram. The complex values ​​calculated from the orthogonal projection view information at different angles and positions are arranged in relative position order to generate the Fourier hologram of the object. (3) Reconstruct the image based on the Fourier hologram simulation, and finally reconstruct the Fourier hologram according to the formula to obtain the reconstructed image of the object.

2. The Fourier hologram generation and reconstruction method based on sparse acquisition according to claim 1, characterized in that, Step (1) of obtaining the multi-view projection view array of the object further includes: Using the virtual computer modeling software 3DS Max, create an object scene, build a virtual camera array, perform preliminary multi-view acquisition and rendering of the objects to obtain projected views, complete the work of building the acquisition platform and rendering the object projection view array, create an object scene and build a camera array, establish a virtual target scene in the virtual computer modeling software 3DS Max and place the target object combination at the specified position, and use the software's built-in free camera to create an n*n equidistant camera array in 3DS Max software. This camera array has a total of n*n cameras, and the distance between any two adjacent cameras is equal.

3. The Fourier hologram generation and reconstruction method based on sparse acquisition according to claim 1, characterized in that, In step (a) of the above, (2) calculates the disparity based on the SAD method block matching integral projection as follows: The projection view is segmented using a segmentation algorithm to obtain different shaped parts of the object. Then, the block matching integral projection method based on the SAD search method is used to obtain the pixel coordinates of each shaped part of the object. The disparity value of each shaped part is calculated based on the pixel coordinates of the object. After discarding invalid disparity values, the average of the valid disparity values ​​of each shaped part is taken as the final disparity value. The quality of the final disparity values ​​obtained by different disparity calculation methods is evaluated. A standardized object is set to obtain the standard disparity. The disparity error and the disparity error rate are calculated based on the relationship between the final disparity value and the standard disparity value. First, a segmentation algorithm is used to segment the object into different shapes, and then the disparity is calculated. After segmenting the object into its various shapes, disparity is calculated. Here, a block matching integral projection method based on the SAD search method is used to analyze the pixels. During the process, block comparisons are performed, and the process of finding matching blocks uses the sum of the absolute values ​​of the differences. After block matching, the grayscale images of the various target images are processed by integral projection in the horizontal and vertical directions. The total number and distribution of target pixels in each row and column in the horizontal and vertical directions are counted, which can intuitively display the specific position of the target object in the corresponding sub-image. After obtaining the pixel position coordinates of each shape of the object, the disparity value of each shape is calculated based on the pixel position coordinates of the object. After discarding invalid disparity values, the average of the valid disparity values ​​of each shape is taken as the final disparity value. After obtaining the final disparity value, the advantages and disadvantages of each disparity calculation method are judged. A standardized object is set up to calculate the standard disparity. The disparity error and the disparity error rate are calculated based on the relationship between the final disparity value and the standard disparity value. The advantages and disadvantages are analyzed by comparison.

4. The Fourier hologram generation and reconstruction method based on sparse acquisition according to claim 1, characterized in that, Step (5) in step (a) further includes: after obtaining the stereo image, introducing the SSIM structural similarity index to measure the similarity between the stereo image generated by imaging and the original object view, and evaluating the quality of the stereo image generated after calculating the disparity using different disparity calculation methods.

Images