A holographic projection method, device and medium based on dynamic capture

Abstract

The application discloses a kind of holographic projection methods, equipment and medium based on dynamic capture, it is related to image data processing technical field, including: to monocular camera acquisition dynamic scene time series image is preprocessed and depth estimation, after generating the depth map corresponding to each frame image, three-dimensional point cloud set is obtained by executing fusion resampling;Three-dimensional point cloud set is mapped to spatial light modulator coordinate system, and square sub-holographic unit is divided according to physical spacing, convolution modeling is carried out to sub-holographic unit, after generating the holographic complex amplitude distribution of spatial light modulator plane, complex amplitude synthesis operation is executed, and initial complex amplitude function value is generated Fourier propagation to focal plane, and focal plane complex light field distribution is formed;The application significantly improves dynamic target three-dimensional holographic reconstruction imaging quality and engineering robustness.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a holographic projection method, device and medium based on dynamic capture. Background Technology

[0002] With the rapid development of 3D display technology, computer-generated holography, capable of completely recording the amplitude and phase information of light waves, is considered one of the most promising true 3D display solutions. In dynamic holographic projection applications, how to acquire the 3D geometric information of dynamic scenes in real time is a key preliminary step that restricts the quality of holographic reconstruction. Traditional 3D scene acquisition solutions mainly rely on binocular stereo vision or structured light-based RGB-D depth cameras (such as the Kinect series). Although these devices can provide dense depth maps, they suffer from problems such as complex hardware structure, high power consumption, poor adaptability to outdoor environments, and significant attenuation of depth accuracy in long-distance or large-field-of-view scenes. In recent years, monocular depth estimation technology based on deep learning has made breakthrough progress. Through convolutional neural networks (CNN) or visual transformer (ViT) architectures, it is possible to infer depth information with relative scale consistency from a single 2D RGB image. This provides a new technical path for simplifying dynamic capture hardware architecture and reducing system costs.

[0003] However, existing technologies have the following shortcomings: 1. It lacks an adaptive step size control mechanism for coherent interference characteristics, which makes it easy to get trapped in local minima or produce oscillatory divergence, making it difficult to achieve high-fidelity intensity matching and uniform speckle suppression simultaneously within a limited number of iterations. 2. The failure to establish a spatially varying point spread function response model resulted in the inability to accurately predict and correct edge diffraction artifacts and high-frequency information loss in the focal plane light field. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a holographic projection method, device, and medium based on dynamic capture, which solves the problems of existing technologies lacking an adaptive step size control mechanism for coherent interference characteristics and failing to establish a spatially varying point spread function response model.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a holographic projection method based on dynamic capture, comprising: The dynamic scene time-series images acquired by the monocular camera are preprocessed and depth estimated to generate a depth map corresponding to each frame. Then, fusion resampling is performed to obtain a three-dimensional point cloud set. The three-dimensional point cloud set is mapped to the spatial light modulator coordinate system, and square sub-holographic units are divided according to physical spacing. The sub-holographic units are convolved and modeled to generate the holographic complex amplitude distribution of the spatial light modulator plane. Then, the complex amplitude synthesis operation is performed to generate the initial complex amplitude function value. The initial complex amplitude function value is propagated to the focal plane through Fourier transform to form the complex light field distribution of the focal plane. Based on the focal plane complex optical field distribution, the holographic complex amplitude field is reconstructed in an iterative manner, and the final complex amplitude holographic field is output. A binary mask is constructed to form an aperture function. Combined with the final complex amplitude holographic field, a spatial variable point diffusion function response is constructed. Combined with sub-holographic units, the final complex optical field distribution is generated. Based on the final complex light field distribution, the phase angle is taken and mapped to a driving grayscale image, which is then loaded into a spatial light modulator to achieve three-dimensional holographic projection.

[0007] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the preprocessing includes denoising and normalization.

[0008] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the depth estimation involves inputting each frame of the standardized image sequence into a pre-trained deep learning depth estimation network and outputting a depth map corresponding to each frame of the standardized image.

[0009] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the step of mapping the three-dimensional point cloud set to the spatial light modulator coordinate system, dividing it into square sub-holographic units according to physical spacing, performing convolution modeling on the sub-holographic units, and generating the holographic complex amplitude distribution of the spatial light modulator plane includes: Based on a 3D point cloud set, the coordinates of all point cloud points are uniformly mapped to the coordinate system of the spatial light modulator (SLM). After obtaining the SLM plane coordinates of the point cloud points, the coordinate plane of the SLM is divided into several square sub-regions as sub-holographic units according to the number of point cloud points in the 3D point cloud set, and the physical spacing of the sub-holographic units is set. ; Traverse the 3D point cloud set and determine the sub-holographic unit to which the SLM plane coordinates of the point cloud point belong by using a rounding function based on the physical spacing of the sub-holographic units. After performing convolution operations on the sub-holographic units using preset sub-holographic basis functions, discrete spatial frequency modeling of the spatial light modulator plane is performed to generate a holographic complex amplitude distribution on the spatial light modulator plane.

[0010] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the step of performing complex amplitude synthesis operation to generate an initial complex amplitude function value is propagated to the focal plane via Fourier transform, forming a complex light field distribution on the focal plane, as follows: Based on a 3D point cloud set, the initial amplitude weight is calculated, and an initial phase free variable is generated for each point in the 3D point cloud set. Obtain the spatial position and phase of each point in the three-dimensional point cloud set at continuous spatial coordinates in the spatial light modulator plane; Based on the initial amplitude weights, initial phase free variables, and spatial position phase, a complex amplitude synthesis operation is performed on each point cloud point to generate an initial complex amplitude function value. The initial complex amplitude function value is then transformed to the focal plane through Fourier propagation to obtain the complex light field distribution function value of the focal plane.

[0011] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the step of reconstructing the holographic complex amplitude field based on the focal plane complex light field distribution in an iterative manner and then outputting the final complex amplitude holographic field includes: Based on the complex optical field distribution function value of the focal plane, optimize the amplitude weight and phase free variable of each point cloud point; After each optimization, the updated amplitude weights and phase free variables are substituted into the initial complex amplitude function value. After keeping the spatial position and phase unchanged, the holographic complex amplitude function value on the spatial light modulator plane of each iteration is reconstructed. Then, the holographic complex amplitude function value on the spatial light modulator plane is transformed to the focal plane to obtain the updated complex light field distribution function value. The normalized root mean square error of the intensity of all point cloud points is obtained as the convergence index. After the maximum number of iterations, the final spatial light modulator complex amplitude holographic field is output.

[0012] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the following steps are taken: constructing a binary mask to form an aperture function, combining it with the final complex amplitude holographic field, constructing a spatially variable point diffusion function response, and combining it with sub-holographic units to generate the final complex light field distribution, as detailed below: Based on the actual effective aperture region of the spatial light modulator, construct binary mask values; By mapping the binary mask values ​​back to the spatial light modulator plane, the aperture function values ​​of the spatial light modulator in continuous spatial coordinates are obtained. Based on the final spatial light modulator complex amplitude holographic field and aperture function value, a spatial variable point diffusion function is constructed, and the response value of the spatial variable point diffusion function is calculated. The sub-holographic unit is frequency domain transformed to generate the response function value in the frequency domain space. Combined with the response value of the spatially variable point spread function, a convolution operation is performed to generate the final complex optical field distribution function value.

[0013] As a preferred embodiment of the holographic projection method based on dynamic capture described in this invention, the step of mapping the phase angle based on the final complex light field distribution to a driving grayscale image and loading it onto a spatial light modulator to achieve three-dimensional holographic projection includes: Based on the final complex light field distribution function value, after performing the phase angle calculation, the pixel phase value required by the spatial light modulator is generated; The phase value is linearly mapped according to the phase modulation depth and gray level of the spatial light modulator to obtain the gray value, and a driving grayscale map is constructed. The driving grayscale image is encoded into an image format and transmitted to a spatial light modulator for incident light wave modulation. After the incident light wave is modulated, the laser source of the spatial light modulator is turned on and uniformly illuminated onto the surface of the spatial light modulator loaded with grayscale image. The reconstructed light field is formed through a Fourier lens and then received by a diffuse reflection screen.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the holographic projection method based on dynamic capture as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the holographic projection method based on dynamic capture as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: by smoothly updating the amplitude weights and synchronously optimizing the phase free variables, combined with random initial phase to break coherent interference fringes, speckle noise is effectively suppressed and oscillation divergence is avoided, achieving high-fidelity intensity matching within a limited number of iterations. Furthermore, by constructing a binary mask based on the actual effective aperture of the spatial light modulator and mapping it to a continuous aperture function, and by performing cascaded convolution in conjunction with the frequency domain response of the sub-holographic units, the spatially varying point spread function is accurately modeled, and edge diffraction artifacts and high-frequency information loss are pre-compensated. Therefore, this invention significantly improves the imaging quality and engineering robustness of dynamic target three-dimensional holographic reconstruction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a holographic projection method based on dynamic capture.

[0019] Figure 2 The flowchart shows the output of the final spatial light modulator complex amplitude holographic field.

[0020] Figure 3 A flowchart for achieving 3D holographic projection. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Reference Figures 1-3 This is one embodiment of the present invention, which provides a holographic projection method based on dynamic capture, including the following steps: S1. Preprocess and depth estimate the dynamic scene time-series images acquired by the monocular camera, generate the depth map corresponding to each frame image, and then perform fusion resampling to obtain a three-dimensional point cloud set.

[0025] Specifically, a monocular camera is used to continuously acquire visual data from dynamic target scenes, resulting in a time-series image sequence.

[0026] It should be noted that the monocular camera is preferably an industrial camera or a consumer-grade RGB camera, and its acquisition frame rate is set to a continuous output mode within a fixed range, such as between 20 and 60 frames per second.

[0027] Furthermore, each frame in the image sequence is denoised and standardized to obtain a standardized image sequence; Each frame of the standardized image sequence is input into a pre-trained deep learning depth estimation network, which outputs a depth map corresponding to each frame of the standardized image.

[0028] It should be noted that the depth estimation network can be a monocular depth estimation model based on a pre-trained convolutional neural network or a visual Transformer structure. The model uses publicly available datasets or pre-trained data to solidify its parameters. The input image needs to be adjusted to the same size as the network input layer, for example, scaled to a fixed aspect ratio according to the network requirements, and cropped or padded as necessary to ensure that the input tensor meets the model's structural constraints.

[0029] Furthermore, after aligning the depth map and the normalized image pixel by pixel, a fusion resampling is performed based on the aligned depth map and the normalized image to generate a 3D point cloud set.

[0030] The expression for generating a 3D point cloud set is: In the formula, Indicates at time The 3D point cloud set generated in time, and They represent the first The x and y coordinates of each point cloud point Indicates the first The depth value corresponding to each point cloud point. Indicates the first The light intensity value corresponding to each point in the point cloud. This indicates the number of points in the point cloud.

[0031] It should be noted that when generating a 3D point cloud through fusion resampling, the 2D coordinates of the image pixels are used as the basic coordinate system, and the depth value corresponding to the pixel is used as the longitudinal distance component in the camera coordinate system. Simultaneously, combined with camera intrinsic parameters, the pixel coordinates are converted into normalized imaging plane coordinates, and then the actual spatial coordinates of the point in 3D space are recovered by combining the depth value. For color information, the corresponding color or grayscale value of the pixel in the normalized image is directly taken as the light intensity attribute of the 3D point.

[0032] S2. Map the three-dimensional point cloud set to the spatial light modulator coordinate system, and divide it into square sub-holographic units according to the physical spacing. Perform convolution modeling on the sub-holographic units to generate the holographic complex amplitude distribution of the spatial light modulator plane. Then, perform complex amplitude synthesis operation to generate the initial complex amplitude function value, which is propagated to the focal plane through Fourier transform to form the complex light field distribution of the focal plane.

[0033] Specifically, based on the 3D point cloud set, the coordinates of all point cloud points are uniformly mapped to the coordinate system of the spatial light modulator (SLM). After obtaining the SLM plane coordinates of the point cloud points, the coordinate plane of the spatial light modulator is divided into several square sub-regions as sub-holographic units (Tile) according to the number of point cloud points in the 3D point cloud set, and the physical spacing of the sub-holographic units is set. .

[0034] Specifically, the coordinates of all point cloud points are uniformly mapped to the coordinate system of the spatial light modulator, as expressed by: In the formula, Indicates the first The coordinates of a point cloud point on the SLM plane This represents the spatial scale conversion factor, which can be set based on the ratio of the reconstructed spatial lateral extent to the physical size of the spatial light modulator.

[0035] It should be noted that the physical spacing of the sub-holographic units... The value can be set by dividing the physical size of the spatial light modulator by the total number of sub-holographic units in the corresponding direction. For example, if the physical length of the spatial light modulator in the horizontal direction is... The number of sub-holographic units divided horizontally is Then the physical spacing between the sub-holographic units in the horizontal direction is... Divide by The same calculation applies to the vertical direction.

[0036] Furthermore, the three-dimensional point cloud set is traversed, and the sub-holographic unit to which the SLM plane coordinates of the point cloud point belong is determined by a rounding function based on the physical spacing of the sub-holographic units. After performing convolution operations on the sub-holographic units using preset sub-holographic basis functions, discrete spatial frequency modeling of the spatial light modulator plane is performed to generate a holographic complex amplitude distribution on the spatial light modulator plane.

[0037] The sub-holographic unit to which the SLM plane coordinates of the point cloud points belong is determined by a rounding function, expressed as follows: In the formula, and They represent the first The row and column indices of the sub-holographic units corresponding to each point cloud point. This represents the rounding function; It should be noted that in actual implementation, if multiple point cloud points fall into the same sub-holographic unit index after rounding, these points are considered to belong to the same sub-holographic unit, and all points in that unit are superimposed. Discrete spatial frequency modeling of the spatial light modulator plane is performed, expressed as: In the formula, This represents the holographic complex amplitude function value on the spatial light modulator plane. Represents continuous spatial coordinates on the spatial light modulator plane. express The number of sub-holographic units in the direction, express The number of sub-holographic units in the direction, This represents a two-dimensional Dirac impulse function. This represents the convolution operation. Indicates the first The continuous spatial coordinates corresponding to each holographic unit The value of the sub-holographic modulation function at the location.

[0038] It should be noted that: the sub-holographic modulation function describes the ideal optical field modulation mode that should be generated by a single point within the corresponding sub-holographic unit region; and the choice of the sub-holographic basis function should match the pixel size and operating wavelength of the spatial light modulator. For example, the analytical form of the spherical wave phase function can be used, with the expression: In the formula, The base of the natural logarithm. Represents the imaginary unit ( ), Indicates the wavenumber of the incident laser ( , (where λ is the wavelength of the incident light wave) This represents the Euclidean distance from the center point of the sub-holographic unit to any consecutive coordinate point on the SLM plane. Indicates the tilt factor ( Through algebraic operations Directly obtain, no angle calculation required (cosine function value) Represents the cosine function. Represents the spatial position from the center of the sub-holographic unit. The angle between the line connecting the two points and the SLM plane normal (z-axis), This represents a two-dimensional rectangular window function. It represents the equivalent propagation distance (in a Fourier holographic reconstruction system, it is taken as the focal length of the Fourier lens; in an image plane holographic system, it is taken as the geometric distance from the SLM to the reconstruction plane). Secondly, the discrete spatial frequency modeling process is mathematically equivalent to: constructing a discrete sampling grid composed of two-dimensional Dirac pulse functions, where non-zero pulse values ​​exist only at the sub-holographic unit index positions corresponding to point cloud points, and zero values ​​are found at other positions; and then convolving this discrete sampling grid with the corresponding sub-holographic modulation function in the spatial domain. In the specific algorithm implementation, this means that in the holographic complex amplitude distribution matrix of the spatial light modulator, complex values ​​are filled only at the center position of each effective sub-holographic unit and its neighborhood, according to the amplitude and phase distribution of the sub-holographic basis function, while the sub-holographic unit regions not hit by any point cloud point index remain at zero values ​​or background values. The final output holographic complex amplitude distribution is a two-dimensional complex matrix that perfectly matches the pixel array size of the spatial light modulator.

[0039] S2.1. Based on the three-dimensional point cloud set, calculate the initial amplitude weight and generate an initial phase free variable for each point in the three-dimensional point cloud set.

[0040] Specifically, for each point in the 3D point cloud set, the corresponding light intensity value is read, the entire 3D point cloud set is traversed to find the maximum light intensity value as the normalization benchmark, the light intensity value of each point is divided by the normalization benchmark to obtain the initial amplitude weight, and then an initial phase free variable is generated for each point. It should be noted that the initial phase free variables are... The random phase value generated by the pseudo-random number generator is introduced to break the coherent interference fringes between point cloud points, suppress speckle noise in the holographic reconstruction image, and make the reconstructed light field more uniform and smooth. and The open interval is set based on the physical periodicity of the light wave phase and the statistical uniformity requirement for speckle suppression in holographic reconstruction; for example, the phase of the light wave is based on... A cyclic quantity with a periodicity, where any phase value is added to or subtracted from it. After multiples of a certain value, the represented optical field vibration state is exactly the same, therefore the effective independent range of phase values ​​is naturally limited to a length of [missing value]. Within the range; S2.2 Obtain the spatial position and phase of each point in the three-dimensional point cloud set at continuous spatial coordinates in the spatial light modulator plane.

[0041] The spatial position phase is determined by the transverse spatial frequency modulation term and the axial propagation phase term, and the specific expression is as follows: In the formula, Indicates the first The coordinates of the point cloud points in continuous space Spatial position phase, This represents the wavelength of the incident light wave. This indicates the focal length of the Fourier lens.

[0042] S2.3. Based on the initial amplitude weight, initial phase free variable, and spatial position phase, perform complex amplitude synthesis operation on each point cloud point to generate initial complex amplitude function value. Then, convert the initial complex amplitude function value to the focal plane through Fourier propagation to obtain the complex light field distribution function value of the focal plane.

[0043] The initial complex amplitude function value is generated as follows: In the formula, Represents the initial complex amplitude function value on the spatial light modulator plane (and) Physically equivalent Indicates the first Initial amplitude weights for each point cloud point Indicates the first The coordinates of the point cloud points in continuous space Spatial position phase, Indicates the first Initial phase free variables for each point cloud point; It should be noted that the generated initial complex amplitude function value is physically equivalent to the holographic complex amplitude distribution obtained by the convolution modeling of the preceding sub-holographic units. Both describe the optical field modulation state that should exist on the spatial light modulator plane. The initial complex amplitude function value is then transformed to the focal plane using Fourier propagation, and the expression is: In the formula, Indicates the first Focal plane spatial coordinates at the next iteration The complex optical field distribution function value on the surface, Represents the two-dimensional Fourier transform operator. Indicates the first Next iteration of the spatial-temporal light modulator plane spatial coordinates The initial complex amplitude function value on.

[0044] It should be noted that the two-dimensional Fourier transform operator actually uses the Fast Fourier Transform algorithm. The input is a two-dimensional complex matrix on the spatial light modulator plane, and the output is a two-dimensional complex matrix on the focal plane. Before performing the transform, it is necessary to confirm that the size of the input matrix is ​​consistent with the pixel resolution of the spatial light modulator, and to verify whether the frequency domain sampling interval meets the physical size requirements of the focal plane according to the sampling theorem. If necessary, zero-value padding is added around the input matrix to increase the frequency domain sampling density. The complex matrix output by the transform is the complex light field distribution function value on the focal plane spatial coordinates. This function value contains the amplitude and phase information of each point on the focal plane. Secondly, the output of the two-dimensional Fourier transform usually requires a spectral centering shift operation, that is, shifting the zero-frequency component to the center of the matrix to match the origin definition of the physical coordinate system of the focal plane. This shift operation can be achieved by alternatingly multiplying the input matrix by a checkerboard factor of positive and negative one before the transformation, or by directly swapping the matrix quadrants after the transformation.

[0045] S3. Based on the focal plane complex optical field distribution, after reconstructing the holographic complex amplitude field in an iterative manner, the final complex amplitude holographic field is output, a binary mask is constructed to form an aperture function, and the spatial variable point diffusion function response is constructed in combination with the final complex amplitude holographic field. The final complex optical field distribution is generated in combination with the sub-holographic unit.

[0046] Specifically, based on the complex light field distribution function value of the focal plane, the amplitude weight and phase free variable of each point cloud point are optimized; After each optimization, the updated amplitude weights and phase free variables are substituted into the initial complex amplitude function value. After keeping the spatial position and phase unchanged, the holographic complex amplitude function value on the spatial light modulator plane of each iteration is reconstructed. Then, the holographic complex amplitude function value on the spatial light modulator plane is transformed to the focal plane (Fourier propagation method transformation to the focal plane) to obtain the updated complex light field distribution function value. The normalized root mean square error (RMSE) of the intensity of all point cloud points is obtained as the convergence index. After the maximum number of iterations, the final spatial light modulator complex amplitude holographic field is output.

[0047] In the amplitude weight optimization, the current complex light field distribution function value of the point cloud points at the focal plane spatial position is extracted and smoothly updated. The update expression is: In the formula, Indicates the first During the nth iteration The update amplitude weight of each point cloud point Represents the iteration convergence coefficient. Indicates the first During the nth iteration The amplitude weight of each point cloud point Indicates the first Focal plane spatial coordinates at the next iteration The amplitude at that point, Indicates the first The target intensity value of the point cloud point (derived from the first point cloud point) (The light intensity values ​​of each point cloud point are generated after normalization to the global maximum value). It should be noted that: iterative convergence coefficients When the coefficient is less than 0.1, the update step size becomes too small, requiring hundreds or even thousands of iterations to reduce the root mean square error of the normalized intensity to an acceptable level. Conversely, when the coefficient is greater than 0.5, due to the complex coherent interference coupling effect between points in the holographic light field, excessive adjustment of the amplitude at a single point can cause drastic fluctuations in the light intensity at other points, leading to significant oscillations or even divergence in the error curve. Therefore, this invention preferably uses 0.3 as the default value, which ensures monotonic convergence and allows the algorithm to reach a steady state within a small number of iterations. When optimizing phase free variables, the phase angle is obtained by complex number operations. The expression for the phase free variable after the next iteration is: In the formula, Indicates the first During the nth iteration The update phase free variable for each point cloud point This indicates the phase angle operation of complex numbers. However, in actual numerical implementation, the four-quadrant arctangent function should be used instead of the single-valued arcsine or inverse cosine function to avoid phase ambiguity. The normalized root mean square error (RMSE) of the intensity of all point cloud points is used as the convergence metric, expressed as: In the formula, Indicates the first Convergence metrics at the next iteration Indicates the first Focal plane spatial coordinates at the next iteration The amplitude at that point.

[0048] It should be noted that when calculating the convergence index, the normalization benchmark of the target intensity value in the denominator must be consistent with the global maximum value used when generating the target intensity value; and the convergence index reflects the overall matching degree between the reconstructed light field and the target light field at the intensity level, with a smaller value indicating a higher matching degree.

[0049] S3.1 Construct binary mask values ​​based on the actual effective aperture region of the spatial light modulator.

[0050] It should be noted that the actual effective aperture area is the modulated pixel area; The binary mask value is constructed using the following expression: In the formula, Represents the binary mask value. and These represent the row and column indices of the pixels of the spatial light modulator in the two-dimensional array. Indicates belonging to, This represents the set of effective operating regions of the spatial light modulator (as specified by the device manufacturer). This indicates that it does not belong to [the group / organization]. Indicates the first Each pixel is located within the actual effective aperture region of the spatial light modulator. Indicates the first One pixel is outside the actual effective aperture area of ​​the spatial light modulator.

[0051] It should be noted that the actual effective aperture area is determined based on the spatial light modulator's manufacturer's manual or actual calibration data. The effective working area does not always cover the entire pixel array; due to manufacturing processes, packaging obstructions, or limitations of the driving circuit, there may be unmodulatorable dead zones or non-responsive pixels at the edges. In practice, a two-dimensional binary matrix with the exact same nominal pixel resolution as the spatial light modulator can be created in computer memory as a mask template. Then, by traversing each row and column index of this matrix, it is determined whether the index falls within the predefined set of effective working areas: if it belongs to an effective area, the mask value at that position is set to logic 1; if it does not belong to an effective area, the mask value at that position is set to logic 0.

[0052] S3.2 Map the binary mask values ​​back to the spatial light modulator plane to obtain the aperture function value of the spatial light modulator in continuous spatial coordinates.

[0053] The expression for mapping the binary mask values ​​back to the spatial light modulator plane is: In the formula, Represents coordinates in continuous space The aperture function value below, This indicates a round-down operation. and These represent the spatial light modulator in direction and The physical pixel spacing in the direction (given by the actual device specifications).

[0054] It should be noted that the aperture function is physically represented as a series of closely arranged rectangular window functions, the size of each rectangular window is exactly equal to the physical spacing of a single pixel, and its transmission or modulation state is consistent with the value of that pixel in the binary mask.

[0055] S3.3 Based on the final spatial light modulator complex amplitude holographic field and aperture function value, construct the spatial variable point diffusion function and calculate the response value of the spatial variable point diffusion function.

[0056] The spatially variable point diffusion function response value is calculated as follows: In the formula, Represents the focal plane coordinates The spatially variable point diffusion function response value at that location. This represents the final spatial light modulator complex amplitude holographic field; It should be noted that the spatially variable spread function response value characterizes the comprehensive diffraction field envelope generated by the entire hologram on the focal plane after considering the actual finite aperture effect of the spatial light modulator. This envelope includes high-frequency loss and edge diffraction effects caused by aperture truncation. When performing the Fourier transform, it is necessary to ensure that the input matrix has been properly zero-padded to avoid cyclic convolution artifacts, and the transform result should be spectrally centered to align the origin of the focal plane coordinates with the center of the matrix.

[0057] S3.4 Perform frequency domain transformation on the sub-holographic unit to generate the response function value in the frequency domain space. Combine the response value of the spatial variable point spread function and perform convolution operation to generate the final complex optical field distribution function value.

[0058] The expression for generating the final complex optical field distribution function value is: In the formula, This represents the final complex light field distribution function value. This represents the value of the sub-holographic modulation function corresponding to the final spatial light modulator complex amplitude holographic field; It should be noted that the frequency domain transformation of the sub-holographic units generates response function values ​​in the frequency domain. Specifically, this involves extracting the sub-holographic modulation function corresponding to the final spatial light modulator complex amplitude holographic field. This function describes the complex amplitude distribution template of a single sub-holographic unit in the spatial domain. A two-dimensional fast Fourier transform is also performed on the sub-holographic modulation function to obtain its response function value in the frequency domain. This sub-holographic modulation function is relative to the final holographic field state. If the sub-holographic basis function parameters have been updated during the iteration process, the updated version must be used; otherwise, the initially preset sub-holographic basis function can be used. The frequency domain response function value reflects the frequency selectivity and bandwidth limitations of the sub-holographic unit itself. Secondly, the convolution operation involves a concatenated convolution of three factors: the first factor is the calculated spatial variable point spread function response value; the second factor is a discrete pulse array composed of the center positions of all sub-holographic units, which has a unit pulse value only at the coordinate position corresponding to the row and column index of each sub-holographic unit multiplied by the physical spacing, and zero at the other positions; the third factor is the frequency domain response function value of the sub-holographic unit.

[0059] S4. Based on the final complex light field distribution, the phase angle is taken and mapped to a driving grayscale image, which is then loaded into the spatial light modulator to realize three-dimensional holographic projection.

[0060] Specifically, based on the final complex light field distribution function value, after performing the phase angle calculation, the pixel phase value required by the spatial light modulator is generated; The phase value is linearly mapped according to the phase modulation depth and gray level of the spatial light modulator to obtain the gray value, and a driving grayscale map is constructed. The driving grayscale image is encoded into an image format and transmitted to a spatial light modulator for incident light wave modulation. After the incident light wave is modulated, the laser source of the spatial light modulator is turned on and uniformly illuminated onto the surface of the spatial light modulator loaded with grayscale image. The reconstructed light field is formed through a Fourier lens and then received by a diffuse reflection screen to realize three-dimensional holographic projection.

[0061] It should be noted that the phase angle calculation can be performed using the four-quadrant arctangent function, and if the phase value is negative, then add... Make it fall into 0 to Interval; if the phase value is greater than or equal to Then subtract The phase value is an integer multiple of the given value until it falls within that interval. The normalized phase value is the phase value that each pixel of the spatial light modulator needs to be loaded with; The phase modulation depth refers to the maximum phase delay that a spatial light modulator can produce under the maximum driving voltage, and is usually specified by the manufacturer. Secondly, the grayscale level is determined by the bit depth of the digital-to-analog converter of the spatial light modulator. For example, 8 bits correspond to 0 to 255 levels, and 10 bits correspond to 0 to 1023 levels. In the specific mapping, it is first determined whether the theoretical phase value exceeds the actual phase modulation depth range. If it does, it is compressed or truncated to the effective range. Then, the effective phase value is divided by the phase modulation depth to obtain the normalization scaling factor. This scaling factor is then multiplied by the maximum value of the grayscale level and rounded down to obtain the grayscale value. It should also be noted that the surface roughness of the diffuse reflection screen should meet the Lambertian scattering condition, and the position of the diffuse reflection screen should be precisely placed at the focal plane of the Fourier lens.

[0062] This embodiment also provides a computer device applicable to the dynamic capture-based holographic projection method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the dynamic capture-based holographic projection method proposed in the above embodiment.

[0063] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0064] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the holographic projection method based on dynamic capture as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0065] In summary, by smoothly updating amplitude weights and synchronously optimizing phase free variables, combined with random initial phase to break coherent interference fringes, speckle noise is effectively suppressed and oscillation divergence is avoided, achieving high-fidelity intensity matching within a finite number of iterations. Furthermore, by constructing a binary mask based on the actual effective aperture of the spatial light modulator and mapping it to a continuous aperture function, and performing cascaded convolution in conjunction with the frequency domain response of the sub-holographic units, the spatially varying point spread function is accurately modeled, pre-compensating for edge diffraction artifacts and high-frequency information loss. Therefore, this invention significantly improves the imaging quality and engineering robustness of dynamic target 3D holographic reconstruction.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A holographic projection method based on dynamic capture, characterized in that, include: The dynamic scene time-series images acquired by the monocular camera are preprocessed and depth estimated to generate a depth map corresponding to each frame. Then, fusion resampling is performed to obtain a three-dimensional point cloud set. The three-dimensional point cloud set is mapped to the spatial light modulator coordinate system, and square sub-holographic units are divided according to physical spacing. The sub-holographic units are convolved and modeled to generate the holographic complex amplitude distribution of the spatial light modulator plane. Then, the complex amplitude synthesis operation is performed to generate the initial complex amplitude function value, which is propagated to the focal plane through Fourier transform to form the complex light field distribution of the focal plane. The complex amplitude synthesis operation generates an initial complex amplitude function value, which is then propagated to the focal plane via Fourier transform, forming a complex optical field distribution on the focal plane, as detailed below: Based on a 3D point cloud set, the initial amplitude weight is calculated, and an initial phase free variable is generated for each point in the 3D point cloud set. Obtain the spatial position and phase of each point in the three-dimensional point cloud set at continuous spatial coordinates in the spatial light modulator plane; Based on the initial amplitude weights, initial phase free variables, and spatial position phase, a complex amplitude synthesis operation is performed on each point cloud point to generate an initial complex amplitude function value. Then, the initial complex amplitude function value is transformed to the focal plane through Fourier propagation to obtain the complex light field distribution function value of the focal plane. Based on the focal plane complex optical field distribution, the holographic complex amplitude field is reconstructed in an iterative manner, and the final complex amplitude holographic field is output. A binary mask is constructed to form an aperture function. Combined with the final complex amplitude holographic field, a spatial variable point diffusion function response is constructed. Combined with sub-holographic units, the final complex optical field distribution is generated. The process of reconstructing the holographic complex amplitude field based on the focal plane complex optical field distribution using an iterative method, and then outputting the final complex amplitude holographic field, includes: Based on the complex optical field distribution function value of the focal plane, optimize the amplitude weight and phase free variable of each point cloud point; After each optimization, the updated amplitude weights and phase free variables are substituted into the initial complex amplitude function value. After keeping the spatial position and phase unchanged, the holographic complex amplitude function value on the spatial light modulator plane of each iteration is reconstructed. Then, the holographic complex amplitude function value on the spatial light modulator plane is transformed to the focal plane to obtain the updated complex light field distribution function value. The normalized root mean square error of the intensity of all point cloud points is obtained as the convergence index. After the maximum number of iterations is reached, the final spatial light modulator complex amplitude holographic field is output. The process involves constructing a binary mask to form an aperture function, combining it with the final complex amplitude holographic field, constructing a spatially variable point diffusion function response, and combining it with sub-holographic units to generate the final complex optical field distribution, as detailed below: Based on the actual effective aperture region of the spatial light modulator, construct binary mask values; By mapping the binary mask values ​​back to the spatial light modulator plane, the aperture function values ​​of the spatial light modulator in continuous spatial coordinates are obtained. Based on the final spatial light modulator complex amplitude holographic field and aperture function value, a spatial variable point diffusion function is constructed, and the response value of the spatial variable point diffusion function is calculated. The sub-holographic unit is frequency domain transformed to generate response function values ​​in the frequency domain space. Combined with the response values ​​of the spatially variable point spread function, a convolution operation is performed to generate the final complex optical field distribution function values. Based on the final complex light field distribution, the phase angle is taken and mapped to a driving grayscale image, which is then loaded into a spatial light modulator to achieve three-dimensional holographic projection.

2. The holographic projection method based on dynamic capture as described in claim 1, characterized in that: The preprocessing includes noise reduction and normalization.

3. The holographic projection method based on dynamic capture as described in claim 1, characterized in that: The depth estimation involves inputting each frame of the standardized image sequence into a pre-trained deep learning depth estimation network and outputting a depth map corresponding to each frame of the standardized image.

4. The holographic projection method based on dynamic capture as described in claim 1, characterized in that: The process of mapping a 3D point cloud set to the spatial light modulator coordinate system, dividing it into square sub-holographic units according to physical spacing, performing convolution modeling on the sub-holographic units, and generating a holographic complex amplitude distribution on the spatial light modulator plane includes: Based on a 3D point cloud set, the coordinates of all point cloud points are uniformly mapped to the coordinate system of the spatial light modulator (SLM). After obtaining the SLM plane coordinates of the point cloud points, the coordinate plane of the SLM is divided into several square sub-regions as sub-holographic units according to the number of point cloud points in the 3D point cloud set, and the physical spacing of the sub-holographic units is set. ; Traverse the 3D point cloud set and determine the sub-holographic unit to which the SLM plane coordinates of the point cloud point belong by using a rounding function based on the physical spacing of the sub-holographic units. After performing convolution operations on the sub-holographic units using preset sub-holographic basis functions, discrete spatial frequency modeling of the spatial light modulator plane is performed to generate a holographic complex amplitude distribution on the spatial light modulator plane.

5. The holographic projection method based on dynamic capture as described in claim 1, characterized in that: The process of mapping the phase angle of the final complex light field distribution to a driving grayscale image and loading it onto a spatial light modulator to achieve three-dimensional holographic projection includes: Based on the final complex light field distribution function value, after performing the phase angle calculation, the pixel phase value required by the spatial light modulator is generated; The phase value is linearly mapped according to the phase modulation depth and gray level of the spatial light modulator to obtain the gray value, and a driving grayscale map is constructed. The driving grayscale image is encoded into an image format and transmitted to a spatial light modulator for incident light wave modulation. After the incident light wave is modulated, the laser source of the spatial light modulator is turned on and uniformly illuminated onto the surface of the spatial light modulator loaded with grayscale image. The reconstructed light field is formed through a Fourier lens and then received by a diffuse reflection screen.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the holographic projection method based on dynamic capture as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the holographic projection method based on dynamic capture as described in any one of claims 1 to 5.

Images