Augmented Reality Device and Method Integrating Sensing and Display Through Scattering Media

By combining modulation lighting modules, information perception and control modules and augmented reality modules in augmented reality technology, using near-infrared light sources and spatial light modulators, real-time observation of non-horizontal targets and high-level three-dimensional scenes in scattering media is achieved. Quality reconstruction and display solve the problem of insufficient visual clarity and resolution in the prior art.

CN118409435BActive Publication Date: 2024-11-26NANCHANG YIJING INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410889357.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-26
Estimated Expiration
2044-07-04

AI Technical Summary

Technical Problem

Existing augmented reality technology is difficult to achieve the perception and display of non-line-of-sight targets in scattering media such as haze and turbid water, resulting in insufficient visual clarity and resolution, limiting its application in scenarios such as emergency rescue and industrial inspection.

Method used

Using a combination of modulated lighting module, information perception and control module and augmented reality module, the speckle image acquisition and reconstruction is carried out using near-infrared light sources, image sensors and upper computers, and a three-dimensional color light field is realized through spatial light modulators and red, green and blue lasers. Modulation and holograms show visual obstacles that break through scattering media.

Benefits of technology

Real-time and intuitive observation of non-sight targets in scattering media, improve observation clarity and convenience, overcome imaging challenges in unstable or unknown scattering environments, improve the spatial resolution of images and the imaging system. Performance, capable of high-quality reconstruction and display of three-dimensional scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118409435B_ABST
    Figure CN118409435B_ABST
Patent Text Reader

Abstract

The present invention discloses an augmented reality device and method integrating perception and display through a scattering medium, the device comprising a modulated lighting module, an information perception and control module and an augmented reality module; the modulated lighting module utilizes an incoherent light source in the near-infrared band to illuminate the target to collect incident light distribution information; the information perception and control module utilizes a host computer to collect speckle images and execute an image reconstruction algorithm, then sends the image to a spatial light modulator controller, and synchronously controls red, green and blue lasers to achieve three-dimensional color light field modulation control; the augmented reality module achieves final three-dimensional image observation by loading a pure phase hologram. The present invention effectively overcomes the imaging problem in an unstable or unknown scattering environment by integrating image enhancement technology and speckle convolution reconstruction method, and achieves strong analytical capabilities even under extreme conditions to reconstruct high-quality images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an augmented reality device and method integrating perception and display through a scattering medium. Background Art

[0002] When light propagates in scattering media such as haze, turbid water and biological tissue, it no longer follows the law of rectilinear propagation, which greatly reduces the ability of traditional optical systems to analyze scene information and restricts the clarity, resolution and other parameters of optical imaging systems. This restriction has brought inconvenience to people's daily life, emergency scenarios such as underwater rescue, and many aspects of scientific research, and an effective visual enhancement method is urgently needed. Although computational optical imaging systems represented by imaging through scattering media can achieve image reconstruction in complex scenes, such systems are complex and usually use traditional upper computer displays to display results, making it difficult to observe non-line-of-sight targets blocked by scattering cutoff in real application scenarios. In recent years, augmented reality technology has developed rapidly. It projects two-dimensional streaming media information to the human eye through binocular vision, which can effectively enhance information in real scenes. However, the sensors such as cameras, accelerometers, and gyroscopes carried by it only imitate the perception function of humans, and it is difficult to perceive non-line-of-sight targets blocked by scattering objects.

[0003] Existing technologies use augmented reality technology to simulate binocular vision, effectively project two-dimensional streaming media information into the user's field of view, enhance the information of real scenes, and provide users with an innovative way to interact with virtual elements in the real world. With the development of camera and sensor technology, the perception ability of augmented reality devices has been continuously improved, approaching the human perception level, and realizing instant perception and response to environmental changes. However, existing augmented reality devices are limited by sensor performance in specific applications such as emergency rescue and industrial inspections, and it is difficult to quickly capture and reconstruct real three-dimensional scenes, especially for non-line-of-sight target detection under conditions where objects are blocked by scattering media such as rain, fog, turbid water, and thick smoke. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies in the prior art and to provide an augmented reality device and method for integrating perception and display through a scattering medium, aiming to break through the visual limitations brought about by scattering media (such as haze, turbid water bodies, biological tissues, etc.), and to improve the clarity and convenience of observation of an augmented reality device for perception and display through a scattering medium, so that users can directly observe the target behind the scattering medium in real time in a naked-eye holographic manner.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions.

[0006] An augmented reality device integrating perception and display through a scattering medium, comprising a modulated lighting module, an information perception and control module and an augmented reality module;

[0007] The modulated illumination module utilizes an incoherent light source in the near-infrared band to illuminate the target to collect incident light distribution information;

[0008] The information perception and control module uses a host computer to collect speckle images and execute an image reconstruction algorithm, then sends the images to the spatial light modulator controller and synchronously controls the red, green and blue lasers to achieve three-dimensional color light field modulation control;

[0009] The augmented reality module realizes the final three-dimensional image observation by loading the pure phase hologram.

[0010] Specifically, the modulated lighting module includes an incoherent near-infrared light source and an illumination target. The light emitted by the incoherent near-infrared light source interacts with the illumination target and is interfered by a scatterer to form a speckle image, which is collected and processed by the information perception and control module.

[0011] Specifically, the information perception and control module includes a filter, an image sensor, a host computer and a phase-type spatial light modulator controller; the image sensor is responsible for collecting speckle images, and the filter is placed in front of the image sensor to filter out stray background light interference; the host computer uses a point spread function to perform deconvolution processing on the speckle image collected by the image sensor, obtains a pure phase-type hologram, and then sends it to the phase-type spatial light modulator controller, and synchronously emits it to the red, green and blue three-color lasers in a specific timing sequence to realize three-dimensional color light field modulation control.

[0012] Specifically, the augmented reality module includes a spatial light modulator, a beam splitting cube, a red, green and blue laser, a collimating lens and a display optical waveguide; the spatial light modulator is loaded with a pure phase hologram and works synchronously with the red, green and blue lasers; the reference light diffracted by the spatial light modulator is transmitted to the display optical waveguide through a beam splitting prism together with the parallel reference light converted by the collimating lens, so that the human eye can observe the three-dimensional information of the original target scene.

[0013] Based on the augmented reality device provided by the above technical solution, the present invention also provides an augmented reality method integrating perception and display through a scattering medium, comprising the following steps:

[0014] Step 1: Background speckle data detection: In a natural environment, multiple frames of noise speckle data that do not contain target signals are collected and used as local noise for subsequent processing;

[0015] Step 2: System calibration: measure the point spread function at different distances to calibrate the system parameters;

[0016] Step 3: Data collection: Under the same detection environment, multiple frames of speckle data containing target information are collected and transmitted to the host computer for scene reconstruction;

[0017] Step 4: Target scene reconstruction: Use stacking averaging, background noise removal, and bilateral filtering techniques to preprocess the collected data to enhance data quality, and use the system point spread function calibrated in step 2 to perform deconvolution processing to achieve scene reconstruction;

[0018] Step 5, pure phase hologram solution: Use grayscale image colorization technology to convert the pre-processed scattering scene into three-wavelength color data, and combine it with the pure phase hologram generation algorithm to realize the construction of pure phase hologram data;

[0019] Step 6, hologram loading: the host computer loads one channel of the calculated pure phase hologram into the spatial light modulator;

[0020] Step 7, holographic display: while executing step 6, synchronously light up the laser of the corresponding band;

[0021] Step 8: Repeat steps 6 to 7 at a certain refresh frequency to realize color hologram display in the form of time division multiplexing.

[0022] Specifically, the process of collecting speckle data in step 3 is as follows:

[0023] After the light source irradiates the target object and passes through the scattering medium, the corresponding speckle pattern will be received on the image sensor. The scattering within a certain range is regarded as useful information, and the scattering outside the range is regarded as useless noise. At this time, if a beam of incoherent light is used to irradiate the target object and pass through the scattering medium, the speckle image obtained on the image sensor is expressed as:

[0024] ;

[0025] In the above formula, represents the speckle image obtained on the detector; represents the original target image; subscript represents the image plane; subscript Represents the plane of an object; Expressed as the point spread function of the observed target and system; It represents the angular correlation of the speckle images in front of the two incident waves, that is, the shape factor function that describes the degree of optical memory effect.

[0026] Specifically, in step 4, the system point spread function calibrated in step 2 is used for deconvolution processing to achieve scene reconstruction, and the process is as follows:

[0027] Introducing the system point spread function calibrated in step 2 into the deconvolution model, we get the following equation:

[0028] ;

[0029] In the above formula, represents the Fourier transform, is the inverse Fourier transform, represents the conjugate of the Fourier transform result; is the speckle measurement image;

[0030] In the prior information learning stage of the model, the denoising score matching function is used to learn the gradient distribution of the target hologram. After the input pure noise image is predicted and corrected, the amplitude image of the reconstructed target is obtained by diffraction through the angular spectrum method. Then, the difference between the reconstructed image and the target image is calculated. Then, the hologram generation subnetwork driven by the diffraction model is used to convert the difference in the amplitude domain into the holographic domain. Subsequently, the amplitude constraint of the target image is introduced by adding the difference hologram to the hologram that has been predicted and corrected before. Finally, a high-quality target hologram is generated through continuous iterations, thereby realizing the reconstruction of the target scene.

[0031] Furthermore, in the hologram prior information learning stage, a generative model is introduced to learn the prior information data distribution, thereby strengthening the generation and diversification of real samples; the diffusion model is learned by maximum likelihood estimation, and a forward SDE process is set based on the score matching generative model, and the corresponding reverse time SDE is defined; the forward diffusion process smoothly transforms the complex data distribution into a known prior distribution by slowly injecting noise, while the reverse time SDE transforms the prior distribution into data by slowly removing noise; the score model perturbs the data distribution by adding noise; in order to sample from the data distribution, a neural network needs to be trained to estimate the logarithmic data distribution The gradient is used to solve the inverse temporal SDE to achieve the goal of generating data from noise; is the target-related prior learned during forward SDE training; due to variance explosion, SDE produces higher quality samples, which can be obtained by constructing a variance explosion-SDE and training a time-based scoring network To estimate , so that , and then replace the corresponding terms in the calculation; the actual score function is difficult to obtain, and the denoised score matching is used to replace it with ; So The Gaussian perturbation kernel centered at . Under regularization conditions, the trained network basically satisfies After the network training is completed, replace To approximately solve the inverse SDE.

[0032] Specifically, the process described in step 5 is to combine the pure phase hologram generation algorithm to construct the pure phase hologram data as follows:

[0033] In the hologram generation stage, by optimizing the hologram generation algorithm, the obtained hologram The reproduced image is as close as possible to the target image , to minimize the difference between the two; this process is expressed as:

[0034] ;

[0035] The gradient descent method is used to iteratively minimize the difference between the reproduced image and the target image, which is expressed as follows:

[0036] ;

[0037] In the above formula, represents the hologram obtained in the last iteration, is the hologram obtained in this round of iteration, represents the iteration step length; Indicates the number of iterations; is the angular spectrum propagation operator; yes The adjoint operator of represents the hologram generation transformation driven by the diffraction model; is the gradient of the data consistent term;

[0038] In order to reduce the ill-posedness of the gradient descent method and obtain the optimal solution, a regularization term is added to constrain it, which is expressed as:

[0039] ;

[0040] In the above formula, Represents data consistency items, represents the regularization parameter, represents the regularization term containing target prior information;

[0041] The pure phase hologram output is achieved by iterating the following formula (6);

[0042] ;

[0043] In the above formula, represents the regularization term containing the prior information of the target of this round of iteration, represents the partial derivative operator; an alternating optimization algorithm is used to decouple the prior information term and the data consistency term; by decoupling, equation (6) is converted into an optimization problem in which two sub-problems are updated alternately:

[0044] ;

[0045] In the above formula, the first formula represents model generation, which is the regularization part of the iterative generation process; the second formula is the data fidelity term;

[0046] The sample estimate of the next time step is provided through a fixed discretization strategy to generate the target image from the prior distribution of the target image, which is expressed as:

[0047] ;

[0048] In the above formula, is Gaussian white noise that conforms to the standard normal distribution. In the above formula, is a time-based scoring network, represents the noise scale of this iteration, Indicates the noise scale for the next iteration;

[0049] In the correction step, the Langevin correction algorithm is used to correct the gradient ascent direction, and equation (8) is further expressed as:

[0050] ;

[0051] In the above formula, is the noise iteration step; the target hologram is generated based on the corrected model and the pure phase hologram data is constructed.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention proposes a solution for an integrated device of perception through scattering media and augmented reality. The core of this technology is to realize imaging through scattering media based on the advantages of augmented reality technology. Through this innovative device, the present invention successfully breaks through the visual obstacles caused by scattering media such as haze, turbid water, and smoke, which not only promotes the accurate reconstruction of images in complex scenes, but also realizes real-time and intuitive observation of non-line-of-sight targets in actual environments, greatly improving the clarity and convenience of observation.

[0054] 2. The present invention effectively overcomes the imaging challenges in unstable or unknown scattering environments by integrating image enhancement technology and speckle convolution reconstruction methods, and achieves high-quality image reconstruction even under extreme conditions. By combining an accurate physical transmission model, a highly reliable and high-quality reconstruction and display of three-dimensional scenes is achieved. In addition, using near-infrared measurement data, this technology can accurately reproduce stereoscopic scenes under three wavelengths of visible light in a holographic form. In general, this technology not only improves the spatial resolution of the image, but also improves the performance of the imaging system when dealing with unstable or unknown scattering environments. Even for non-line-of-sight targets blocked by scattering objects, high-quality images can be reconstructed. Experiments have shown that the integrated device can not only overcome the adverse effects of scattering media, has strong resolution capabilities to achieve clear imaging, but also achieve high-quality holographic display. The system resolution is 11.314 lp / mm, and the corresponding line pair width is 88.39 μm, which fully proves that the perception and display integrated augmented reality device proposed by the present invention through the scattering medium can effectively distinguish subtle targets and achieve high-resolution imaging through the scattering medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is a structural block diagram of an augmented reality device integrating perception and display through a scattering medium according to the present invention;

[0057] Figure 2 It is a flow chart of a method for integrating perception and display of augmented reality through a scattering medium according to the present invention;

[0058] Figure 3 It is a full process block diagram of the prior information learning stage and the hologram generation stage of the present invention;

[0059] Figure 4 A schematic diagram of a hologram generation subnetwork of the present invention;

[0060] Figure 5 Schematic diagram of system imaging experimental results in an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of resolution test results in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the following detailed description is given to each step of the method proposed by the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0063] Example

[0064] like Figure 1 As shown, a perception and display integrated augmented reality device through a scattering medium includes a modulation lighting module, an information perception and control module and an augmented reality module;

[0065] The modulated illumination module utilizes an incoherent light source in the near-infrared band to illuminate the target to collect incident light distribution information;

[0066] The information perception and control module uses a host computer to collect speckle images and execute an image reconstruction algorithm, then sends the images to the spatial light modulator controller and synchronously controls the red, green and blue lasers to achieve three-dimensional color light field modulation control;

[0067] The augmented reality module realizes the final three-dimensional image observation by loading the pure phase hologram.

[0068] Specifically, the modulated lighting module includes an incoherent near-infrared light source and an illumination target. The light emitted by the incoherent near-infrared light source interacts with the illumination target and is interfered by a scatterer to form a speckle image, which is collected and processed by the information perception and control module.

[0069] Specifically, the information perception and control module includes a filter, an image sensor, a host computer and a phase-type spatial light modulator controller; the image sensor is responsible for collecting speckle images, and the filter is placed in front of the image sensor to filter out stray background light interference; the host computer uses a point spread function to perform deconvolution processing on the speckle image collected by the image sensor, obtains a pure phase-type hologram, and then sends it to the phase-type spatial light modulator controller, and synchronously emits it to the red, green and blue three-color lasers in a specific timing sequence to realize three-dimensional color light field modulation control.

[0070] Specifically, the augmented reality module includes a spatial light modulator, a beam splitting cube, a red, green and blue laser, a collimating lens and a display optical waveguide; the spatial light modulator is loaded with a pure phase hologram and works synchronously with the red, green and blue lasers; the reference light diffracted by the spatial light modulator is transmitted to the display optical waveguide through a beam splitting prism together with the parallel reference light converted by the collimating lens, so that the human eye can observe the three-dimensional information of the original target scene.

[0071] like Figure 2 As shown, based on an augmented reality device provided by the above technical solution, the present invention also provides an augmented reality method integrating perception and display through a scattering medium, comprising the following steps:

[0072] Step 1: Background speckle data detection: In a natural environment, multiple frames of noise speckle data that do not contain target signals are collected and used as local noise for subsequent processing;

[0073] Step 2: System calibration: measure the point spread function at different distances to calibrate the system parameters;

[0074] Step 3: Data collection: Under the same detection environment, multiple frames of speckle data containing target information are collected and transmitted to the host computer for scene reconstruction;

[0075] Step 4: Target scene reconstruction: Use stacking averaging, background noise removal, and bilateral filtering techniques to preprocess the collected data to enhance data quality, and use the system point spread function calibrated in step 2 to perform deconvolution processing to achieve scene reconstruction;

[0076] Step 5, pure phase hologram solution: Use grayscale image colorization technology to convert the pre-processed scattering scene into three-wavelength color data, and combine it with the pure phase hologram generation algorithm to realize the construction of pure phase hologram data;

[0077] Step 6, hologram loading: the host computer loads one channel of the calculated pure phase hologram into the spatial light modulator;

[0078] Step 7, holographic display: while executing step 6, synchronously light up the laser of the corresponding band;

[0079] Step 8: Repeat steps 6 to 7 at a certain refresh frequency to realize color hologram display in the form of time division multiplexing.

[0080] Specifically, the process of collecting speckle data in step 3 is as follows:

[0081] After the light source irradiates the target object and passes through the scattering medium, the corresponding speckle pattern will be received on the image sensor. The scattering within a certain range is regarded as useful information, and the scattering outside the range is regarded as useless noise. At this time, if a beam of incoherent light is used to irradiate the target object and pass through the scattering medium, the speckle image obtained on the image sensor is expressed as:

[0082] ;

[0083] In the above formula, represents the speckle image obtained on the detector; represents the original target image; subscript represents the image plane; subscript Represents the plane of an object; Expressed as the point spread function of the observed target and system; It represents the angular correlation of the speckle images in front of the two incident waves, that is, the shape factor function that describes the degree of optical memory effect.

[0084] Specifically, in step 4, the system point spread function calibrated in step 2 is used for deconvolution processing to achieve scene reconstruction, and the process is as follows:

[0085] Introducing the system point spread function calibrated in step 2 into the deconvolution model, we get the following equation:

[0086] ;

[0087] In the above formula, represents the Fourier transform, is the inverse Fourier transform, represents the conjugate of the Fourier transform result; is the speckle measurement image;

[0088] like Figure 3 As shown, in the prior information learning stage of the model, the denoising score matching function is used to learn the gradient distribution of the target hologram. After the input pure noise image is predicted and corrected, the amplitude image of the reconstructed target is obtained by diffraction through the angular spectrum method. Then, the difference between the reconstructed image and the target image is calculated. Then, the hologram generation subnetwork driven by the diffraction model is used to convert the difference in the amplitude domain into the holographic domain. Subsequently, the amplitude constraint of the target image is introduced by adding the difference hologram to the hologram that has been predicted and corrected before. Finally, a high-quality target hologram is generated through continuous iterations, thereby realizing the reconstruction of the target scene.

[0089] In the hologram prior information learning stage, a generative model is introduced to learn the prior information data distribution, strengthen the generation of real samples and diversify data; the diffusion model is learned by maximum likelihood estimation, and a forward SDE process is set based on the score matching generative model, and the corresponding reverse time SDE is defined; the forward diffusion process smoothly transforms the complex data distribution into a known prior distribution by slowly injecting noise, while the reverse time SDE transforms the prior distribution into data by slowly removing noise; the score model perturbs the data distribution by adding noise; in order to sample from the data distribution, a neural network needs to be trained to estimate the logarithmic data distribution The gradient is used to solve the inverse temporal SDE to achieve the goal of generating data from noise; is the target-related prior learned during forward SDE training; due to variance explosion, SDE produces higher quality samples, which can be obtained by constructing a variance explosion-SDE and training a time-based scoring network To estimate , so that , and then replace the corresponding terms in the calculation; the actual score function is difficult to obtain, and the denoised score matching is used to replace it with ; So The Gaussian perturbation kernel centered at . Under regularization conditions, the trained network basically satisfies After the network training is completed, replace To approximately solve the inverse SDE.

[0090] In the hologram generation stage, the hologram generation algorithm is optimized to make the obtained hologram reconstruction image as close to the target image as possible. , to minimize the difference between the two; this process is expressed as:

[0091] ;

[0092] The gradient descent method is used to iteratively minimize the difference between the reproduced image and the target image, which is expressed as follows:

[0093] ;

[0094] In the above formula, represents the hologram obtained in the last iteration, is the hologram obtained in this round of iteration, represents the iteration step length; Indicates the number of iterations; is the angular spectrum propagation operator; yes The adjoint operator of represents the hologram generation transformation driven by the diffraction model; is the gradient of the data consistent term;

[0095] Among them, the diffraction model driven hologram generation subnetwork is shown in Figure 4, and the detailed process is described as follows: according to the wavelength size, each channel of the image is converted into amplitude, and then connected with the depth map in the channel direction to form the input; the input passes through the target complex amplitude generator subnetwork, and the target complex amplitude generator subnetwork predicts the amplitude distribution and phase distribution of the target field; then the target field is transmitted to the SLM field; the SLM field is decomposed into amplitude and phase and connected in the channel direction. The phase encoder subnetwork converts the connected field into a pure phase hologram;

[0096] It can be seen from the Bayesian maximum a posteriori estimation method that if the complex distribution of the prior information data in the target data set can be accurately predicted, a high-quality reproduced image can be generated; the prior information obtained by the model can be used as a regularization term in the optimization equation to limit the solution space; the model-based method also needs to solve the data consistency problem to ensure the consistency between the reproduced image and the target data.

[0097] In order to reduce the ill-posedness of the gradient descent method and obtain the optimal solution, a regularization term is added to constrain it, which is expressed as:

[0098] ;

[0099] In the above formula, Represents data consistency items, represents the regularization parameter, represents the regularization term containing target prior information;

[0100] The pure phase hologram output is achieved by iterating the following formula (6);

[0101] ;

[0102] In the above formula, represents the regularization term containing the prior information of the target of this round of iteration, represents the partial derivative operator;

[0103] The alternating optimization algorithm is used to decouple the prior information term and the data consistency term. By decoupling, equation (6) is converted into an optimization problem in which two sub-problems are updated alternately:

[0104] ;

[0105] In the above formula, the first formula represents model generation, which is the regularization part of the iterative generation process; the second formula is the data fidelity term;

[0106] The sample estimate of the next time step is provided through a fixed discretization strategy to generate the target image from the prior distribution of the target image, which is expressed as:

[0107] ;

[0108] In the above formula, is Gaussian white noise that conforms to the standard normal distribution; is a time-based scoring network, represents the noise scale of this iteration, Indicates the noise scale for the next iteration;

[0109] In the correction step, the Langevin correction algorithm is used to correct the gradient ascent direction, and equation (8) is further expressed as:

[0110] ;

[0111] In the above formula, is the noise iteration step size;

[0112] The target hologram is generated based on the modified model and the pure phase hologram data is constructed.

[0113] The technical effect of the present invention is further illustrated by an example below.

[0114] Firstly, in order to reduce background light interference and scattering effects of scatterers in complex scenes and realize imaging of natural targets through scattering media, the device uses a light-emitting diode (LED) with a wavelength of 625 nm and a bandwidth of 17 nm as an external illumination source of incoherent near-infrared light, which is located in the near-infrared window band; the light beam emitted by the light source is first adjusted through two lenses to adjust the focal position, and then the beam diameter is adjusted through an aperture to focus on the dynamically changing surface of the target; the light field is modulated by a 220-mesh ground glass scattering plate, and the resulting speckle image is collected by a scientific-grade complementary metal oxide semiconductor (sCMOS) camera with a resolution of 1920×1200, a pixel interval of 5.86 μm, and a frame rate of 40 fps, which is used to capture images carrying target information; the image collected by the camera is restored to the original dynamic target image hidden in the scattering medium through the image enhancement method and the convolution method through the scattering medium.

[0115] The specific collection steps are as follows:

[0116] Step 1: Background speckle data detection: In a natural environment, multiple frames of noise speckle data that do not contain target signals are collected and used as local noise for subsequent processing;

[0117] Step 2: System calibration: measure the point spread function at different distances to calibrate the system parameters;

[0118] Step 3, data acquisition: Under the same detection environment, collect multiple frames of speckle data containing target information and transmit them to the host computer for scene reconstruction;

[0119] Step 4: Target scene reconstruction: Use stacking averaging, background noise removal, and bilateral filtering techniques for preprocessing to enhance data quality, and use the calibrated system point spread function to achieve scene reconstruction;

[0120] Step 5, hologram solution: Combine the pre-processed scattering scene with the hologram generation algorithm to construct the hologram data in real time;

[0121] Secondly, in order to realize the naked eye 3D display of the scene, the light source used in this embodiment is a green laser with a wavelength of 532 nm; the laser light source is transmitted to the beam expander through an optical fiber, and the beam after expansion is collimated into a parallel reference light by a collimating lens, and the light field is oriented by a polarizer and a beam splitter prism; the parallel reference light after this series of processing is irradiated onto the plane of a pure phase spatial light modulator, which has a resolution of 1920×1080 pixels and an interval of 8 μm between each pixel, ensuring the accurate presentation of the hologram and high-quality 3D visual effects; then, the upper computer calculates the hologram and sends it to the spatial light modulator controller, and the light field after the spatial light modulator modulates the phase is focused by a lens and transmitted to the display optical waveguide through a beam splitter prism; the human eye can observe the information of the original target scene from the right side of the display optical waveguide; the execution steps of this part of the function are as follows:

[0122] Step 6, hologram loading: the host computer loads the calculated hologram into the spatial light modulator;

[0123] Step 7, holographic display: when executing step 1, light up the laser;

[0124] In order to test the imaging and display capabilities of the device, an incoherent LED light source is used to illuminate the Figure 5 (a) shows the target to be imaged by the resolution target. Figure 5 The second line pair shown in (b) is the target area. After the light field emitted by the target passes through the scattering medium, the sCMOS camera detector can only observe Figure 5 The messy speckle image shown in (c) loses a lot of target information and cannot identify the target. After image reconstruction, we get Figure 5 The imaging result shown in (d) is shown in FIG. 1 . It can be seen that the method of the present invention can effectively identify all the information of the second group of line pairs and can effectively reconstruct the line pairs and numbers. The result is similar to Figure 5 The imaging area shown in (b) is basically consistent. After the computational generation of holographic methods, the Figure 5 The pure phase hologram shown in (e) is loaded onto a spatial light modulator and illuminated by a coherent light source to obtain Figure 5 The holographic reconstruction result is shown in (f).

[0125] Depend on Figure 5 The five-frame reconstruction results shown in the figure show that the augmented reality device provided by the present invention can overcome the adverse effects of scattering media, has strong resolution capabilities, and can still achieve clear imaging for dynamic targets. At the same time, the holographic reproduction results show that the device provided by the present invention can achieve high-quality holographic display. The entire system achieves the purpose of integrating scattering medium imaging and holographic display.

[0126] In order to test the minimum imaging resolution that the device can achieve, the third group of right line pairs 1-4 of the resolution test image are used as imaging targets and scanned from top to bottom in turn to perform resolution testing, such as Figure 6 As shown, Figure 6 (a) is the resolution target; Figure 6 (b) is the target to be imaged; Figure 6 (c) is the measured data of the corresponding range collected by the sCMOS camera. Figure 6 (d) is the reconstruction result of the measured data. It can be seen from the reconstruction results that when the imaging target range is within line pair 1, a clear result can be reconstructed. As the target range becomes smaller, the reconstruction result becomes more and more blurred until the resolution limit of the device is reached at line pair 4.

[0127] Figure 6 The results of the resolution test experiment in show that the augmented reality device provided by the present invention can effectively distinguish the line pair target with the serial number 4 in the third group in the resolution test image. Under this condition, the resolution of the device is 11.314 lp / mm, and the corresponding line pair width is 88.39 μm, which proves that the augmented reality device provided by the present invention can effectively distinguish fine targets and achieve high-resolution imaging through scattering media.

[0128] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An augmented reality device integrating perception and display through a scattering medium, characterized in that: It includes a modulated lighting module, an information perception and control module, and an augmented reality module; The modulated illumination module utilizes an incoherent light source in the near-infrared band to illuminate the target to collect incident light distribution information; The modulated lighting module includes an incoherent near-infrared light source and an illumination target. The light emitted by the incoherent near-infrared light source interacts with the illumination target and is disturbed by the scatterer to form a speckle image. The speckle image is collected and processed by the information perception and control module. The information perception and control module uses a host computer to collect speckle images and execute an image reconstruction algorithm, then sends the images to the spatial light modulator controller and synchronously controls the red, green and blue lasers to achieve three-dimensional color light field modulation control; The information perception and control module includes a filter, an image sensor, a host computer and a phase-type spatial light modulator controller; the image sensor is responsible for collecting speckle images, and the filter is placed in front of the image sensor to filter out stray background light interference; the host computer uses a point spread function to perform deconvolution processing on the speckle image collected by the image sensor, obtains a pure phase-type hologram, and then sends it to the phase-type spatial light modulator controller, and synchronously emits it to the red, green and blue three-color lasers in a specific timing sequence to realize three-dimensional color light field modulation control; The augmented reality module realizes the final three-dimensional image observation by loading the pure phase hologram; The augmented reality module includes a spatial light modulator, a beam splitting cube, a red, green and blue laser, a collimating lens and a display optical waveguide; the spatial light modulator is loaded with a pure phase hologram and works synchronously with the red, green and blue laser; The reference light diffracted by the spatial light modulator is transmitted to the display optical waveguide through a beam splitter prism together with the parallel reference light converted by the collimating lens, so that the human eye can observe the three-dimensional information of the original target scene.

2. A method for integrated augmented reality through perception and display of a scattering medium, using the augmented reality device as claimed in claim 1, characterized in that: The following steps are involved: Step 1: Background speckle data detection: In a natural environment, multiple frames of noise speckle data that do not contain target signals are collected and used as local noise for subsequent processing; Step 2: System calibration: measure the point spread function at different distances to calibrate the system parameters; Step 3: Data collection: Under the same detection environment, multiple frames of speckle data containing target information are collected and transmitted to the host computer for scene reconstruction; Step 4: Target scene reconstruction: Use stacking averaging, background noise removal, and bilateral filtering techniques to preprocess the collected data to enhance data quality, and use the system point spread function calibrated in step 2 to perform deconvolution processing to achieve scene reconstruction; Step 5, pure phase hologram solution: Use grayscale image colorization technology to convert the pre-processed scattering scene into three-wavelength color data, and combine it with the pure phase hologram generation algorithm to realize the construction of pure phase hologram data; Step 6, hologram loading: the host computer loads one channel of the calculated pure phase hologram into the spatial light modulator; Step 7, holographic display: while executing step 6, synchronously light up the laser of the corresponding band; Step 8: Repeat steps 6 to 7 at a certain refresh frequency to realize color hologram display in the form of time division multiplexing.

3. The method for integrated augmented reality through perception and display of a scattering medium according to claim 2, characterized in that: The process of collecting speckle data in step 3 is as follows: After the incoherent near-infrared light source is irradiated on the illumination target and passes through the scattering medium, the corresponding speckle pattern will be received on the image sensor. The scattering within a certain range is regarded as useful information, and the scattering outside the range is regarded as useless noise. At this time, if a beam of incoherent light is used to irradiate the target object and pass through the scattering medium, the speckle image obtained on the image sensor is expressed as: ; In the above formula, represents the speckle image obtained on the detector; represents the original target image; subscript represents the image plane; the subscript Represents the plane of an object; Expressed as the point spread function of the observed target and system; It represents the angular correlation of the speckle images in front of the two incident waves, that is, the shape factor function that describes the degree of optical memory effect.

4. The method for integrated perception and display of augmented reality through a scattering medium according to claim 2, characterized in that: In step 4, the system point spread function calibrated in step 2 is used for deconvolution processing to achieve scene reconstruction. The process is as follows: Introducing the system point spread function calibrated in step 2 into the deconvolution model, we get the following equation: ; In the above formula, represents the Fourier transform, is the inverse Fourier transform, represents the conjugate of the Fourier transform result; is the speckle measurement image; In the prior information learning stage of the model, the denoising fractional matching function is used to learn the gradient distribution of the target hologram. After the input pure noise image is predicted and corrected, the amplitude image of the reproduced target is obtained by diffraction through the angular spectrum method. Then, the difference between the reproduced image and the target image is calculated. The hologram generation subnetwork driven by the diffraction model is used to convert the difference in the amplitude domain into the holographic domain. Subsequently, the amplitude constraint of the target image is introduced by adding the difference hologram to the hologram that has been predicted and corrected before. Finally, a high-quality target hologram is generated through continuous iterations, thereby achieving target scene reconstruction.

5. The method for integrated perception and display of augmented reality through a scattering medium according to claim 2, characterized in that: The process described in step 5 is to combine the pure phase hologram generation algorithm to construct the pure phase hologram data as follows: In the hologram generation stage, by optimizing the hologram generation algorithm, the obtained hologram The reproduced image is as close as possible to the target image , to minimize the difference between the two; this process is expressed as: ; The gradient descent method is used to iteratively minimize the difference between the reproduced image and the target image, which is expressed as follows: ; In the above formula, represents the hologram obtained in the last iteration, is the hologram obtained in this round of iteration, represents the iteration step length; Indicates the number of iterations; is the angular spectrum propagation operator; yes The adjoint operator of represents the hologram generation transformation driven by the diffraction model; is the gradient of the data consistent term; In order to reduce the ill-posedness of the gradient descent method and obtain the optimal solution, a regularization term is added to constrain it, which is expressed as: ; In the above formula, Represents data consistency items, represents the regularization parameter, represents the regularization term containing target prior information; The pure phase hologram output is achieved by iterating the following formula (6); ; In the above formula, represents the regularization term containing the prior information of the target of this round of iteration, represents the partial derivative operator; The alternating optimization algorithm is used to decouple the prior information term and the data consistency term. By decoupling, equation (6) is converted into an optimization problem in which two sub-problems are updated alternately: ; In the above formula, the first formula represents model generation, which is the regularization part of the iterative generation process; the second formula is the data fidelity term; The sample estimate of the next time step is provided through a fixed discretization strategy to generate the target image from the prior distribution of the target image, which is expressed as: ; In the above formula, is Gaussian white noise that conforms to the standard normal distribution; is a time-based scoring network, represents the noise scale of this iteration, Indicates the noise scale for the next iteration; In the correction step, the Langevin correction algorithm is used to correct the gradient ascent direction, and equation (8) is further expressed as: ; In the above formula, is the noise iteration step size; The target hologram is generated based on the modified model and the pure phase hologram data is constructed.

Citation Information

Patent Citations

  • Augmented reality holographic display method based on complex amplitude modulation

    CN106842575A

  • Compact augmented reality near-to-eye display device

    CN110488490A

  • Color holographic near-to-eye AR display system based on white light illumination and color hologram calculation method

    CN112882228A

  • Holographic near-to-eye display system based on electric control polarization modulator and eye pupil box expansion method

    CN113608354A

  • Applications of diffuse medium imaging

    US20190072897A1