Infrared turbulence degraded image simulation method based on unreal engine

Abstract

The invention provides an infrared turbulence degraded image simulation method based on an unreal engine. The method comprises the following implementation steps: constructing a three-dimensional mountain simulation scene based on the unreal engine; generating a fuzzy kernel texture image based on the calculated two-dimensional point spread function; and rendering the three-dimensional mountain scene based on an unreal engine, then carrying out convolution on a rendering result and the fuzzy kernel texture image, and obtaining a simulated infrared turbulence degraded image through a convolution result. According to the invention, parallel convolution is carried out on each frame of image after three-dimensional mountain scene rendering and the blurred kernel texture image generated based on the two-dimensional point spread function, so that the obtained real-time turbulence blurring effect of the mountain scene well maintains the details of the terrain; meanwhile, the influence of spatial frequency response information of an imaging system and attenuation effect information of atmospheric turbulence on image detail transmission on the image is considered, the simulation precision is effectively improved, the complexity of generating the real-time turbulence fuzzy effect of the mountain scene is reduced through parallel convolution, and good real-time performance is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image simulation, and relates to an infrared turbulent degradation image simulation method based on the Unreal Engine, which can be applied to fields such as aerospace, security monitoring, and meteorological monitoring. Background Art

[0002] Infrared imaging technology has wide applicability and powerful functionality. Its main features include applicability, strong anti-interference ability, strong environmental adaptability, etc. However, obtaining real turbulent images requires specific environmental conditions, which not only requires high-precision measurement equipment but also needs to overcome the limitations of environmental conditions. Therefore, obtaining real turbulent images requires high costs. Infrared imaging simulation technology involves simulating the infrared radiation distribution of a scene in terms of time, space, spectrum, and radiation quantity. This technology can generate infrared images that are difficult to obtain under restricted conditions. Images are the most commonly used information carriers in infrared imaging systems. High-quality and high-resolution images can better convey the information of the target. However, atmospheric turbulence can significantly reduce the imaging quality of optical systems. The non-uniformity of atmospheric density and temperature will cause changes in the atmospheric refractive index. In the spatial dimension, the farther away from the target, the longer the exposure time, the more severely affected by atmospheric disturbances, and the blurrier the image; in the time dimension, random jitters will occur in different regions of the image over time. Therefore, atmospheric turbulence will cause phenomena such as blurring, noise, phase distortion, and content loss in the images collected by infrared imaging systems.

[0003] Infrared turbulent degradation image simulation generates degraded images that are highly close to real observation conditions by constructing an atmospheric turbulence physical model and combining the transfer characteristics of the infrared imaging system to simulate the random interference effects generated by atmospheric turbulence during the infrared signal transmission process.

[0004] The existing typical simulation of infrared turbulent degraded images is implemented based on the OGRE engine. This method first builds a three-dimensional terrain scene containing infrared targets and imaging systems based on the OGRE engine, generates a point spread function through the atmospheric modulation transfer function, obtains the blur kernel through sampling and normalization processing, then converts the blur kernel into a texture image format, and performs a convolution operation with the rendering result of the simulation scene. This operation process is executed based on the serial computing mode of the CPU, and finally, a simulation image with turbulent degradation effect is output through the simulation imaging system. This method uses the atmospheric modulation transfer function to generate a point spread function that conforms to the physical laws of turbulence. The degraded image simulated using this point spread function can improve the simulation accuracy compared to the degraded image simulated by traditional simulation methods. Then, the point spread function is sampled and normalized into a blur kernel texture image, and then real-time rendering is performed with the built three-dimensional scene, solving the problem of poor real-time performance in traditional static scene simulation. However, in order to ensure the smooth operation of the scene, during the process of building the three-dimensional terrain scene, the terrain details will be compressed and simplified to a certain extent, resulting in differences from the high-frequency details of the real terrain. Moreover, the generated point spread function only considers the atmospheric modulation transfer function and ignores the influence of the imaging system on the imaging result, thus affecting the simulation accuracy. At the same time, this method performs convolution on the texture image and the rendering result of the simulation scene through the serial computing mode of the CPU, and the calculation process is relatively complex and time-consuming, affecting the real-time performance of the simulation. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects existing in the above-mentioned prior art, and propose a method for simulating infrared turbulent degraded images based on the Unreal Engine, which is used to solve the technical problems of poor accuracy and real-time performance of the simulation results in the prior art.

[0006] To achieve the above purpose, the technical solution adopted by the present invention includes the following steps:

[0007] (1) Construct a three-dimensional mountain simulation scene based on the Unreal Engine:

[0008] Construct a three-dimensional mountain simulation scene based on the Unreal Engine, including a virtual three-dimensional mountain scene, and a three-dimensional mountain simulation scene with multiple infrared radiation characteristic targets and imaging systems added.

[0009] (2) Calculate the two-dimensional point spread function:

[0010] Calculate the comprehensive modulation transfer function MTF through the modulation transfer function MTF of the imaging system diff and the modulation transfer function MTF of atmospheric turbulence turb , and perform an inverse Fourier transform on the MTF optics , to obtain the two-dimensional point spread function h in the spatial domain; optics

[0011] ​(3) Generating a texture image of a blur kernel based on a two-dimensional point spread function:

[0012] After radially sampling the two-dimensional point spread function h, normalize each sampled data, and save the two-dimensional blur kernel H after normalizing all sampled data as a texture image in EXR format with the same specifications;

[0013] (4) Convolving the rendering result with the blur kernel texture image after rendering a three-dimensional mountain scene based on the Unreal Engine:

[0014] Based on the Unreal Engine, perform multiple real-time renderings of the three-dimensional mountain scene, and perform parallel convolution on each frame of the rendered image and the blur kernel texture image to obtain the real-time turbulent blur effect of the mountain scene;

[0015] (5) Obtaining the simulation result of the infrared turbulent degradation image:

[0016] Image the real-time turbulent blur effect of the mountain scene based on the imaging system to obtain an infrared turbulent degradation image.

[0017] Compared with the prior art, the present invention has the following advantages: the real-time turbulent blur effect of the mountain scene;

[0018] 1. The three-dimensional mountain simulation scene constructed by the present invention contains the height data in the digital elevation model of the real mountain scene. The generated two-dimensional point spread function contains the spatial frequency response information of the imaging system itself and the attenuation effect information of atmospheric turbulence on the transmission of image details. By performing parallel convolution on each frame of the image after rendering the three-dimensional mountain scene and the blur kernel texture image generated based on the two-dimensional point spread function, the obtained real-time turbulent blur effect of the mountain scene not only better preserves the details of the terrain but also considers the influence of the imaging system on the image, effectively improving the simulation accuracy.

[0019] 2. The present invention performs parallel convolution on the pixel points in the rendered image and the blur kernel texture image based on multiple threads parallel in the GPU parallel architecture in the Unreal Engine, avoiding the defect of long time consumption caused by the complex calculation process in serial convolution using the CPU in the prior art, and having good real-time performance. Description of the Drawings

[0020] Figure 1 It is a flowchart for implementing the present invention.

[0021] Figure 2 It is a blur kernel texture map of an embodiment of the present invention.

[0022] Figure 3 It is a rendering of the three-dimensional mountain scene of the present invention.

[0023] Figure 4This is the simulation result diagram of the infrared turbulent degraded image of the present invention. Specific embodiments

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0025] Refer to Figure 1 , the present invention includes the following steps:

[0026] Step 1) Use the Unreal Engine to construct a finely modeled three-dimensional mountain scene;

[0027] Based on the digital elevation model obtained from satellite remote sensing, use the terrain processing software GDAL library to extract the elevation data of the real mountain in the model and store it in a 16-bit grayscale height map. The grayscale height map shows the undulation of the mountain terrain with the depth of the grayscale value. Apply virtual geometry technology in the terrain editor system of the Unreal Engine to parse the grayscale height map. According to the change of the height data in the grayscale height map, reasonably determine the number of triangular patches. In the area with a high grayscale value, increase the number of triangular patches; in the area with a low grayscale value, appropriately reduce the number of triangular patches. Build the scene through the combination of numerous triangular patches to ensure the geometric consistency between the three-dimensional mountain scene and the real terrain.

[0028] Add multiple infrared radiation feature targets in this simulation scene. According to the temperature T tem and spectral reflectivity ρ(λ) of the target surface area, divide the thermal radiation area, and then store the values of T tem and ρ(λ) in the green and blue channels of the target UV texture map Figure 3 respectively. Dynamically sample the UV texture map through the Unreal Engine material system and calculate the infrared radiation luminance L(λ) of the infrared target. Finally, present the radiation distribution of the infrared target through UV mapping technology to obtain the infrared radiation feature target, where the formula for L(λ) is:

[0029] L(λ) = ρ(λ)·T tem

[0030] λ represents the infrared wavelength of the simulation scene.

[0031] Step 2) Calculate the two-dimensional point spread function:

[0032] Through the combined calculation of the modulation transfer function MTF diff of the imaging system and the modulation transfer function MTF turb of atmospheric turbulence, the comprehensive modulation transfer function MTF optics can be obtained. This function reflects both the spatial frequency response of the imaging system itself and the attenuation effect of atmospheric turbulence on the transmission of image details. The modulation transfer function MTF diffDescribes the ability of the imaging system to maintain contrast and details at different spatial frequencies during the process from the object to the image; while the atmospheric turbulence modulation transfer function MTF turb Quantitatively characterizes the phase perturbation and high-frequency attenuation effects caused by turbulence in the atmosphere, which usually lead to image blurring and detail loss. The comprehensive modulation transfer function MTF optics The calculation formula is:

[0033] MTF optics = MTF diff ×MTF turb

[0034]

[0035] v nor = f x / f oco

[0036] f oco = D / λ

[0037] Where v nor Represents the normalized frequency of the imaging system, exp{·} represents the exponential operation with e as the base, D represents the pupil diameter of the imaging system, r 0 Represents the atmospheric coherence length, α represents the long and short exposure coefficient, R represents the distance between the imaging system and the infrared target, Represents the atmospheric refractive index structure constant, f x Represents the spatial angular frequency of the imaging system, f oco Represents the cut-off frequency of the imaging system.

[0038] Performs the inverse Fourier transform on the obtained MTF optics Among them, the formula for performing the inverse Fourier transform on the comprehensive modulation transfer function MTF optics In the frequency domain, the MTF at the u-th row and v-th column, MTF optics (u, v) represents the result is:

[0039]

[0040]

[0041] Where h(x, y) represents the two-dimensional point spread function at the x-th row and y-th column in the spatial domain of size N×N, ∫{·} represents the integral operation, and du and dv represent the accumulation operation.

[0042] Step 3) Generate the texture image of the blur kernel based on the two-dimensional point spread function:

[0043] Perform m - times of radial sampling outward along the positive and negative directions of x and y respectively from the numerical peak center point of the two - dimensional point - spread function h in the spatial domain, and normalize all the sampled data to obtain a two - dimensional blur kernel H of size (2m + 1)×(2m + 1). Then save H as a texture image in EXR format with the same specification as H.

[0044] The calculation formula for the blur kernel value H[i,j] in the i - th row and j - th column is as follows:

[0045]

[0046] where Δx and Δy respectively represent the sampling step sizes along the x and y directions, (i,j)∈[0,2m + 1], h(iΔx,jΔy) represents the value of the point - spread function at the i·Δx - th row and j·Δy - th column.

[0047] In this embodiment, m = 4, and the size of the obtained two - dimensional blur kernel H is 9×9. The corresponding blur - kernel texture image is as Figure 2 shown. It can be clearly observed from the figure that the value in the central region of the blur - kernel texture image reaches the highest, and the value gradually decreases as it spreads to the surrounding areas, showing a distribution trend of high in the center and low in the periphery.

[0048] Step 4) After rendering the three - dimensional mountain scene based on the Unreal Engine, perform convolution on the rendering result and the blur - kernel texture image:

[0049] The Unreal Engine rasterizes the geometric data in the three - dimensional scene into two - dimensional pixels in a rendering window with a resolution of (P row ,P col ), and then uses the pixel shader and fragment shader to color the pixels to achieve real - time scene rendering. The scene - rendering frame rate is T, and T is affected by the hardware performance and the resolution of the rendering window. The convolution operation is performed on each frame of the rendering result and the blur - kernel texture image to simulate the blurring effect of turbulence. This convolution operation is based on the parallel architecture of the GPU. Each pixel point in the rendered image is assigned to a parallel thread in the parallel architecture of the GPU for convolution. Multiple parallel threads can perform simultaneously, realizing the synchronous convolution of multiple pixel points, thereby significantly reducing the operation latency, accelerating the overall rendering process, and ensuring the efficiency and accuracy of real - time image processing. The convolution formula for the pixel I(p,q,t) at the p - th row and q - th column of the t - th frame of the rendered image and the blur - kernel value H[i,j] at the i - th row and j - th column is as follows:

[0050]

[0051] where, (p,q)∈(P row ,P col),where \(t\in[1,T]\), \(S(p - i,q - j,t)\) represents the value of the pixel at the \((p - i)\)-th row and \((q - j)\)-th column in the rendering result of the \(t\)-th frame.

[0052] The present invention uses the Unreal Engine to render a three-dimensional mountain scene, and the result is as Figure 3 shown. It can be clearly observed from the figure that the mountain contour, grass and trees of the mountain scene constructed based on the Unreal Engine are clearly visible, and the details of the target shape in the scene are specific and rich.

[0053] Step 5) Obtain the simulation result of the infrared turbulent degradation image:

[0054] Save the values of the pixels in the real-time turbulent blur effect of the mountain scene observed in the viewport of the imaging system in the infrared turbulent degradation image as Figure 4 shown. Compared with Figure 3 the original, after being affected by the degradation effect of turbulent blur, the infrared turbulent degradation image as a whole shows an obvious blurring phenomenon. The clarity of the mountain contour, grass and trees in the mountain scene is significantly reduced, and the details of the target shape in the scene are lost. This change is consistent with the influence effect of turbulence on infrared imaging.

Claims

1. A method for simulating infrared turbulence-degraded images based on Unreal Engine, characterized in that The steps include: (1) Building a 3D mountain simulation scene based on Unreal Engine: Based on Unreal Engine, a virtual 3D mountain scene is built, as well as a 3D mountain simulation scene with multiple infrared radiation feature targets and imaging systems added; (2) Calculate the two-dimensional point spread function: Modulation transfer function (MTF) of the imaging system diff Modulation transfer function (MTF) of atmospheric turbulence turb Calculate the comprehensive modulation transfer function MTF optics , and MTF optics Perform inverse Fourier transform to obtain the two-dimensional point spread function h in the spatial domain; (3) Generate a texture image of the blur kernel based on a two-dimensional point spread function: After radial sampling of the two-dimensional point spread function h, each sampled data is normalized, and the two-dimensional blur kernel H after all the normalized sampled data is saved as a texture image in EXR format with the same specifications; (4) After rendering the 3D mountain scene based on the Unreal Engine, the rendering result is convolved with the blur kernel texture image: Based on the Unreal Engine, the three-dimensional mountain scene is rendered multiple times in real time, and each rendered frame is convolved with the blur kernel texture image in parallel to obtain the real-time turbulence blur effect of the mountain scene; (5) Obtain the simulation results of infrared turbulence degradation images: Based on the imaging system, the real-time turbulence blurring effect of the mountain scene is imaged to obtain an infrared turbulence-degraded image.

2. The method according to claim 1, characterized in that The three-dimensional mountain scene described in step (1) is constructed by: The height data in the digital elevation model of the real mountain scene is converted into a grayscale height map, and through the Unreal Engine terrain system, multiple virtual triangular faces are created according to the height data in the grayscale height map, and then the multiple triangular faces are combined to form a three-dimensional mountain scene with the same geometric dimensions as the real mountain scene.

3. The method according to claim 1, characterized in that The comprehensive modulation transfer function MTF described in step (2) optics , the calculation formula is: MTF optics =MTF diff ×MTF turb v nor =f x / f oco f oco =D / λ Among them, v nor represents the normalized frequency of the imaging system, exp{·} represents the exponential operation with e as the base, D represents the pupil diameter of the imaging system, r0 represents the atmospheric coherence length, λ represents the infrared wavelength in the simulation scene, α represents the long and short exposure factor, and R represents the distance between the imaging system and the infrared target. represents the atmospheric refractive index structure constant, f x represents the spatial angular frequency of the imaging system, f oco Represents the cutoff frequency of the imaging system.

4. The method according to claim 3, characterized in that The MTF described in step (2) optics Perform inverse Fourier transform, where the MTF optics The comprehensive modulation transfer function MTF of the uth row and vth column in the medium frequency domain optics (u,v) represents the result of inverse Fourier transform: where h(x,y) represents the two-dimensional point spread function at the xth row and yth column in the N×N spatial domain, ∫{·} represents the integration operation, and du and dv represent the accumulation operation.

5. The method according to claim 1, characterized in that: After radial sampling of the two-dimensional point spread function h in step (3), each sampled data is normalized. The implementation steps are: Take the peak value center point of the two-dimensional point spread function h in the spatial domain and perform m radial samplings outward along the positive and negative directions of x and y, respectively, and normalize each sampled data to obtain a two-dimensional blur kernel H of size (2m+1)×(2m+1), where the formula for normalizing the sampled data in the i-th row and j-th column is: Where H[i,j] represents the blur kernel after normalization of the sampled data in the i-th row and j-th column, Δx and Δy represent the sampling steps along the x and y directions respectively, (i,j)∈[0,2m+1], h(iΔx,jΔy) represents the point spread function value at the i·Δxth row and the j·Δyth column.

6. The method according to claim 1, characterized in that The real-time rendering of the three-dimensional mountain scene based on Unreal Engine described in step (4) is implemented by the following steps: Based on the pixel shader and fragment shader in the Unreal Engine, the three-dimensional mountain scene is rendered T times in real time, and the resolutions of rows and columns in the rendering window are P row , P col T frames of rendered images, where T ≥ [60].

7. The method according to claim 6, characterized in that The step (4) of performing parallel convolution of the rendering result with the blur kernel texture image frame by frame is specifically performed by simultaneously convolving the pixel points in the rendered image with the blur kernel texture image through multiple threads in parallel of the GPU parallel architecture in the Unreal Engine, wherein the convolution formula for the pixel I(p,q,t) in the pth row and qth column of the rendered image of the tth frame and the blur kernel H[i,j] in the ith row and jth column is: Among them, (p,q)∈(P row ,P col ), t∈[1,T], S(pi,qj,t) represents the value of the (pi)th row and (qj)th column in the rendering result of the tth frame.

8. The method according to claim 1, characterized in that The imaging of the real-time turbulence blur effect of the mountain scene described in step (5) is specifically carried out in a method in which the imaging system saves the value of each pixel in the real-time turbulence blur effect of the mountain scene observed in its viewport, and obtains an infrared turbulence degraded image with the same resolution as the imaging system.

Images