Real-time simulation method and system for pneumatic fuzzy effect

Abstract

The invention provides a real-time simulation method and system for a pneumatic fuzzy effect, and the method comprises the steps: 1, creating a frame buffer object, and drawing the frame buffer data of each frame of an infrared scene rendered by a main camera on a texture; 2, the rendered texture image data are read, and pneumatic fuzzy effect processing of the image data is completed on a GPU through a shader; and step 3, setting a slave camera viewport and a projection matrix, and rendering the image data processed by the pneumatic blurring effect to a scene rendering window to complete real-time simulation of the pneumatic blurring effect. According to the method, the problem of real-time simulation of the pneumatic fuzzy effect in infrared guidance semi-physical simulation is solved, the rendered infrared scene frame cache data is rendered to the texture, and fuzzy effect processing is carried out in parallel by utilizing the GPU, so that the requirement of real-time simulation is met.

Description

Technical Field

[0001] The invention relates to the technical field of infrared guidance control semi-physical simulation, and in particular to a real-time simulation method and system for aerodynamic fuzzy effect. Background Art

[0002] Optical imaging detection technology is an important development direction in the field of precision guidance. However, due to the interaction between the aircraft and the atmosphere during high-speed flight, the temperature of the hood increases, the gas density around the hood changes, the gas refractive index pulsation of the flow field or the high temperature causes the detection window to deform, which in turn causes the phase difference of the target or scene in the optical imaging system inside the aircraft hood to increase sharply, resulting in a blur effect. This aerodynamic blur effect has an adverse effect on infrared imaging terminal guidance, reducing the aircraft's ability to detect, track and identify targets.

[0003] With the development of simulation technology, the design, verification, identification and evaluation of infrared imaging guidance detectors are increasingly dependent on semi-physical simulation technology. Semi-physical simulation is a powerful means of aircraft development, which has the advantages of improving development quality, shortening development cycle and saving development funds. The infrared guidance semi-physical simulation system mainly provides dynamic infrared scenes similar to actual combat processes for infrared imaging detectors, simulating the field environment under a variety of complex conditions to test the performance and adaptability of infrared imaging detectors in complex optoelectronic environments. One of the most important issues that cannot be ignored in the semi-physical simulation system is real-time performance. The constructed infrared scene needs to be rendered within the specified time. At present, there are two technical approaches for semi-physical simulation of aero-optical effects at home and abroad: (1) Using digital image processing technology to realize the simulation of aero-optical effects (Shi Ketian, Cheng Xiaoli, Ma Handong. Numerical simulation of aero-optical effects of flow field around optical hood [J]. Infrared and Laser Engineering, 2010, 39(1): 6-11, Zhu Zhengtian. Analysis of aero-optical effects of turbulent flow field of high-speed aircraft hood [D]. Nanjing: Nanjing University of Science and Technology, 2016.); (2) Using hardware to realize the modulation of aero-optical effects of infrared physical radiation (an aero-optical effect simulation device CN201810421662.4, a method and system for simulating aero-optical effects CN201310193486.0). However, the main problem with the above two methods is that the traditional method of light tracing to obtain aero-optical effects and then processing the images is very inefficient and cannot meet the real-time requirements of semi-physical simulation. Using hardware to achieve aero-optical effects increases the design cost and complexity on the one hand, and increases the load of the turntable on the other hand, reducing the response performance of the turntable in the semi-physical simulation. Summary of the invention

[0004] In view of the defects in the prior art, an object of the present invention is to provide a real-time simulation method and system for aerodynamic blur effect.

[0005] The real-time simulation method of aerodynamic blur effect provided by the present invention comprises:

[0006] Step 1: Create a frame buffer object and draw the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture;

[0007] Step 2: Read the rendered texture image data and use the shader on the GPU to complete the aerodynamic blur effect processing of the image data;

[0008] Step 3: Set the camera viewport and projection matrix, render the image data processed by the aerodynamic blur effect to the scene rendering window, and complete the real-time simulation of the aerodynamic blur effect.

[0009] Preferably, the step 1 comprises:

[0010] The main camera's viewport size and projection matrix are set based on the aircraft's infrared detector's field of view and resolution. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the maximum resolution is used as the main camera's viewport size.

[0011] Create a frame buffer object FBO and a texture object, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

[0012] Preferably, the step 2 comprises:

[0013] Create a quad node according to the resolution of the infrared detector, set the texture mode of the quad rendering state node, bind the rendered frame buffer object to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is executed by the image processor GPU to process vertices and pixels to generate the final image;

[0014] Theoretical calculation method is used to generate a simulated image with aerodynamic blur effect. The simulation process is described as follows:

[0015]

[0016] Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, h(x,y) is the point spread function PSF generated by the atmospheric turbulence effect; (x,y) is the position of the original image;

[0017] There are multiple vortices in the turbulent field. The turbulence statistics theory is used for simulation. For a single vortex, the point spread function PSF is as follows:

[0018]

[0019] Where: h z (x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ zxz , yz h z (x,y) fuzzy factor in x and y directions;

[0020] The expression of random vortex intensity is:

[0021]

[0022] Where: k z is the random intensity factor of the zth vortex; a is the scale parameter of the distribution;

[0023] For the entire atmospheric turbulence, the overall point spread function expression is:

[0024]

[0025] Where: Z is the number of vortices in the turbulent field.

[0026] Preferably, the process of generating an aerodynamic blur effect image using a generative adversarial network is as follows:

[0027] Define the network structure and loss function of the generator and discriminator. The expressions are as follows:

[0028]

[0029] Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ;

[0030] The measured original images and the measured images or simulated images with aerodynamic blur effects are used as training samples to train the generator and the discriminator. The training process is as follows: randomly initialize the parameters of the generator and the discriminator; the generator receives the noise vector and converts it into a data sample; the discriminator receives the samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively; for the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified; for the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator; repeat the above process iteratively, and in each round of iteration, the two networks change their weights in turn until convergence;

[0031] After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output; after the convolution kernel parameters are output, they are passed into the shader, and the image is calculated using the GPU rendering pipeline. The calculation process is: when performing convolution calculation on a pixel of the original infrared scene image, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. After the calculation of all pixel values ​​of the original infrared scene image is completed, the infrared scene processed with a blurred effect can be generated.

[0032] Preferably, the step 3 comprises:

[0033] According to the resolution of the aircraft's infrared detector, set the viewport of the camera, and set the two-dimensional orthographic projection matrix and observation matrix;

[0034] 2D orthographic projection matrix Mat p Contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows:

[0035]

[0036] Set the observation matrix to the identity matrix Mat v , which is expressed as follows:

[0037]

[0038] Add a quadrilateral node to the scene node and place it at the origin of the three-dimensional space. Set the camera position and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect.

[0039] The real-time simulation system of aerodynamic blur effect provided by the present invention comprises:

[0040] Module M1: Create a frame buffer object and draw the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture;

[0041] Module M2: reads the rendered texture image data and uses the shader on the GPU to complete the aerodynamic blur effect processing of the image data;

[0042] Module M3: Set the camera viewport and projection matrix, render the image data processed by the aerodynamic blur effect to the scene rendering window, and complete the real-time simulation of the aerodynamic blur effect.

[0043] Preferably, the module M1 comprises:

[0044] The main camera's viewport size and projection matrix are set based on the aircraft's infrared detector's field of view and resolution. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the maximum resolution is used as the main camera's viewport size.

[0045] Create a frame buffer object FBO and a texture object, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

[0046] Preferably, the module M2 comprises:

[0047] Create a quad node according to the resolution of the infrared detector, set the texture mode of the quad rendering state node, bind the rendered frame buffer object to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is executed by the image processor GPU to process vertices and pixels to generate the final image;

[0048] Theoretical calculation method is used to generate a simulated image with aerodynamic blur effect. The simulation process is described as follows:

[0049]

[0050] Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, h(x,y) is the point spread function PSF generated by the atmospheric turbulence effect; (x,y) is the position of the original image;

[0051] There are multiple vortices in the turbulent field. The turbulence statistics theory is used for simulation. For a single vortex, the point spread function PSF is as follows:

[0052]

[0053] Where: h z(x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ xz , yz h z (x,y) fuzzy factor in x and y directions;

[0054] The expression of random vortex intensity is:

[0055]

[0056] Where: k z is the random intensity factor of the zth vortex; a is the scale parameter of the distribution;

[0057] For the entire atmospheric turbulence, the overall point spread function expression is:

[0058]

[0059] Where: Z is the number of vortices in the turbulent field.

[0060] Preferably, the process of generating an aerodynamic blur effect image using a generative adversarial network is as follows:

[0061] Define the network structure and loss function of the generator and discriminator. The expressions are as follows:

[0062]

[0063] Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ;

[0064] The measured original images and the measured images or simulated images with aerodynamic blur effects are used as training samples to train the generator and the discriminator. The training process is as follows: randomly initialize the parameters of the generator and the discriminator; the generator receives the noise vector and converts it into a data sample; the discriminator receives the samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively; for the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified; for the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator; repeat the above process iteratively, and in each round of iteration, the two networks change their weights in turn until convergence;

[0065] After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output; after the convolution kernel parameters are output, they are passed into the shader, and the image is calculated using the GPU rendering pipeline. The calculation process is: when performing convolution calculation on a pixel of the original infrared scene image, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. After the calculation of all pixel values ​​of the original infrared scene image is completed, the infrared scene processed with a blurred effect can be generated.

[0066] Preferably, the module M3 comprises:

[0067] According to the resolution of the aircraft's infrared detector, set the viewport of the camera, and set the two-dimensional orthographic projection matrix and observation matrix;

[0068] 2D orthographic projection matrix Mat p Contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows:

[0069]

[0070] Set the observation matrix to the identity matrix Mat v , which is expressed as follows:

[0071]

[0072] Add a quadrilateral node to the scene node and place it at the origin of the three-dimensional space. Set the camera position and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] The present invention introduces the Render to Texture (RTT) technology in the off-screen rendering mode in the traditional infrared scene rendering process, proposes a real-time simulation method of aerodynamic blur effect, and solves the real-time simulation problem of aerodynamic blur effect in infrared guidance semi-physical simulation. Its purpose is to render the rendered infrared scene frame cache data onto the texture and use the GPU to perform blur effect processing in parallel, thereby meeting the needs of real-time simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0076] Figure 1 It is the principle diagram of the real-time simulation method of aerodynamic fuzzy effect;

[0077] Figure 2 It is a flow chart of aerodynamic blurring effect generation based on generative adversarial networks;

[0078] Figure 3a and Figure 3b They are the network structure diagrams of the generator and the discriminator respectively;

[0079] Figure 4a To add aerodynamic blur to the front aircraft image, Figure 4b To add aerodynamic blur to the aircraft image, Figure 4c To add aerodynamic blur to the scene image, Figure 4d Image of the scene after adding aerodynamic blur. DETAILED DESCRIPTION

[0080] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0081] Example 1

[0082] The present invention proposes a real-time simulation method for aerodynamic blur effect. The principle of the method is as follows: Figure 1 As shown, the method mainly comprises the following steps:

[0083] Step 1: Create a frame buffer object and draw the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture;

[0084] The main camera's viewport size and projection matrix are set by the field of view and resolution of the aircraft's infrared detector. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the texture coordinates are not standardized when a rectangular texture is used, and the texture coordinates are addressed by pixel, which makes the texture mapping process time-consuming. In order to standardize the drawn texture coordinates (i.e., 0.0 to 1.0), the maximum resolution is used as the main camera's viewport size.

[0085] Create a frame buffer object (FBO) and texture objects, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

[0086] Step 2: Read the rendered texture image data, and use the shader on the GPU to complete the aerodynamic blur effect processing of the image data.

[0087] Create a quad node with the resolution size of the infrared detector, set the texture mode of the quad rendering state node, bind the texture object in step 1 to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is usually executed by the graphics processing unit (GPU) to process vertices and pixels to produce the final image.

[0088] Common blur effect processing operations include Gaussian blur, box blur, double blur, granular blur, etc. A generative model of aerodynamic blur effect based on deep learning can also be constructed, and the convolution kernel obtained by training is used for blur processing. The present invention adopts a generative adversarial network to learn the characteristics of aerodynamic blur effect. Its principle is as follows Figure 2 As shown in the figure, the generator G randomly samples z from the latent space as input, and its output G(z) needs to imitate the real samples in the training set as much as possible. The input of the discriminator D is the real sample x or the output G(z) of the generated model, and the output is D(x) or D(G(z)). Its purpose is to distinguish the output of the generator from the real sample as much as possible. The generator should deceive the discriminator as much as possible. The two models compete with each other and continuously adjust parameters. The ultimate goal is to make it impossible for the discriminator to determine whether the output of the generator is real.

[0089] Since there is currently little real aero-optical effect image data, in order to expand the amount of training set data, a theoretical calculation method is used to generate simulated images with aerodynamic blur effects. The simulation process can be described as:

[0090]

[0091] Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, and h(x,y) is the point spread function (PSF) produced by the atmospheric turbulence effect.

[0092] There are multiple vortices in the turbulent field. Each vortex can be considered as a "lens" and can be simulated using turbulence statistics theory. For a single vortex, the point spread function is as follows:

[0093]

[0094] Where: h z (x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ xz , yz h z (x,y) Fuzziness factor in the x,y direction.

[0095] The existence of vortices in the turbulent field is not single. The turbulent field is formed by the joint action of multiple vortices of different scales and intensities. The expression of vortex intensity is:

[0096]

[0097] Where: k z is the random intensity factor of the zth vortex, and its probability distribution roughly satisfies the uniform distribution; a is the scale parameter of the distribution.

[0098] For the entire atmospheric turbulence, the overall point spread function expression is:

[0099]

[0100] Where: Z is the number of vortices in the turbulent field.

[0101] The process of generating an image of aerodynamic blur effect using a generative adversarial network is as follows:

[0102] 1) Define the network structure and loss function of the generator and discriminator. The network structure of the generator and discriminator is as follows Figure 3a and Figure 3b As shown, the generator and discriminator loss functions are as follows:

[0103]

[0104] Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ;

[0105] 2) Initialize the input and output of the generator and discriminator and the network training method. Use the measured original images and the measured images or simulated images with aerodynamic blur effects as training samples to train the generator and discriminator. The training process is as follows: ① Initialize the network: Randomly initialize the parameters of the generator and discriminator. ② Generate samples: The generator receives the noise vector and converts it into a data sample. ③ Distinguish the authenticity: The discriminator receives samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively. ④ Back propagation and update: For the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified. For the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator. ⑤ Iterative training: Repeat the above process. In each round of iteration, the two networks change their weights in turn until convergence.

[0106] 3) After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output.

[0107] After the convolution kernel parameters are output, they are passed into the shader, and the GPU rendering pipeline is used to calculate the image. The calculation process is as follows: when a pixel of the original infrared scene image is convolved, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. When all pixel values ​​of the original infrared scene image are calculated, the infrared scene processed with blur effect can be generated.

[0108] Step 3: Set the camera viewport and projection matrix, render the processed image data to the scene rendering window, and complete the real-time simulation of the aerodynamic blur effect.

[0109] According to the resolution of the aircraft's infrared detector, set the viewport from the camera, and set the two-dimensional orthographic projection matrix and observation matrix. The two-dimensional orthographic projection matrix is ​​a special type of orthographic projection matrix, which is mainly used for the projection of a two-dimensional image onto a two-dimensional screen. It mainly contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows:

[0110]

[0111] Set the observation matrix to the identity matrix, which is expressed as follows:

[0112]

[0113] Add the quadrilateral node created in step 2 to the scene node and place it at the origin of the three-dimensional space. Set the camera position (viewpoint position) and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect. Figure 4a to Figure 4b Shown is a comparison chart before and after adding aerodynamic blur effect to different aerodynamic model scenes during the simulation process.

[0114] Example 2

[0115] The present invention also provides a real-time simulation system for aerodynamic blur effect. The real-time simulation system for aerodynamic blur effect can be realized by executing the process steps of the real-time simulation method for aerodynamic blur effect. That is, those skilled in the art can understand the real-time simulation method for aerodynamic blur effect as a preferred implementation of the real-time simulation system for aerodynamic blur effect.

[0116] The real-time simulation system of aerodynamic blur effect provided by the present invention comprises: module M1: creating a frame buffer object, drawing the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture; module M2: reading the rendered texture image data, and completing the aerodynamic blur effect processing of the image data using a shader on the GPU; module M3: setting the slave camera viewport and projection matrix, rendering the image data processed by the aerodynamic blur effect into the scene rendering window, and completing the real-time simulation of the aerodynamic blur effect.

[0117] The module M1 comprises:

[0118] The main camera's viewport size and projection matrix are set based on the aircraft's infrared detector's field of view and resolution. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the maximum resolution is used as the main camera's viewport size.

[0119] Create a frame buffer object FBO and a texture object, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

[0120] The module M2 comprises:

[0121] Create a quad node according to the resolution of the infrared detector, set the texture mode of the quad rendering state node, bind the rendered frame buffer object to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is executed by the image processor GPU to process vertices and pixels to generate the final image;

[0122] Theoretical calculation method is used to generate a simulated image with aerodynamic blur effect. The simulation process is described as follows:

[0123]

[0124] Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, h(x,y) is the point spread function PSF generated by the atmospheric turbulence effect; (x,y) is the position of the original image;

[0125] There are multiple vortices in the turbulent field. The turbulence statistics theory is used for simulation. For a single vortex, the point spread function PSF is as follows:

[0126]

[0127] Where: h z (x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ xz , yz h z (x,y) fuzzy factor in x and y directions;

[0128] The expression of random vortex intensity is:

[0129]

[0130] Where: k z is the random intensity factor of the zth vortex; a is the scale parameter of the distribution;

[0131] For the entire atmospheric turbulence, the overall point spread function expression is:

[0132]

[0133] Where: Z is the number of vortices in the turbulent field.

[0134] The process of generating an image with aerodynamic blur effect using a generative adversarial network is as follows:

[0135] Define the network structure and loss function of the generator and discriminator. The expressions are as follows:

[0136]

[0137] Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ;

[0138] The measured original images and the measured images or simulated images with aerodynamic blur effects are used as training samples to train the generator and the discriminator. The training process is as follows: randomly initialize the parameters of the generator and the discriminator; the generator receives the noise vector and converts it into a data sample; the discriminator receives the samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively; for the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified; for the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator; repeat the above process iteratively, and in each round of iteration, the two networks change their weights in turn until convergence;

[0139] After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output; after the convolution kernel parameters are output, they are passed into the shader, and the image is calculated using the GPU rendering pipeline. The calculation process is: when performing convolution calculation on a pixel of the original infrared scene image, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. After the calculation of all pixel values ​​of the original infrared scene image is completed, the infrared scene processed with a blurred effect can be generated.

[0140] The module M3 comprises:

[0141] According to the resolution of the aircraft's infrared detector, set the viewport of the camera, and set the two-dimensional orthographic projection matrix and observation matrix;

[0142] 2D orthographic projection matrix Mat p Contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows:

[0143]

[0144] Set the observation matrix to the identity matrix Mat v , which is expressed as follows:

[0145]

[0146] Add a quadrilateral node to the scene node and place it at the origin of the three-dimensional space. Set the camera position and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect.

[0147] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0148] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A real-time simulation method for aerodynamic blur effect, characterized in that: include: Step 1: Create a frame buffer object and draw the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture; Step 2: Read the rendered texture image data and use the shader on the GPU to complete the aerodynamic blur effect processing of the image data; Step 3: Set the camera viewport and projection matrix, render the image data processed by the aerodynamic blur effect to the scene rendering window, and complete the real-time simulation of the aerodynamic blur effect.

2. The real-time simulation method of aerodynamic blur effect according to claim 1, characterized in that: The step 1 comprises: The main camera's viewport size and projection matrix are set based on the aircraft's infrared detector's field of view and resolution. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the maximum resolution is used as the main camera's viewport size. Create a frame buffer object FBO and a texture object, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

3. The real-time simulation method of aerodynamic blur effect according to claim 1, characterized in that: The step 2 comprises: Create a quad node according to the resolution of the infrared detector, set the texture mode of the quad rendering state node, bind the rendered frame buffer object to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is executed by the image processor GPU to process vertices and pixels to generate the final image; Theoretical calculation method is used to generate a simulated image with aerodynamic blur effect. The simulation process is described as follows: Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, h(x,y) is the point spread function PSF generated by the atmospheric turbulence effect; (x,y) is the position of the original image; There are multiple vortices in the turbulent field. The turbulence statistics theory is used for simulation. For a single vortex, the point spread function PSF is as follows: Where: h z (x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ xz , yz h z (x,y) fuzzy factor in x and y directions; The expression of random vortex intensity is: Where: k z is the random intensity factor of the zth vortex; a is the scale parameter of the distribution; For the entire atmospheric turbulence, the overall point spread function expression is: Where: Z is the number of vortices in the turbulent field.

4. The real-time simulation method of aerodynamic blurring effect according to claim 3 is characterized in that: The process of generating an image of aerodynamic blur effect using a generative adversarial network is as follows: Define the network structure and loss function of the generator and discriminator. The expressions are as follows: Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ; The measured original images and the measured images or simulated images with aerodynamic blur effects are used as training samples to train the generator and the discriminator. The training process is as follows: randomly initialize the parameters of the generator and the discriminator; the generator receives the noise vector and converts it into a data sample; the discriminator receives the samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively; for the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified; for the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator; repeat the above process iteratively, and in each round of iteration, the two networks change their weights in turn until convergence; After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output; After the convolution kernel parameters are output, they are passed into the shader, and the GPU rendering pipeline is used to calculate the image. The calculation process is as follows: when performing convolution calculation on a pixel of the original infrared scene image, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. After the calculation of all pixel values ​​of the original infrared scene image is completed, the infrared scene processed with the blurred effect can be generated.

5. The real-time simulation method of aerodynamic blurring effect according to claim 4, characterized in that: The step 3 comprises: According to the resolution of the aircraft's infrared detector, set the viewport of the camera, and set the two-dimensional orthographic projection matrix and observation matrix; 2D orthographic projection matrix Mat p Contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows: Set the observation matrix to the identity matrix Mat v , which is expressed as follows: Add a quadrilateral node to the scene node and place it at the origin of the three-dimensional space. Set the camera position and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect.

6. A real-time simulation system for aerodynamic blurring effect, characterized in that: include: Module M1: Create a frame buffer object and draw the frame buffer data of each frame of the infrared scene rendered by the main camera onto the texture; Module M2: reads the rendered texture image data and uses the shader on the GPU to complete the aerodynamic blur effect processing of the image data; Module M3: Set the camera viewport and projection matrix, render the image data processed by the aerodynamic blur effect to the scene rendering window, and complete the real-time simulation of the aerodynamic blur effect.

7. The real-time simulation system of aerodynamic blurring effect according to claim 1, characterized in that: The module M1 comprises: The main camera's viewport size and projection matrix are set based on the aircraft's infrared detector's field of view and resolution. When the infrared detector's horizontal resolution is inconsistent with its vertical resolution, the maximum resolution is used as the main camera's viewport size. Create a frame buffer object FBO and a texture object, bind the main camera to the FBO, render the infrared simulation scene from the aircraft's perspective according to the aircraft's motion trajectory in the scene during the semi-physical simulation, and draw the frame buffer data onto the texture object.

8. The real-time simulation system of aerodynamic blurring effect according to claim 6, characterized in that: The module M2 comprises: Create a quad node according to the resolution of the infrared detector, set the texture mode of the quad rendering state node, bind the rendered frame buffer object to the quad node, turn on the shader mode of the rendering state node, create the corresponding vertex shader and fragment shader, and pass the blur effect processing operation to the shader. The shader is executed by the image processor GPU to process vertices and pixels to generate the final image; Theoretical calculation method is used to generate a simulated image with aerodynamic blur effect. The simulation process is described as follows: Where: g(x,y) is the aerodynamic blur effect image, f(x,y) is the original image, h(x,y) is the point spread function PSF generated by the atmospheric turbulence effect; (x,y) is the position of the original image; There are multiple vortices in the turbulent field. The turbulence statistics theory is used for simulation. For a single vortex, the point spread function PSF is as follows: Where: h z (x,y) is the PSF of the zth vortex, A(z) is the random vortex intensity, x z ,y z is the random offset position of the vortex center of mass, λ xz , yz h z (x,y) fuzzy factor in x and y directions; The expression of random vortex intensity is: Where: k z is the random intensity factor of the zth vortex; a is the scale parameter of the distribution; For the entire atmospheric turbulence, the overall point spread function expression is: Where: Z is the number of vortices in the turbulent field.

9. The real-time simulation system of aerodynamic blurring effect according to claim 8, characterized in that: The process of generating an image of aerodynamic blur effect using a generative adversarial network is as follows: Define the network structure and loss function of the generator and discriminator. The expressions are as follows: Where: min G max D V(D,G) is the goal of the entire optimization, which is to minimize the loss of the generator G and maximize the loss of the discriminator D; The real sample m comes from the real data distribution p data The expected value of (x); The generated sample G(n) comes from the standard normal distribution p output by the generator G n The expected value of (n); n is p n Random sampling in ; The measured original images and the measured images or simulated images with aerodynamic blur effects are used as training samples to train the generator and the discriminator. The training process is as follows: randomly initialize the parameters of the generator and the discriminator; the generator receives the noise vector and converts it into a data sample; the discriminator receives the samples from the real data set and the samples generated by the generator, and gives the probability scores of them belonging to the real data respectively; for the discriminator, minimize the loss function of the probability that the real sample is correctly classified and the probability that the generated sample is misclassified; for the generator, maximize the probability that the generated sample is mistaken for a real sample by the discriminator; repeat the above process iteratively, and in each round of iteration, the two networks change their weights in turn until convergence; After the training is completed, the convolution kernel parameters of the trained generative adversarial network model are output; After the convolution kernel parameters are output, they are passed into the shader, and the GPU rendering pipeline is used to calculate the image. The calculation process is as follows: when performing convolution calculation on a pixel of the original infrared scene image, the center of the convolution kernel is placed on the pixel, and then the pixel within the convolution kernel range is multiplied by the corresponding weight, and the result is used as the new pixel value of the pixel. After the calculation of all pixel values ​​of the original infrared scene image is completed, the infrared scene processed with the blurred effect can be generated.

10. The real-time simulation system of aerodynamic blurring effect according to claim 9, characterized in that: The module M3 comprises: According to the resolution of the aircraft's infrared detector, set the viewport of the camera, and set the two-dimensional orthographic projection matrix and observation matrix; 2D orthographic projection matrix Mat p Contains four parameters: left, right, bottom, and top. Set the left parameter size to 0, the right parameter size to the horizontal resolution, the bottom parameter size to 0, and the top parameter size to the vertical resolution. The expression is as follows: Set the observation matrix to the identity matrix Mat v , which is expressed as follows: Add a quadrilateral node to the scene node and place it at the origin of the three-dimensional space. Set the camera position and draw the scene observed from the camera to the corresponding scene rendering window. At this time, the scene displayed in the scene rendering window is the infrared scene with a blur effect.

Images