Airport infrared information fusion target presentation system and design method thereof

Abstract

The invention relates to an airport infrared information fusion target presentation system and a design method thereof, and belongs to the technical field of image processing. In order to solve the problem that the current airport scene monitoring is greatly influenced by poor night illumination conditions and severe weather, the method improves the observation experience of human eyes by converting an infrared image into a visible light image, and comprises the steps of obtaining airport scene information, selecting a proper conversion algorithm according to different scene information, and obtaining the airport scene information. And constructing an airport scene visible light and infrared image data set for training the generative adversarial network, and improving a generator and a discriminator to improve the sense of reality and the resolution of the generated image. In addition, training is carried out in a cyclic generative adversarial mode at night, so that dependence on supervision data is reduced. The converted image is suitable for human eyes to observe and timely discover and process problems, and the airport scene monitoring effect at night and in severe weather can be improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a target presentation system for airport fused infrared information and a design method thereof. Background Art

[0002] With the rapid development of my country's civil aviation industry, airport scene monitoring is becoming increasingly difficult. In existing video surveillance networks, all-weather monitoring is often required, while visible light video is greatly affected by poor lighting conditions at night and bad weather. Therefore, the effective use of infrared images is of great practical significance for improving airport monitoring. However, the radiance distribution of target scenes reflected by infrared images is very different from the distribution of visible light images that the human eye is accustomed to. Long-term observation is not only prone to fatigue, but also affects the perception of targets and the understanding of scenes.

[0003] At the same time, with the rapid development of deep learning and computer vision technology, image style transfer, as an emerging image processing technology, is being applied in more and more fields. It has become a trend to make full use of the structural knowledge of infrared images to generate realistic daytime visible light images. Through conversion, while meeting the functions, it can also present color visible light images with richer information and more comfortable for the human eye. However, the current deep learning-based algorithms, such as pix2pix, CycleGAN, I2V-GAN, etc., face difficulties and challenges in the implementation process, such as the inability to obtain paired data, difficulty in generating high-resolution images, lack of details and real texture, etc. Summary of the invention

[0004] To solve the above problems, the present invention discloses an airport target presentation system integrating infrared information and a design method thereof, which improves the human eye observation experience by converting infrared images into visible light images, and solves the problem that the current airport scene monitoring is greatly affected by poor lighting conditions and bad weather at night.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] The present invention provides a method for designing a target presentation system for an airport that integrates infrared information, comprising: firstly obtaining a visible light video stream and an infrared video stream through a visible light and infrared dual-light camera at the airport, then obtaining airport scene information including day and night information and meteorological information through a video analysis algorithm based on the visible light video stream, and judging whether it is daytime based on the day and night information of the airport.

[0007] If the judgment result is yes, the processing method is selected according to the airport weather information.

[0008] If it is a sunny day, the visible light video stream is directly output.

[0009] If it is foggy, rainy, snowy or other bad weather, the infrared to visible light image conversion algorithm based on conditional GAN ​​is used to convert each frame of infrared image into a visible light image and output it. The specific steps are as follows:

[0010] Step 1: Use the airport visible light and infrared dual-light camera to collect visible light images and infrared images of the airport scene at the same time during the day to build a dual-light dataset.

[0011] Step 2: Spatial calibration is performed on the visible light images and infrared images in the dataset. Since the field of view of the visible light image is larger than that of the infrared image, the visible light image needs to be aligned according to the relative position of the infrared image and transformed to the same size. The specific method is to first select a set of key points of the visible light image and the infrared image, then calculate the affine matrix, and finally perform an affine transformation on the visible light image to obtain a precisely matched dual-light dataset.

[0012] Step 3: Construct a conditional generative adversarial network and input the infrared image as conditional information into the generator and discriminator to achieve conditional image generation. The loss function is as follows:

[0013] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))]

[0014] Among them, G is the generator, D is the discriminator, E represents expectation, x represents infrared image, y represents real visible light image, z represents random noise, G(x,z) represents the generated fake image, D(x,y) represents the probability that the discriminator thinks the real image is the real image, and D(x,G(x,z)) represents the probability that the discriminator thinks the fake image is the real image.

[0015] In order to improve the realism and resolution of generated images, a coarse-to-fine generator and multi-scale discriminator architecture is adopted. The generator is decomposed into a global generation network and a local enhancement network. The global generation network is first used to generate low-resolution images, and then the local enhancement network is used to increase details. This method is adopted because directly generating high-resolution images is prone to training instability and image quality degradation. Through staged generation, the global layout and local details can be effectively integrated, thereby improving the realism and resolution of the image.

[0016] The multi-scale discriminator further ensures the authenticity of the image at different scales, thereby improving the image quality while maintaining high resolution. Three discriminators of different scales are used to process images of different scales. They have the same network structure. When calculating the optimization target, the task can be converted into a multi-objective training task. The optimization target at this time becomes:

[0017]

[0018] Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, which is used to measure the generator G and the discriminator D k The loss of confrontation between them.

[0019] This approach is adopted because high-resolution image synthesis requires capturing the global consistency and detail features of images at different scales, and the multi-scale discriminator can better distinguish between real and synthetic images.

[0020] In addition, since the generator needs to generate images of different scales, in order to make the training more stable, a feature matching loss based on the discriminator is introduced. This is because using only the traditional adversarial loss may lead to a lack of diversity and details in the generated images. Feature matching loss can promote the generated images to be more similar to the real images at different levels, improve the stability of training and the quality of generated images. Its loss function is as follows:

[0021]

[0022] Among them, G is the generator, D k is the kth discriminator, T represents the total number of layers, N i represents the number of elements of the i-th layer feature, Represents the discriminator D k The extracted i-th layer feature, E (x,y) represents the expectation of the input x and the true output y, and ‖.‖1 represents the L1 norm.

[0023] At this point, the total loss function is as follows:

[0024]

[0025] Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, L FM (G,D k ) is the feature matching loss function.

[0026] Step 4: Input the data set prepared in step 2 into the network model built in step 3 for training. When the confrontation between the generator and the discriminator reaches a balanced state, the model training is considered complete and the model is saved.

[0027] Step 5: Input the daytime infrared image into the trained network model to obtain the daytime visible light image.

[0028] If the judgment result is no, then use the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN to convert each frame of infrared image into a visible light image and output it. The specific steps are as follows:

[0029] Step A: Use the airport visible light and infrared dual-light camera to collect daytime visible light images and nighttime infrared images of the airport scene, intercept 256×256 daytime visible light images and 256×256 nighttime infrared images, construct a dual-light dataset, and divide the dataset into domain A and domain B, where domain A is nighttime infrared images and domain B is daytime visible light images. In order to increase the sample capacity of the dataset and prevent overfitting in the training of the neural network, image flipping and rotation are used to expand the dataset.

[0030] Step B: Construct two generating networks and two discriminative networks, namely: Generator 1 for infrared image to visible light image, Generator 2 for visible light image to infrared image, Discriminator 1 for infrared image domain, Discriminator 2 for visible light image domain. Based on Generator 1, Generator 2, Discriminator 1 and Discriminator 2, a cyclic generative adversarial network is constructed, and the cycle consistency loss is used without the need for high-precision pairing of infrared images and visible light images, so as to achieve the purpose of unsupervised learning from infrared images to visible light images. The generator uses a convolution module for downsampling, a convolution residual module for feature extraction, and a deconvolution module for upsampling to form a U-net network structure.

[0031] Step C: Input the data set prepared in step 1 into the network model built in step 2 for training. When the confrontation between the generator and the discriminator reaches a balanced state, the model training is considered complete and the model is saved. The network parameters are updated by minimizing the loss function to obtain the optimal network model.

[0032] Step D: Since the current unsupervised image conversion algorithm based on CycleGAN can only generate images with a resolution of 256×256, and the generated images have serious blurred details, the night infrared image is cropped into multiple 256×256 sub-images, which are input into the trained optimal network model to obtain the same number of 256×256 daytime visible light images, which are then spliced ​​into a complete visible light image according to the cropping rules. Finally, a clear visible light image is obtained through super-resolution image restoration.

[0033] On the other hand, the present invention provides a target presentation system for airport fusion infrared information, including: a video acquisition module, an information acquisition module, a scene determination module, an algorithm matching module, an algorithm processing module and a video display module.

[0034] The video acquisition module is used to acquire visible light and infrared video data captured in real time by a visible light and infrared dual-light camera.

[0035] The information acquisition module is used to acquire airport scene information and transmit it to the scene determination module.

[0036] The scene determination module is used to determine whether it is daytime and whether it is sunny according to the airport scene information.

[0037] The algorithm matching module is used to match the appropriate infrared to visible light image conversion algorithm according to the determined scene information. If it is daytime, the processing method is selected according to the airport weather information; if it is sunny, no algorithm processing is used; if it is foggy, rainy, snowy or other bad weather, the infrared to visible light image conversion algorithm based on conditional GAN ​​is used. If it is night, the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN is used.

[0038] The algorithm processing module is used to process the received video data, directly output a visible light video stream, or use an infrared to visible light image conversion algorithm to convert each frame of infrared image into a visible light image and output it.

[0039] The video display module is used to display the image output by the algorithm processing module in real time. The processed image is more suitable for human eye observation, thereby improving the monitoring efficiency.

[0040] The beneficial effects achieved by the present invention are:

[0041] The present invention selects a suitable conversion algorithm according to different scene information, and uses infrared images to generate color visible light images with richer information and more comfortable for human eyes, so as to achieve all-weather monitoring of the airport, which is conducive to improving monitoring efficiency. In night scenes, the core idea of ​​CycleGAN cycle consistency is used. When constructing sample data, there is no need to strictly perform registration and pairing. The network is trained in an unsupervised training method, which greatly saves manpower and improves the quality of images generated by unsupervised training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a method for designing a target presentation system for airport fusion infrared information provided by an embodiment of the present invention;

[0043] Figure 2 It is a flow chart of constructing airport visible light and infrared data sets provided by an embodiment of the present invention;

[0044] Figure 3 is a network structure diagram of a conditional GAN ​​generator provided by an embodiment of the present invention;

[0045] Figure 4It is a CycleGAN network process framework diagram provided by an embodiment of the present invention;

[0046] Figure 5 It is a flow chart of an algorithm for converting unpaired nighttime infrared images to daytime visible light images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0048] Embodiment 1

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for designing a target presentation system for airport fusion infrared information, including:

[0050] S1. First, obtain the visible light video stream and infrared video stream through the airport's visible light and infrared dual-light camera, and then obtain the airport scene information, including day and night information and weather information, through the video analysis algorithm based on the visible light video stream, and judge whether it is daytime based on the airport's day and night information. If the judgment result is yes, select a processing method based on the airport's weather information.

[0051] S2. If it is a sunny day, directly output the visible light video stream.

[0052] S3. If the weather is bad, such as foggy, rainy, snowy, etc., the infrared to visible light image conversion algorithm based on conditional GAN ​​is used to convert each frame of infrared image into a visible light image and output it.

[0053] Specifically, the infrared to visible light image conversion method based on conditional GAN ​​described in S3 includes:

[0054] Step 1: Figure 2 As shown in FIG. 1 , an airport visible light and infrared dual-light camera is used to collect visible light images and infrared images of the airport scene at the same time during the day to construct a dual-light dataset.

[0055] Step 2: If Figure 2 As shown in the figure, the visible light images and infrared images in the dataset are spatially calibrated. Since the field of view of the visible light image is larger than that of the infrared image, the visible light image needs to be aligned according to the relative position of the infrared image and transformed to the same size. The specific method is to first select a set of key points of the visible light image and the infrared image, then calculate the affine matrix, and finally perform an affine transformation on the visible light image to obtain a precisely matched dual-light dataset.

[0056] Step 3: Construct a conditional generative adversarial network and input the infrared image as conditional information into the generator and discriminator to achieve conditional image generation. The loss function is as follows:

[0057] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))]

[0058] Among them, G is the generator, D is the discriminator, E represents expectation, x represents infrared image, y represents real visible light image, z represents random noise, G(x,z) represents the generated fake image, D(x,y) represents the probability that the discriminator thinks the real image is the real image, and D(x,G(x,z)) represents the probability that the discriminator thinks the fake image is the real image.

[0059] like Figure 3 As shown in the figure, the generator adopts a hierarchical network structure, including a global generation network and one or more local enhancement networks. The global generation network G1 is used to build the basic framework and general outline of the image, set the overall style and tone, including a convolution layer front end for feature extraction, a series of residual blocks for processing and transmitting deep features, and a transposed convolution layer back end for upsampling, which is sufficient to capture the global information of the image at a lower resolution but not enough to refine local details. The local enhancement network G2 is used to increase local details on the basis of the global generator to improve the resolution and quality of the image. Each local enhancement network also consists of a convolution layer front end, a residual block, and a transposed convolution layer back end. Compared with the global generation network, the local enhancement network focuses on higher resolution image parts to refine details and textures. Each local enhancement network increases the output resolution to twice that of the previous network, that is, the resolution is doubled, for example, from 512×256 to 1024×512, and then to 2048×1024, and so on.

[0060] First, a low-resolution visible light image of the airport is generated using the global generation network G1. This step captures the layout and main elements of the entire scene, such as buildings, roads, and the sky. Then, the local enhancement network G2 refines this low-resolution image, adding details such as the texture of the road surface and the details of the aircraft, thereby producing a high-resolution, detail-rich image. The input to G2 is the element-wise sum of the feature map of G2 and the last feature map of G1, combining global structure and local details to produce a high-resolution image. While generating the image, three discriminators (D1, D2, D3) of different scales evaluate the authenticity of the generated image. The discrimination is performed at three different scales and the results are averaged. The three scales for discrimination are: original image, 1 / 2 downsampled original image, and 1 / 4 downsampled original image. The original scale is responsible for evaluating the authenticity of the image at the highest resolution, focusing on the subtle details of the image, such as texture, edges, etc. The medium scale evaluates the image after reducing the resolution once, balancing the authenticity of the global structure and local details. The lowest scale works at a further reduced resolution, focusing mainly on the global consistency and layout of the image. When calculating the optimization target, the task can be converted into a multi-objective training task. The optimization target becomes:

[0061]

[0062] Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, which is used to measure the generator G and the discriminator D k The loss of confrontation between them.

[0063] In addition, since the generator needs to generate images of different scales, in order to make the training more stable, a feature matching loss based on the discriminator is introduced. This is because using only the traditional adversarial loss may lead to a lack of diversity and details in the generated images. Feature matching loss can promote the generated images to be more similar to the real images at different levels, improve the stability of training and the quality of generated images. Its loss function is as follows:

[0064]

[0065] Among them, G is the generator, D k is the kth discriminator, T represents the total number of layers, N i represents the number of elements of the i-th layer feature, Represents the discriminator D k The extracted i-th layer feature, E (x,y) represents the expectation of the input x and the true output y, and ‖.‖1 represents the L1 norm.

[0066] At this point, the total loss function is as follows:

[0067]

[0068] Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, L FM (G,D k ) is the feature matching loss function.

[0069] Step 4: Input the data set prepared in step 2 into the network model built in step 3 for training. When the confrontation between the generator and the discriminator reaches a balanced state, the model training is considered complete and the model is saved.

[0070] Step 5: Input the daytime infrared image into the trained network model to obtain the daytime visible light image.

[0071] S4. If the judgment result is no, use the CycleGAN-based unpaired nighttime infrared to daytime visible light image conversion algorithm to convert each frame of infrared image into a visible light image and output it.

[0072] Specifically, the CycleGAN-based unpaired nighttime infrared to daytime visible light image conversion method described in S4 includes:

[0073] Step 1: Figure 2 As shown in the figure, the airport visible light and infrared dual-light camera is used to collect the airport scene daytime visible light images and nighttime infrared images, and 256×256 daytime visible light images and 256×256 nighttime infrared images are intercepted to construct a dual-light dataset, which is divided into domain A and domain B, where domain A is nighttime infrared images and domain B is daytime visible light images. In order to increase the sample capacity of the dataset and prevent overfitting in the training of the neural network, the dataset is expanded by image flipping and rotation.

[0074] Step 2: If Figure 4 As shown, two generating networks and two discriminative networks are constructed, namely: a generator 1 for infrared images to visible light images, a generator 2 for visible light images to infrared images, a discriminator 1 for infrared image domain, and a discriminator 2 for visible light image domain. A recurrent generative adversarial network is constructed based on the generator 1, the generator 2, the discriminator 1, and the discriminator 2. The generator uses a convolution module for downsampling, a convolution residual module for feature extraction, and a deconvolution module for upsampling to form a U-net network structure.

[0075] Step 3: If Figure 4As shown in the figure, the data set prepared in step 1 is input into the network model constructed in step 2 for training. The real infrared image in the data set is generated by generator 1 to generate a fake visible light image, and then the fake infrared image is obtained by generator 2. The real infrared image and the fake infrared image are respectively used as the input of discriminator 1 to judge whether it is true or false. The fake infrared image is restored to the input infrared image as much as possible through the cycle consistency loss. Similarly, the real visible light image in the data set is generated by generator 2 to generate a fake infrared image, and then the fake visible light image is obtained by generator 1. The real visible light image and the fake visible light image are respectively used as the input of discriminator 2 to judge whether it is true or false. The fake visible light image is restored to the input visible light image as much as possible through the cycle consistency loss. The goal of the generating network is to generate as real fake data as possible to deceive the discriminating network. The goal of the discriminating network is to improve its ability to distinguish true and false data. The two constantly improve their abilities in the confrontation. The network parameters are updated by minimizing the loss function to obtain the optimal network model.

[0076] Step 4: Figure 5 As shown in the figure, the night infrared image is cropped into multiple 256×256 sub-images, which are input into the trained network model to obtain the same number of 256×256 daytime visible light images, which are then spliced ​​into a complete visible light image according to the cropping rules. Finally, a clear visible light image is obtained through super-resolution image restoration.

[0077] Embodiment 2:

[0078] The embodiment of the present invention provides a target presentation system for airport fused infrared information, comprising: a video acquisition module, an information acquisition module, a scene determination module, an algorithm matching module, an algorithm processing module and a video display module.

[0079] The video acquisition module is used to acquire visible light and infrared video data captured in real time by a visible light and infrared dual-light camera.

[0080] The information acquisition module is used to acquire airport scene information and transmit it to the scene determination module.

[0081] The scene determination module is used to determine whether it is daytime and whether it is sunny according to the airport scene information.

[0082] The algorithm matching module is used to match the appropriate infrared to visible light image conversion algorithm according to the determined scene information. If it is daytime, the processing method is selected according to the airport weather information; if it is sunny, no algorithm processing is used; if it is foggy, rainy, snowy or other bad weather, the infrared to visible light image conversion algorithm based on conditional GAN ​​is used. If it is night, the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN is used.

[0083] The algorithm processing module is used to process the received video data, directly output a visible light video stream, or use an infrared to visible light image conversion algorithm to convert each frame of infrared image into a visible light image and output it.

[0084] The video display module is used to display the image output by the algorithm processing module in real time. The processed image is more suitable for human eye observation, thereby improving the monitoring efficiency.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A method for designing a target presentation system for airport fusion infrared information, characterized in that: First, the airport's visible light and infrared dual-light cameras are used to obtain visible light video streams and infrared video streams. Then, based on the visible light video streams, a video analysis algorithm is used to obtain airport scene information, including day and night information and meteorological information. Based on the airport's day and night information, it is determined whether it is daytime. If the judgment result is yes, a processing method is selected according to the airport weather information; If it is a sunny day, the visible light video stream is directly output; If it is foggy, rainy, or snowy weather, the infrared-to-visible light image conversion algorithm based on conditional GAN ​​is used to convert each frame of infrared image into a visible light image and output it; if the judgment result is no, the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN is used to convert each frame of infrared image into a visible light image and output it.

2. The method for designing a target presentation system for airport fusion infrared information according to claim 1 is characterized in that: The infrared to visible light image conversion method based on conditional GAN ​​specifically includes the following steps: Step 1: Use the airport visible light and infrared dual-light camera to collect visible light images and infrared images of the airport scene at the same time during the day to build a dual-light dataset; Step 2: Spatial calibration of visible light images and infrared images in the dataset; Step 3: Construct a conditional generative adversarial network and input the infrared image as conditional information into the generator and discriminator to achieve conditional image generation; adopt a coarse-to-fine generator and multi-scale discriminator architecture; and introduce a feature matching loss based on the discriminator; Step 4: Input the data set prepared in step 2 into the network model built in step 3 for training. When the confrontation between the generator and the discriminator reaches a balanced state, the model training is considered complete and the model is saved. Step 5: Input the daytime infrared image into the trained network model to obtain the daytime visible light image.

3. The method for designing a target presentation system for airport fusion infrared information according to claim 2 is characterized by: The method for spatially calibrating the visible light image and the infrared image in step 2 specifically includes: selecting a group of key points of the visible light image and the infrared image; calculating an affine matrix; performing an affine transformation on the visible light image; and obtaining a precisely matched dual-light data set.

4. According to claim 2, the infrared to visible light image conversion method based on conditional GAN ​​is characterized in that: The loss function of step 3 is as follows: Where G is the generator, D is the discriminator, E represents the expectation, x represents the infrared image, y represents the real visible light image, z represents random noise, G(x,z) represents the generated fake image, D(x,y) represents the probability that the discriminator considers the real image to be the real image, and D(x,G(x,z)) represents the probability that the discriminator considers the fake image to be the real image; The coarse-to-fine generator and multi-scale discriminator architecture specifically includes: decomposing the generator into a global generation network and a local enhancement network, first using the global generation network to generate a low-resolution image, and then using the local enhancement network to increase details; Three discriminators of different scales are used to process images of different scales. They have the same network structure. When calculating the optimization target, the task can be converted into a multi-objective training task. The optimization target at this time becomes: Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, which is used to measure the generator G and the discriminator D k The loss of confrontation between them; The feature matching loss based on the discriminator is introduced, and its loss function is as follows: Among them, G is the generator, D k is the kth discriminator, T represents the total number of layers, N i represents the number of elements of the i-th layer feature, Represents the discriminator D k The extracted i-th layer feature, E (x,y) represents the expectation of input x and true output y, ‖.‖1 represents the L1 norm; At this point, the total loss function is as follows: Among them, G is the generator, D1, D2, D3 are three discriminators of different scales, and L cGAN (G,D k ) is the loss function of the conditional generative adversarial network, L FM (G,D k ) is the feature matching loss function.

5. According to claim 1, the method for converting infrared to visible light images based on conditional GAN ​​is characterized in that: If the judgment result is no, then use the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN to convert each frame of infrared image into a visible light image and output it. The specific steps are as follows: Step A: Use the airport visible light and infrared dual-light camera to collect visible light images of the airport scene during the day and infrared images at night to build a dual-light dataset; Step B: construct two generating networks and two discriminative networks, namely: a generator 1 for infrared images to visible light images, a generator 2 for visible light images to infrared images, a discriminator 1 for infrared image domain, and a discriminator 2 for visible light image domain; construct a cyclic generative adversarial network based on the generator 1, the generator 2, the discriminator 1 and the discriminator 2, and use the cycle consistency loss without the need for high-precision pairing of infrared images and visible light images, so as to achieve the purpose of unsupervised learning from infrared images to visible light images; Step C: Input the data set prepared in step 1 into the network model constructed in step 2 for training. When the confrontation between the generator and the discriminator reaches a balanced state, the model training is considered complete and the model is saved. Update the network parameters by minimizing the loss function to obtain the optimal network model. Step D: Input the night infrared image into the optimal network model to obtain the daytime visible light image.

6. The infrared to visible light image conversion method based on conditional GAN ​​according to claim 5, characterized in that: The step A captures 256×256 daytime visible light images and 256×256 nighttime infrared images, and divides the data set into domain A and domain B, wherein domain A is nighttime infrared images and domain B is daytime visible light images; in order to increase the sample capacity of the data set and prevent overfitting in the training of the neural network, the data set is expanded by image flipping and rotation.

7. The method for converting infrared to visible light images based on conditional GAN ​​according to claim 5, characterized in that: The generator in step B uses a convolution module for downsampling, a convolution residual module for feature extraction, and a deconvolution module for upsampling to form a U-net network structure.

8. The method for converting infrared to visible light images based on conditional GAN ​​according to claim 5, characterized in that: In step D, the night infrared image is cropped into multiple 256×256 sub-images, which are respectively input into the trained optimal network model to obtain the same number of 256×256 daytime visible light images, which are then spliced ​​into a complete visible light image according to the cropping rules, and finally a clear visible light image is obtained through super-resolution image restoration.

9. An airport target presentation system integrating infrared information, characterized in that: include: Video acquisition module, information acquisition module, scene determination module, algorithm matching module, algorithm processing module and video display module; The video acquisition module is used to collect visible light and infrared video data captured in real time by a visible light and infrared dual-light camera; The information acquisition module is used to acquire airport scene information and transmit it to the scene determination module; The scene determination module is used to determine whether it is daytime and whether it is sunny according to the airport scene information; The algorithm matching module is used to match the appropriate infrared to visible light image conversion algorithm according to the determined scene information; if it is daytime, the processing method is selected according to the airport weather information; if it is sunny, the algorithm processing is not used; if it is foggy, rainy, snowy or other bad weather, the infrared to visible light image conversion algorithm based on conditional GAN ​​is used; if it is night, the unpaired nighttime infrared to daytime visible light image conversion algorithm based on CycleGAN is used; The algorithm processing module is used to process the received video data, directly output a visible light video stream or use an infrared to visible light image conversion algorithm to convert each frame of infrared image into a visible light image and output it; The video display module is used to display the image output by the algorithm processing module in real time. The processed image is more suitable for human eye observation, thereby improving the monitoring efficiency.

Images