Heterogeneous image search method, device and storage medium based on generative adversarial network

Abstract

The present invention discloses a heterogeneous image search method, device, and storage medium based on a generative adversarial network, comprising the following steps: inputting a visible light image of a target area to be measured into a trained HR-GAN network, generating an infrared simulated image from the visible light image, and then comparing and analyzing the infrared image of the target area to the generated infrared simulated image to detect the nature and quantity of unexploded ordnance on the surface; the HR-GAN network training step includes introducing a high-resolution network, HRnet, as a generative network within the Pix2pix architecture to maintain a high-resolution representation of the image within the network, thereby alleviating the problem of unclear texture and high noise in the generated infrared image; and introducing a Restomer module within the generative network to promote learning of multi-scale local-global representations of the image, thereby alleviating the problem of missing structure in the generated infrared image. Target detection experiments have shown that the average target detection rate using the present invention reaches 88.89%, which can basically meet the requirement of a detection accuracy exceeding 85%.

Description

Technical Field

[0001] The present invention relates to the technical field of heterologous image detection, and in particular to a heterologous image search method, device and storage medium based on a generative adversarial network. Background Art

[0002] Surface unexploded ordnance (UXO) typically includes mines, grenades, artillery shells, aerial bombs, and bullets of various calibers. Detection of surface UXO typically involves push-type ground-based equipment or low-altitude drone detection, which is time-consuming and labor-intensive. This is particularly problematic when conducting live-fire exercises during the day and searching for UXO at night. This means that visible light images of the impact zone are captured before the shooting begins, but afterward, when it is dark and the UXO needs to be promptly detected and removed, the visible light images are of poor quality and difficult to distinguish. However, shortwave infrared images are of higher quality. Effectively utilizing visible light and infrared images for heterogeneous image detection is a key approach to all-day UXO detection.

[0003] Heterogeneous images typically refer to images acquired in different wavelengths using different types of imaging detection devices. Due to the different imaging mechanisms of visible light and infrared images, their features differ significantly. Directly cross-matching the acquired visible light and infrared images makes it difficult to effectively detect target information. However, converting visible light images to infrared images, or vice versa, to fuse heterogeneous images into homogeneous image matching is much easier to achieve, and methods for detecting differences between homogeneous images are already mature. Summary of the Invention

[0004] The present invention proposes a heterogeneous image search method, device, and storage medium based on a generative adversarial network, which can solve at least one of the technical problems in the background technology.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A heterogeneous image search method based on a generative adversarial network (GAN) involves inputting a visible light image of a target area to be detected into a trained HR-GAN network, generating an infrared simulated image from the visible light image, and then comparing and analyzing the infrared image of the target area with the generated infrared simulated image to detect the nature and quantity of unexploded ordnance on the surface.

[0007] The HR-GAN network training steps include introducing a high-resolution network, HRnet, as a generative network in the Pix2pix architecture to maintain the high-resolution representation of the image in the network and alleviate the problems of unclear texture and high noise in the generated infrared images.

[0008] The Restomer module is introduced into the generative network to promote the learning of multi-scale local-global representation of images and alleviate the problem of missing structure in generated infrared images.

[0009] Furthermore, the ECA model is introduced on the basis of HRnet to form an improved ECA-HRnet network;

[0010] Among them, ECA-HRnet input image size is 256×192, and it starts with a stem composed of two 3×3 convolutions with a stride of 1; after the stem, the resolution is reduced to 1 / 4 of the input image feature map resolution;

[0011] Its backbone network consists of four steps. In the first step, the network consists of four residual units. Each residual unit consists of an ECA-Bottleneck, which embeds the ECA module into a Bottleneck. A 3×3 convolution kernel is then used to reduce the width of the feature map resolution.

[0012] In the second step, a multi-resolution module is repeated separately;

[0013] In the third step, four multi-resolution modules are repeated one by one;

[0014] In the fourth step, three multi-resolution modules are repeated respectively;

[0015] Each multi-resolution module contains two parts: one is parallel multi-resolution convolution and the other is multi-resolution fusion;

[0016] Branches of different resolutions are connected in parallel, and each branch contains four residual units. Each unit is composed of ECA-BasicBlock, that is, the ECA module is embedded in the BasicBlock. Branches of different resolutions perform feature fusion to complete information exchange. The process from low resolution to high resolution is mainly achieved through bilinear upsampling, and the process from high resolution to low resolution adopts one or more strided convolutions, that is, convolution layers with a convolution kernel of 3×3 and a stride of 2.

[0017] Furthermore, in order to obtain different local channel information, the mapping relationship between channel C and convolution kernel size K is expressed as:

[0018] C=Φ(K)=2 (γ×K-b) (1)

[0019] Therefore, the K value will change adaptively with the number of channels, expressed as:

[0020]

[0021] Where K is the convolution kernel size, which represents the interaction between the current channel feature and K channel information, C represents the number of channels, b and γ are fixed values, set to 1 and 2 respectively.

[0022] Furthermore, the Restomer module is introduced into the generative network HRnet; first, 1×1 convolution is used to increase the dimension, then 3×3 group convolution is used to divide the features into three blocks, and finally the classic self-attention calculation is performed.

[0023] Furthermore, the loss function of the HR-GAN network is:

[0024]

[0025] Where, The loss function of the conditional generative adversarial network is borrowed. D represents the discriminator, G represents the generator, x represents the input image, y represents the corresponding real image, and z represents the random variable.

[0026] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0027] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0028] As can be seen from the above technical solution, the present invention's heterogeneous image search method based on a generative adversarial network (GAN) optimizes and improves the existing GAN model to ensure uninterrupted detection around the clock and in all weather conditions, effectively solving the problem of heterogeneous image detection of surface unexploded ordnance. This paper proposes a HR-GAN network model, which optimizes and improves the existing GAN model to achieve improved performance across various metrics compared to other networks. Target detection experiments demonstrate that the HR-GAN-based method for detecting differences in heterogeneous images of surface unexploded ordnance achieves an average target detection rate of 88.89%, essentially meeting the requirement for detection accuracy exceeding 85%. However, due to the easy introduction of noise and loss of some small target image information during the generation of the infrared simulated image, problems such as missed and false detections can occur when using the infrared simulated image and the actual infrared image to detect small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the image conversion flow chart;

[0030] Figure 2 This is the schematic diagram of the CycleGAN network;

[0031] Figure 3This is the InfraGAN algorithm flow chart;

[0032] Figure 4 This is a schematic diagram of the HR-GAN network architecture according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the HRnet network architecture according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of the ECA-HRnet network architecture according to an embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram of the ECA network architecture according to an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the ECA-BasicBlock network architecture according to an embodiment of the present invention;

[0037] Figure 9 This is a schematic diagram of the ECA-Bottleneck network architecture according to an embodiment of the present invention;

[0038] Figure 10 This is a schematic diagram of the Restomer network architecture according to an embodiment of the present invention;

[0039] Figure 11 This is a graph showing the change in the loss function according to an embodiment of the present invention;

[0040] Figure 12 The following are the image conversion results of the four algorithms in the embodiment of the present invention;

[0041] Figure 13 This is an infrared image feature point extraction diagram according to an embodiment of the present invention;

[0042] Figure 14 This is an infrared image feature matching graph according to an embodiment of the present invention;

[0043] Figure 15 This is a diagram showing the difference point detection results according to an embodiment of the present invention;

[0044] Figure 16 This is a diagram showing the detection results of heterogeneous images of unexploded ordnance on the surface according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0046] Generative Adversarial Nets (GAN) is a deep learning model that mainly consists of two neural networks that compete with each other, namely the generator and the discriminator. The generator selects some random noise from the distribution and tries to generate a distribution similar to the output from it. Its purpose is to generate a forged distribution that is exactly similar to the real distribution. In other words, the forged output image should look like a real image. The discriminator is responsible for receiving real data or data samples generated by the generator and evaluating them. At the same time, the discriminator will output a probability value. When this value is close to 1, it means that the data is real, and when it is close to 0, it means that it is generated. It is used to judge the authenticity of the input data. The discriminator not only evaluates the authenticity of the generated data, but also provides feedback to the generator to encourage the generator to improve the quality of the generated data. The image conversion process is as follows Figure 1 shown.

[0047] Generative adversarial networks (GANs) were proposed by Goodfellow et al. in 2014. Through comparison, it was found that the quality of images generated by this method was better than that of images generated by other methods. Since then, many scholars have continued to explore the path of generative adversarial networks in the field of image generation. At first, the generated images were random and the image quality was difficult to control. In order to solve such problems, Mirza et al. proposed the CGAN network in 2015. This network uses label information in the evaluation process of the discriminator. After multiple generation and discrimination games, the image generated by the generator is directional, and for the first time, the control of the generated image content is achieved. After that, Phillip et al., based on in-depth research on the CGAN network, replaced the random noise input of the GAN network with an image, and used it to solve the image conversion problem. In 2017, they proposed the Pix2pixGAN network, which has a good image conversion effect. However, this network also has limitations and is mainly used on paired datasets. In the same year, in order to solve the image conversion problem of unpaired datasets, Park et al. proposed a CycleGAN network with cycle consistency. Its principle is as follows Figure 2 As shown. In this network, the source domain image and the target domain image can be converted to each other, and the image obtained after a round-trip conversion is the same as the source domain image. In this process, two generators need to be constructed, one for converting the source domain to the target domain, and the other for converting the target domain to the source domain. By using two generators for mutual conversion, the image conversion problem of unpaired datasets is effectively solved. Later, in order to solve the mode collapse problem, the Infra-network component was introduced based on previous research and the InfraGAN network was proposed. The network generator and discriminator are trained together, and the input random noise is mapped to a latent space. The generator then generates images from this latent space to improve the quality of the generated image. The process is as follows Figure 3 shown.

[0048] HR-GAN algorithm

[0049] To address the problems of unclear texture, missing structure and high noise in infrared images generated by current infrared image generation algorithms, the present invention improves the Pix2pix model and proposes a HR-GAN infrared image generation algorithm. Its overall architecture is as follows: Figure 4 As shown in the figure, the corresponding infrared equivalent image is obtained for a given input visible light image. The main contents are as follows:

[0050] (1) A high-resolution network (HRnet) is introduced as the generative network in the Pix2pix architecture to maintain the high-resolution representation of the image in the network and alleviate the problem of unclear texture and high noise in the generated infrared images;

[0051] (2) The Restomer module is introduced into the generative network to promote the learning of multi-scale local-global representation of images and alleviate the problem of missing structure in the generated infrared images.

[0052] HRnet Network Improvements

[0053] Since the feature extraction network loses image resolution during the sampling process, in order to solve the problem of image resolution loss, the research team of the University of Science and Technology of China and Microsoft Research jointly proposed the HRnet network, which can always maintain the high resolution of the image throughout the learning process. Its model architecture is as follows: Figure 5 shown.

[0054] Channel attention can improve the performance of convolutional neural networks to a certain extent, but the current designs of many attention mechanisms are relatively complex in terms of parameter quantity and structure, which increases the complexity of the model to a certain extent, making the complex human posture detection network model more complex. In order to balance the contradiction between complexity and model performance, the Efficient Channel Attention (ECA) model takes into account the complexity and performance of the network, and can perform local cross-channel interactions between channels without reducing the dimension. This efficient channel attention can effectively reduce the complexity of the model while ensuring that the network performance is not reduced. Therefore, this paper introduces the ECA model based on HRnet to form an improved ECA-HRnet network. The network architecture is as follows: Figure 6As shown. The ECA-HRnet input image size is 256×192, and it starts with a stem consisting of two 3×3 convolutions with a stride of 1. After the stem, the resolution is reduced to 1 / 4 of the resolution of the input image feature map. Its backbone network mainly consists of 4 steps. In the first step, the network consists of 4 residual units, each residual unit consists of ECA-Bottleneck (the ECA module is embedded in the Bottleneck), and then a convolution kernel 3×3 convolution is used to reduce the width of the feature map resolution. In the second (third and fourth) steps, one (four, three) multi-resolution modules are repeated respectively. Each multi-resolution module contains two parts, one is parallel multi-resolution convolution, and the other is multi-resolution fusion. Branches of different resolutions are connected in parallel, and each branch contains four residual units. Each unit is composed of ECA-BasicBlock (ECA module is embedded in BasicBlock); branches of different resolutions perform feature fusion to complete information exchange. The process from low resolution to high resolution is mainly achieved through bilinear upsampling, and the process from high resolution to low resolution mainly adopts one or more strided convolutions (convolution layers with convolution kernel 3×3 and stride 2).

[0055] In order to obtain different local channel information, the mapping relationship between channel C and convolution kernel size K can be expressed as:

[0056] C=Φ(K)=2 (γ×K-b) (1)

[0057] Therefore, the K value will change adaptively with the number of channels and can be expressed as:

[0058]

[0059] Where K is the convolution kernel size, which represents the interaction between the current channel feature and K channel information, C represents the number of channels, b and γ are fixed values, set to 1 and 2 respectively.

[0060] Assume that the feature map input is C×H×W, and after a GAP layer, the feature map size becomes C×1×1. Since the HRnet network is relatively deep and has more intermediate layers, these intermediate layers determine the network's feature extraction and fusion capabilities, and small convolution kernels can extract more features. In addition, in human posture detection, the connectivity of human joints is related to the nodes of the front and rear joints, but not to other joint nodes. Therefore, the convolution kernel K of the one-dimensional convolution is set to 3. The Sigmoid function is used to generate the corresponding channel weights to indicate the importance of each channel feature, and the input features are multiplied and weighted to complete the feature recalibration. The ECA module structure is as follows: Figure 7 shown.

[0061] Since the ECA module structure is relatively simple and can be directly embedded into the existing network framework, the deep convolutional neural network embedded with the ECA module can be called ECA-Net. Bottleneck and BasicBlock are the classic convolution units commonly used in ResNet. ECA-BasicBlock is obtained by embedding the ECA module structure into the BasicBlock convolution unit. Its structure is as follows Figure 8 As shown in Figure 2, ECA-Bottleneck is formed by embedding the ECA module structure into the Bottleneck structure. Its network architecture is as follows: Figure 9 shown.

[0062] Lightweight Transformer-Restomer

[0063] In order to obtain clear infrared images in real time, the Restomer module is introduced into the generative network HRnet. Since convolutional neural networks (CNNs) perform well in learning generalizable image priors in large-scale data, these models have been widely used in image restoration tasks. Studies have found that the Transformer model

[17] performs well in speech and visual image processing. This model improves the shortcomings of convolutional neural networks, such as limited receptive field and incompatibility with input content, but at the same time increases computational complexity, and the computational complexity shows a quadratic growth trend with the increase of spatial resolution, so it is difficult to use in high-resolution image processing. However, by adding embedding design to the multi-head attention and feedforward network, it can capture multiple long-distance pixel interactions at the same time, which is suitable for large image processing. Unlike the general Transformer model, Restomer is not the common patch-wise when performing token calculation in the self-attention model, but pixel-wise. First, 1×1 convolution is used to increase the dimension, and then 3×3 group convolution is used to divide the features into three blocks, and finally the classic self-attention calculation is performed. The architecture of the Restomer module is as follows: Figure 10 shown.

[0064] Loss Function

[0065] The loss function of the generator and discriminator is the same as that of GAN. The discriminator's goal is to detect the simulated image produced by the generator with the highest probability. The goal of the generator is to generate images that mislead the discriminator, making it unable to correctly distinguish between true and false. The loss function of the HR-GAN network is:

[0066]

[0067] Where, The loss function of the conditional generative adversarial network is borrowed. D represents the discriminator, G represents the generator, x is the input image, y is the corresponding real image, and z is a random variable. For the discriminator, taking the maximum value of D(x,y), that is, the probability of identifying the real image is the largest, and taking the minimum value of D(x,G(x,z)), that is, when generating a simulated image, the probability of judging the simulated image as real is the smallest, that is, 1-D(x,G(x,z)) is the largest; for the generator ζ cGAN For example, the first term is a constant, and the second term can be minimized. L1 (G) is for the generator, because the generator can not only mislead the discriminator, but also be as real as possible and consistent with the content of the input image, so an L1 loss function is added.

[0068] Algorithm performance testing

[0069] Purpose of the experiment

[0070] In order to test the quality of infrared simulation images generated by the HR-GAN network, comparative experiments were carried out using four network models: CycleGAN, Pix2pix, InfraGAN, and HR-GAN. By comparing and analyzing the quality of infrared images generated by the four network models, it is shown that the HR-GAN network has significant advantages in converting visible light images to generate infrared images, laying a solid foundation for the next step of detecting differences in heterogeneous images of surface unexploded ordnance.

[0071] Experimental procedures

[0072] (1) Install the hardware platform

[0073] The experiment uses hardware such as Intel Core i7-11700K@3.60GHz CPU and NVIDIA RTX 3080GPU. The experimental environment is Python 3.7 and adopts the Pytorch learning framework.

[0074] (2) Training network model

[0075] When training the network, the Adam algorithm is first used to optimize the training network, where the two momentum parameters are preset to 0.5 and 0.999 respectively. Secondly, to ensure that the network can converge smoothly, the number of model training times is preset to 200. During the entire training process, the learning rate of the generative network is preset to 0.0002 for the first 100 times, and the learning rate of the adversarial network is preset to 0.000002. In the next 100 times, the learning rates of both networks show a linear downward trend and eventually drop to 0. The change of the loss function is shown as follows: Figure 11 shown.

[0076] (3) Calculation of evaluation indicators

[0077] Six objective evaluation indicators are used to evaluate the quality of generated images, including Mean Squared Error (MSE), Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Structural Simi-larity Index Measure (SSIM), Multi-scale Structural Similarity Index Measure (MS-SSIM) and Learned Perceptual Image Patch Similarity (LPIPS).

[0078] Experimental results

[0079] The experiments on converting visible light images of unexploded ordnance on the surface into infrared simulation images were carried out, and the CycleGAN network, Pix2pix network, InfraGAN network and the HR-GAN network proposed in this chapter were used to generate the infrared images of unexploded ordnance on the surface generated by the four network models (such as Figure 12 The obtained image information is calculated, and the evaluation indicators are shown in Table 1.

[0080] Table 1 Performance comparison of infrared image simulation algorithms

[0081]

[0082] Comparing the results, we found that compared to the unregistered image generation method CycleGAN, the registered image generation methods Pix2Pix, InfraGAN, and HR-GAN achieved better results on the dataset. Specifically, the HR-GAN network improved the peak signal-to-noise ratio (PSNR) by at least 6.66% compared to the other networks; the mean squared error (MSE) by at least 12.0%; and the mean absolute error (MAE) by at least 11.48%, both lower than the other networks. The HR-GAN network improved the structural similarity (SSIM) by at least 11.48%, reaching 95.78% structural similarity. The HR-GAN network remained the best on the multi-scale structural similarity (MS-SSIM) metric, achieving 96.9% multi-scale structural similarity. And the HR-GAN network reduced the learning-perceptual image patch similarity (LPIPS) by at least 57.03%. Judging from the calculation results of the above six evaluation indicators, the quality of infrared simulation images generated by the HR-GAN network is better than that of the CycleGAN network, Pix2pix network and InfraGAN network, which shows that the infrared simulation images generated by the HR-GAN network are more similar to the original infrared images, and the generated infrared images have clearer structural and texture information.

[0083] Difference detection of heterogeneous images based on HR-GAN algorithm

[0084] Purpose of the experiment

[0085] Through multiple experiments detecting differences between visible light and infrared images, and calculating the average detection rate, we validated the effectiveness of infrared simulation images generated by generative adversarial networks for detecting unexploded ordnance (UXO) on the ground. Visible light images of the target area to be tested were fed into the trained HR-GAN network, which then generated an infrared simulation image. The infrared image of the target area to be tested was then compared with the generated infrared simulation image to detect the nature and quantity of UXO on the ground.

[0086] Experimental procedures

[0087] To detect the difference points of heterogeneous images, we first use the trained HR-GAN network to convert the first collected visible light image (without unexploded ordnance) into an infrared simulation image. Then, we extract the feature points of the infrared simulation image and the later collected infrared image of the target area (with unexploded ordnance). The extraction results are shown in Figure 2. Figure 13 As shown; SURF algorithm is then used to perform feature cross matching between the infrared simulation image and the infrared image. The image matching results are shown in Figure 14 As shown in the figure, after the two images are processed by pixel grayscale difference, a binary image can be obtained, and finally the target information can be detected by threshold processing, as shown in the figure. Figure 15 shown.

[0088] Experimental results

[0089] Through 45 heterogeneous image detection experiments, parachute-dropped structures were used for surface unexploded ordnance detection. By counting the ratio of the number of detected targets to the actual number of preset targets, the target detection results were obtained as follows: Figure 16 As shown, the final average target detection rate is 88.89%.

[0090] Experiments have shown that although the HR-GAN network can generate infrared simulation images of surface unexploded ordnance, and the structural similarity of the generated infrared simulation images compared with the original images is 95.78%, due to the small proportion of pixels occupied by surface unexploded ordnance images under high-altitude imaging conditions, noise may be generated in the generated infrared simulation images. When cross-matching with the actual infrared images with surface unexploded ordnance, the pixels of the surface unexploded ordnance may be offset, so the target cannot be detected, resulting in an average target detection rate of 88.89%. At the same time, the generated infrared simulation images may lack some image features. After pixel difference processing between the original infrared image and the infrared simulation image, noise will appear on the difference map, resulting in false alarms and misjudgments.

[0091] in conclusion

[0092] In order to ensure uninterrupted detection around the clock and in all weather conditions, and to effectively solve the problem of detecting heterogeneous images of unexploded ordnance on the surface, the present invention optimizes and improves the existing generative adversarial network model, thereby proposing a HR-GAN network model. The various indicators of the infrared simulation images generated by this model are improved compared to those of other networks. Target detection experiments have shown that the average target detection rate of the HR-GAN-based method for detecting difference points in heterogeneous images of unexploded ordnance on the surface reaches 88.89%, which can basically meet the requirement of a detection accuracy higher than 85%. However, since noise is easily introduced in the process of generating infrared simulation images, and some small target image information is lost, problems such as missed detection and false detection occur when using infrared simulation images and actual collected infrared images for small target detection.

[0093] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0094] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0095] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the heterogeneous image search methods based on a generative adversarial network in the above embodiments.

[0096] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0097] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0098] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0099] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A heterogeneous image search method based on generative adversarial networks, characterized in that: The following steps are included: The visible light image of the target area to be tested is input into the trained HR-GAN network, and an infrared simulation image is generated from the visible light image. The infrared image of the target area to be tested is then compared and analyzed with the generated infrared simulation image to detect the nature and quantity of unexploded ordnance on the surface. The HR-GAN network training steps include introducing a high-resolution network, HRnet, as a generative network in the Pix2pix architecture to maintain the high-resolution representation of the image in the network and alleviate the problems of unclear texture and high noise in the generated infrared images. The Restomer module is introduced into the generative network to promote the learning of multi-scale local-global representations of images and alleviate the problem of missing structure in generated infrared images. The ECA model is introduced on the basis of HRnet to form an improved ECA-HRnet network; Among them, ECA-HRnet input image size is 256×192, and it starts with a stem composed of two 3×3 convolutions with a stride of 1; after the stem, the resolution is reduced to 1 / 4 of the input image feature map resolution; Its backbone network consists of four steps. In the first step, the network consists of four residual units. Each residual unit consists of an ECA-Bottleneck, which embeds the ECA module into a Bottleneck. A 3×3 convolution kernel is then used to reduce the width of the feature map resolution. In the second step, a multi-resolution module is repeated separately; In the third step, four multi-resolution modules are repeated one by one; In the fourth step, three multi-resolution modules are repeated respectively; Each multi-resolution module contains two parts: one is parallel multi-resolution convolution and the other is multi-resolution fusion; Branches of different resolutions are connected in parallel, and each branch contains four residual units. Each unit is composed of an ECA-BasicBlock, that is, an ECA module embedded in a BasicBlock. Branches of different resolutions perform feature fusion to complete information exchange. The process from low resolution to high resolution is mainly achieved by bilinear upsampling, and the process from high resolution to low resolution uses one or more strided convolutions, that is, convolution layers with a convolution kernel of 3×3 and a stride of 2. The loss function of the HR-GAN network is: （3） Where, The loss function of the conditional generative adversarial network is borrowed. D represents the discriminator, G represents the generator, x represents the input image, y represents the corresponding real image, and z represents the random variable.

2. The heterogeneous image search method based on a generative adversarial network according to claim 1, characterized in that: In order to obtain different local channel information, the mapping relationship between channel C and convolution kernel size K is expressed as: （1） Therefore, the K value will change adaptively with the number of channels, expressed as: （2） Where K is the convolution kernel size, which represents the interaction between the current channel feature and K channel information, C represents the number of channels, b and γ are fixed values, set to 1 and 2 respectively.

3. The heterogeneous image search method based on a generative adversarial network according to claim 2, characterized in that: The Restomer module is introduced into the generative network HRnet; first, 1×1 convolution is used to increase the dimension, then 3×3 group convolution is used to divide the features into three blocks, and finally the classic self-attention calculation is performed.

4. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 3.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 3.

Images