A Fusion Method for Heterogeneous Remote Sensing Images
Through homogeneous change image generation model, the optical image and radar image are fused to generate fusion images containing content features and style features, solving the problems of information integrity and style texture under complex terrain conditions, and achieving fast and efficient heterogeneous remote sensing images Fusion.
Patent Information
- Application Number
- CN202210493488.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-07
AI Technical Summary
The prior art is difficult to effectively fuse optical images and radar images under complex terrain conditions, ensuring the integrity of content information and making the style texture approach the target image.
A homogeneous change image generation model is adopted to generate iterative image samples by learning optical images and radar image samples, and a loss function is constructed to update model parameters and generate a fusion image. The fusion image contains the content features of the optical images and the style and form features of the radar image.
The content information integrity and style texture of the fused image under complex terrain conditions are improved, and the fused image is quickly generated, which is suitable for homogeneous spatial transformation of heterogeneous remote sensing images under complex terrain conditions of various land objects.
Smart Images

Figure CN114863235B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing, and in particular to a method for fusing heterogeneous remote sensing images. Background Art
[0002] Remote sensing images include optical images and radar images, which refer to image data about land objects, sea surfaces, etc. obtained by space-based or air-based imaging sensors. Optical images have high resolution, rich color information, and are easy to interpret; however, the image quality is poor under poor lighting conditions such as at night and under cloud cover. Radar has the advantages of all-day, all-weather, and strong penetration, but the quality of radar images is affected by the clutter level and the interpretability is relatively poor.
[0003] How to complement the advantages of optical images and radar images to obtain images that can ensure the integrity of content information without loss even under complex terrain conditions and whose transformed style texture is close to the target is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of the above problems, an embodiment of the present invention provides a method for fusing heterogeneous remote sensing images to overcome the above problems or at least partially solve the above problems.
[0005] An embodiment of the present invention provides a method for fusing heterogeneous remote sensing images, the method comprising:
[0006] Acquire optical and radar images;
[0007] Inputting the optical image and the radar image into a homogeneous variation image generation model to obtain a fused image, wherein the fused image includes content features of the optical image and style features of the radar image;
[0008] The homogeneous variation image generation model is a model that learns to generate image samples to be iterated based on optical image samples and radar image samples, and iterates the image samples to be iterated to generate fused image samples.
[0009] Optionally, the homogeneous variation image generation model is obtained by training an initial model, wherein the initial model includes an image generation module and a first feature extraction module with fixed weights; the step of training the initial model includes at least:
[0010] Inputting the optical image samples and the radar image samples into the initial model;
[0011] Using the first feature extraction module, extracting a first content feature of the optical image sample and a first style feature of the radar image sample;
[0012] Inputting the first content feature and the first style form feature into the image generation module, and outputting a first intermediate image sample;
[0013] Constructing a loss function according to the current input and output of the image generation module;
[0014] Based on the loss function, the model parameters of the initial model are updated to obtain the homogeneous variation image generation model.
[0015] Optionally, the initial model further includes a second feature extraction module with random weights; and the step of training the initial model further includes:
[0016] Acquire the to-be-iterated image sample generated by the image generation module, wherein the to-be-iterated image sample is: an image sample that minimizes the loss function when the input of the image generation module is the first content feature and the first style form feature;
[0017] Using the second feature extraction module, extracting a second content feature of the optical image sample and a second style feature of the radar image sample;
[0018] Acquire a second intermediate image sample generated by the image generation module, where the second intermediate image sample is: an image sample that minimizes the loss function when the input of the image generation module is the second content feature and the second style form feature;
[0019] Iterating the image sample to be iterated according to the second intermediate image sample, comprising:
[0020] When an iteration termination condition is met, the second intermediate image sample is used as the fused image sample output by the homogeneous variation image generation model.
[0021] Optionally, when an iteration termination condition is satisfied, using the second intermediate image sample as the fused image sample output by the homogeneous variation image generation model comprises:
[0022] Calculating an iterative convergence error between the second intermediate image sample and the to-be-iterated image sample;
[0023] When the iterative convergence error is less than an iterative convergence error threshold, or when the number of times the second intermediate image samples are generated reaches a preset number, the second intermediate image samples are used as the fused image samples output by the homogeneous variation image generation model.
[0024] Optionally, the iterating the image samples to be iterated according to the second intermediate image samples further includes:
[0025] When the iteration termination condition is not met, the second intermediate image sample is used as the image sample to be iterated, and the steps of: using the second feature extraction module to extract the second content feature of the optical image sample and the second style form feature of the radar image sample are performed.
[0026] Optionally, when the current input of the image generation module is the first content feature and the first style feature, constructing the loss function according to the current input and output of the image generation module includes:
[0027] Using the first feature extraction module, extracting a first content feature and a first style feature of the first intermediate image sample;
[0028] The loss function is constructed according to the difference between the first content feature of the first intermediate image sample and the first content feature of the optical image sample, and the difference between the first style form feature of the first intermediate image sample and the first style form feature of the radar image sample.
[0029] Optionally, when the current input of the image generation module is the second content feature and the second style form feature, constructing the loss function according to the current input and output of the image generation module includes:
[0030] Using the second feature extraction module, extracting a second content feature and a second style feature of the second intermediate image sample;
[0031] The loss function is constructed according to the difference between the second content feature of the second intermediate image sample and the second content feature of the optical image sample, and the difference between the second style form feature of the second intermediate image sample and the second style form feature of the radar image sample.
[0032] Optionally, after obtaining the fused image, the method further includes:
[0033] Acquire a target radar image, where the target radar image is an image from the same radar as the radar image and observes the same area as the optical image, and the area includes a target proposal area;
[0034] Inputting the fused image and the target radar image into a classifier to obtain target proposal regions of the fused image and the target radar image respectively;
[0035] Obtaining a common target proposal region of the fused image and the target radar image according to respective target proposal regions of the fused image and the target radar image;
[0036] The common target proposal region and the target radar image are fused to obtain an image with enhanced target proposal region.
[0037] Optionally, the classifier is trained by the following steps:
[0038] Acquire a plurality of labeled box samples from the radar image samples and the fused image samples, wherein the labels indicate whether the box samples are the target proposal areas;
[0039] Obtaining a gradient norm map of each of the plurality of box samples;
[0040] Inputting the gradient norm map of each of the plurality of box samples into an initial classifier to obtain a classification prediction result of whether each of the plurality of box samples is a target proposal area;
[0041] According to the respective labels and classification prediction results of the plurality of box samples, the parameters of the initial classifier are updated to obtain the classifier.
[0042] Optionally, the fusing the common target proposal region and the target radar image to obtain an image enhanced with the target proposal region includes:
[0043] Based on the Copra model, a joint probability density function of background clutter in the target radar image and the fused image is calculated, wherein the joint probability density function represents the distribution of background clutter common to the target radar image and the fused image;
[0044] Calculating background clutter suppression matrices of the target radar image and the fused image according to the joint probability density function;
[0045] Acquiring the similarity between the target radar image and the fused image;
[0046] An image enhanced by the target proposal region is obtained according to the common target proposal region, the target radar image, the background clutter suppression matrix and the similarity.
[0047] The embodiments of the present invention include the following advantages:
[0048] In this embodiment, the homogeneous variation image generation model can generate a fused image directly based on the optical image and the radar image, which is relatively fast. The fused image includes the content features of the optical image and the style and form features of the radar image, and combines the advantages of the optical image and the radar image. Therefore, the fused image can still ensure the integrity of the content information without loss even under complex terrain conditions, and the style texture is close to the radar image. On the basis of learning to generate the image samples to be iterated based on the optical image samples and the radar image samples, the homogeneous variation image generation model further learns to iteratively generate the fused image samples from the image samples to be iterated. Therefore, the fused image generated by the homogeneous variation image generation model can further improve the homogeneity in the new content feature and style feature subspace, and is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0050] Figure 1 is a flowchart of the steps of a method for fusing heterogeneous remote sensing images in an embodiment of the present invention;
[0051] Figure 2 is a schematic diagram of the structure of the first feature extraction module in an embodiment of the present invention;
[0052] Figure 3 It is a schematic diagram of the process of loss function construction and notification transformation feature fusion in an embodiment of the present invention;
[0053] Figure 4 is a schematic diagram of the image iteration process in an embodiment of the present invention;
[0054] Figure 5 This is the training process and testing process of target enhancement in the embodiment of the present invention;
[0055] Figure 6 It is a flow chart of a heterogeneous remote sensing image fusion method based on homogeneous transformation and target enhancement in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0057] Reference Figure 1 As shown, a flowchart of a method for fusing heterogeneous remote sensing images according to an embodiment of the present invention is shown. Figure 1 As shown, the fusion method of heterogeneous remote sensing images may specifically include the following steps:
[0058] Step S11: acquiring an optical image and a radar image;
[0059] Step S12: Input the optical image and the radar image into a homogeneous variation image generation model to obtain a fused image, wherein the fused image includes content features of the optical image and style features of the radar image; wherein the homogeneous variation image generation model is a model that has learned to generate image samples to be iterated based on optical image samples and radar image samples, and iterates the image samples to be iterated to generate fused image samples.
[0060] Optical images are images obtained by optical photography systems, which can be visible light black and white panchromatic images, color infrared images, multi-band photography images, thermal infrared photography images, etc. obtained by aerial remote sensing. Radar images refer to SAR images generated by SAR (Synthetic Aperture Radar) systems, which can be installed on aircraft, satellites, spacecraft and other flight platforms.
[0061] The homogeneous variation image generation model can generate a fused image based on the input optical image and radar image, where the radar image is used to provide style features. Therefore, it is not required that the radar image and the optical image observe the same area, and there is no need to align the radar image and the optical image. The following article will introduce how to enhance the target proposal area based on the target radar image and the fused image. In order to enhance the target proposal area, the radar image used to generate the fused image should come from the same radar as the target radar image.
[0062] Compared with generating homogeneous transformation images using optical images and radar images that do not require registration, generating homogeneous transformation images (fused images) using optical images and registered radar images has a wider range of application scenarios, faster speed, and higher practicality. For example, when observing a scene for target recognition or change detection, it is assumed that an optical image is obtained first, and if you want to obtain a radar image that is registered with the optical image, you need to wait for a long time, which makes the homogeneous transformation time-consuming. However, by adopting the technical solution of the present application, there is no need to align the optical image and the radar image, and the homogeneous transformation can be directly completed based on the homogeneous data of the radar sensor, providing more time-effective data for subsequent target enhancement, which is conducive to the rapid development of subsequent tasks. Therefore, the technical solution of the present application using optical images and radar images that do not require registration to perform homogeneous transformation to obtain fused images has a wider range of application scenarios and faster speed.
[0063] Sometimes there may be shadows, uneven lighting conditions and other effects in optical images. In order to generate better fused images, the optical images can be preprocessed, and the radar images and preprocessed optical images can be input into the homogeneous variation image generation model. Optionally, the Sobel operator can be used to preprocess the optical image to extract the edge contours of the minute ground object information in the optical image, and 20% of the original information can be added, and then Gaussian noise with a normalized mean of 0.15 to 0.25 and a normalized variance of 0.01 can be added. Preprocessing the optical image can overcome the effects of shadows, uneven lighting conditions, etc., and retain the basic image semantic information.
[0064] The optical image and the radar image are input into the homogeneous variation image generation model, which can extract the content features of the optical image and the style and form features of the radar image, and then perform homogeneous spatial transformation on the content features of the optical image and the style and form features of the radar image to generate a fused image. The content features of the fused image are close to those of the optical image, and the style and form features of the fused image are close to those of the radar image. The fused image is sufficient to ensure the integrity of the content semantic information of the image, and can describe the image content and style and form information of the remote sensing image at all scales from concrete to abstract even in complex terrain environments, and can fully and accurately perform homogeneous spatial transformation of heterogeneous remote sensing images under complex terrain conditions of various landforms.
[0065] The homogeneous variation image generation model is a model that learns how to generate iterative image samples and how to optimize iterative image samples through optical image samples and radar image samples. The training steps of the homogeneous variation image generation model will be described in detail later.
[0066] By adopting the technical solution of the embodiment of the present application, the homogeneous variation image generation model can generate a fused image directly based on the optical image and the radar image, which is relatively fast. The fused image includes the content features of the optical image and the style and form features of the radar image, and combines the respective advantages of the optical image and the radar image. Therefore, the fused image can still ensure that the integrity of the content information is not lost even under complex terrain conditions, and the style texture is close to the radar image. Based on learning to generate image samples to be iterated based on optical image samples and radar image samples, the homogeneous variation image generation model further learns to iteratively generate fused image samples from image samples to be iterated. Therefore, the fused image generated by the homogeneous variation image generation model can further improve the homogeneity in the new content feature and style and form feature subspaces, and is more accurate.
[0067] The homogeneous variation image generation model is obtained by training the initial model, and the initial model includes an image generation module and a first feature extraction module with fixed weights. Among them, the weights of the image generation module included in the initial model are to be trained, and the image generation module is used to perform homogeneous spatial transformation on the input content features and style form features to generate images / image samples. Through training, the content features of the images / image samples output by the image generation module will become closer and closer to the content features of the input image generation module, and the style form features will become closer and closer to the style form features of the input image generation module. It can be understood that the homogeneous variation image generation model also has an image generation module, and the weights of the image generation module in the homogeneous variation image generation model are trained.
[0068] The first feature extraction module with fixed weights can be a convolutional neural network with a multi-layer structure constructed based on VGGnet (a deep convolutional neural network), from which the content features of the optical image and the style features of the radar image can be extracted. It can be understood that the first content feature refers to the content feature extracted from the first feature extraction module with fixed weights, and the first style feature refers to the style feature extracted from the first feature extraction module with fixed weights.
[0069] Figure 2 The schematic diagram of the structure of the first feature extraction module is shown. The first feature extraction module can include multiple convolutional layers, activation layers and pooling layers, and can effectively extract image features under complex terrain conditions. However, unlike VGGnet, the first feature extraction module has the following three characteristics: there is no fully connected layer, the weights of the entire network are fixed, and the output of the network is each selected network layer.
[0070] The fully connected layer is used to output the global information of the image, but it loses the local position information of the image. Under complex terrain conditions with multiple objects, different objects are contained in each position of the image, so the position information can communicate the homogeneous spatial transformation mapping of the same local objects between heterogeneous images. The function of the fully connected layer does not meet the requirements of homogeneous spatial transformation, so the first feature extraction module does not have a fully connected layer.
[0071] The weights of the first feature extraction module are based on the results of VGGnet training. The network of the first feature extraction module does not undergo traditional training or fine-tuning of network parameters, but rather follows a similar network working process, extracting features under complex terrain conditions based on the output of the trained and fine-tuned network.
[0072] The output of the first feature extraction module is not the VGGnet feedforward network terminal output, but the output of each selected layer of the convolutional neural network. Therefore, the homogeneous spatial transformation features of remote sensing images from concrete to abstract and multi-scale under complex terrain conditions of various objects can be extracted.
[0073] like Figure 2 As shown, according to the characteristics of the outputs of different layers of the constructed convolutional neural network, the corresponding network layer can be selected to separate the content features and style features of the output image.
[0074] Alternatively, the deeper or deepest convolutional layer in the VGGnet network close to the object scale can be used directly as the content feature output, because the content semantic information of the object in the remote sensing image is generally one of the most abstract features at the scale of the specified object. For convenience, the content feature of the image is recorded as Fc.
[0075] Optionally, the pooling layers of VGGnet at each scale can be used to extract style form features based on the Gram texture operator. The style form of the image is described by the texture features of each scale of the image. The VGGnet network with multiple scales can effectively cover the texture information of each scale of the radar image. At the same time, the texture features are suitable for extraction by the mean operation of the VGGnet pooling layer. Finally, the Gram texture operator is used to convert the pooling output of each scale into a style form feature representation. For convenience, the style form feature of the image is recorded as Fs.
[0076] The content features and style features extracted based on the first feature extraction module can not only cover the more specific and smaller-scale homogeneous space transformation features, but also extract more abstract and larger-scale features through a multi-level network structure. In addition, the content features and style features in the homogeneous space transformation features are separated to ensure the integrity of the content semantic information of the image and improve the accuracy of the transformed synthetic image. In summary, even in complex terrain environments, the extracted content features and style features can describe the image content and style information of remote sensing images at all scales from specific to abstract, and can fully and accurately perform homogeneous space transformation of heterogeneous remote sensing images under complex terrain conditions of various landforms.
[0077] Based on the above technical solution, in order to train the initial model, it is first necessary to obtain optical image samples and radar image samples. The optical image samples and radar image samples do not need to observe the same area, and do not need to be registered.
[0078] The optical image sample and the radar image sample are input into the initial model, and the first feature extraction module in the initial model can extract the first content feature of the optical image sample and the first style form feature of the radar image sample. The image generation module can fuse the content features and style form features between heterogeneous images to perform homogeneous spatial transformation of heterogeneous remote sensing images. Therefore, the first content feature and the first style form feature are input into the image generation module, and the image generation module can generate a first intermediate image sample.
[0079] According to the current input and output of the image generation module, a loss function can be constructed. Based on the loss function, the model parameters of the initial model are updated to obtain a homogeneous variation image generation model.
[0080] In the case where the current input of the image generation module is the first content feature and the first style feature, constructing the loss function may include: using the first feature extraction module to extract the first content feature and the first style feature of the first intermediate image sample; constructing the loss function according to the difference between the first content feature of the first intermediate image sample and the first content feature of the optical image sample, and the difference between the first style feature of the first intermediate image sample and the first style feature of the radar image sample. The loss function constructed based on the first intermediate image sample can be expressed by the following formula:
[0081]
[0082] in, represents the loss function constructed based on the first intermediate image sample; is the first intermediate image sample; is an optical image sample / a preprocessed optical image sample; is a radar image sample; express The first content feature; express The first content feature; express The first stylistic form characteristics; express The first style form feature of ; ||·|| represents the 2-norm. λ c is a preset constant, generally set in the range of 0.01-0.05, which respectively represents the weight of content features and style features in the fusion process. The result of homogeneous space transformation can be expressed as:
[0083] The loss function consists of two additive terms. The first additive term measures and The distance between the content features is expected to be as small as possible so that and The content characteristics between them are as close as possible; the second additive term measures and The distance between the style features between the two; it is hoped that the distance is as small as possible so that and The style form characteristics between them are as close as possible. In this way, in the process of homogeneous space transformation, it can be ensured that the content semantic information of the image is not lost or destroyed, and the style form of the first intermediate image sample is ensured to be similar to the style form of the radar image sample.
[0084] By constructing the loss function and minimizing the result of homogeneous space transformation, the content features and style features of heterogeneous images can be fused respectively. In this way, even in complex terrain conditions with multiple objects, the integrity of the content information can be guaranteed not to be lost during the homogeneous space transformation, and the transformed style texture can be made close to the target.
[0085] In the actual execution process, it is necessary to initialize To meet the calculation requirements in the loss function. Generally, the initialization image can use the input image, or use the fusion result of fixed network weights, and perform homogeneous feature transformation fusion output to update the first intermediate image sample. Figure 3 A schematic diagram of the process of loss function construction and notification transformation feature fusion is shown.
[0086] It can be understood that updating the model parameters of the initial model based on the loss function constructed based on the first intermediate image sample is essentially mainly updating the weights of the image generation module.
[0087] On the basis of the above technical solution, the initial model can also include a second feature extraction module with random weights. The second feature extraction module has a similar structure to the first feature extraction module, including multiple convolutional layers, activation layers and pooling layers, and can effectively extract image features under complex terrain conditions. The second feature extraction module has the following three characteristics: there is no fully connected layer, the weights of the entire network are randomized, and the output of the network is each selected network layer. Among them, the randomization of the weights of the entire network means that the second feature extraction module first obtains the trained weights according to the traditional method of training or fine-tuning the parameter values of the network, and then mixes Gaussian white noise on the trained weights to achieve the network output of the extracted features under complex terrain conditions of various landforms.
[0088] Based on the training of the initial model using the loss function constructed based on the first intermediate image sample, the initial model can be further trained. The first content feature and the first style form feature are input into the image generation sample to obtain the image sample to be iterated generated by the image generation module; the image generation module has been trained based on the loss function constructed based on the first intermediate image sample. The image sample to be iterated is: when the input of the image generation module is the first content feature of the optical image sample and the first style form feature of the radar image sample, the image sample that minimizes the loss function.
[0089] The second content feature of the optical image sample and the second style form feature of the radar image sample are extracted by using the second feature extraction module; the second content feature refers to the content feature extracted from the second feature extraction module with random weights, and the second style form feature refers to the style form feature extracted from the second feature extraction module with random weights.
[0090] The second content feature and the second style feature are input into the image generation sample to obtain the second intermediate image sample generated by the image generation module. When the current input of the image generation module is the second content feature and the second style feature, a loss function can be constructed according to the current input and output of the image generation module.
[0091] A second feature extraction module is used to extract second content features and second style features of the second intermediate image sample. A loss function is constructed based on the difference between the second content features of the second intermediate image sample and the second content features of the optical image sample, as well as the difference between the second style features of the second intermediate image sample and the second style features of the radar image sample, and the weights of the image generation module are updated based on the loss function.
[0092] It can be understood that, in the process of updating the weights of the image generation module based on the loss function, a plurality of second intermediate image samples are generated according to the second content features of the optical image samples and the second style features of the radar image samples. The second content features of the optical image samples and the second style features of the radar image samples are input into the updated image generation module, and the output second intermediate image samples are: image samples that minimize the loss function when the input of the image generation module is the second content features of the optical image samples and the second style features of the radar image samples.
[0093] The image samples to be iterated are iterated according to the second intermediate image sample that minimizes the loss function. Specifically, when the iteration termination condition is met, the second intermediate image sample that minimizes the loss function is used as the fused image sample, which is the output of the homogeneous variation image generation model.
[0094] The iteration termination condition may be: the iterative convergence error between the second intermediate image sample that minimizes the loss function and the image sample to be iterated is less than the iterative convergence error threshold, or the number of iterations reaches a preset number. The iterative convergence error between the second intermediate image sample that minimizes the loss function and the image sample to be iterated can be obtained by Calculate, where K represents the number of iterations, initially K = 0, represents the second intermediate image sample obtained at the K+1th iteration, Characterize the second intermediate image sample obtained after the Kth iteration; when K=0, the second intermediate image sample obtained is the image sample to be iterated obtained based on the first content features of the optical image sample and the first style and form features of the radar image sample; when K=1, the second intermediate image sample obtained is: the second intermediate image sample that minimizes the loss function based on the second content features of the optical image sample and the second style and form features of the radar image sample extracted for the first time by the second feature extraction module; when K=2, the second intermediate image sample obtained is: the second intermediate image sample that minimizes the loss function based on the second content features of the optical image sample and the second style and form features of the radar image sample extracted for the second time by the second feature extraction module.
[0095] Calculating the iterative convergence error between the second intermediate image sample and the image sample to be iterated once can be regarded as one iteration. In one iteration, the second intermediate image sample that minimizes the loss function can be generated once, or the second feature extraction module can extract the second content feature and the second style form feature once.
[0096] In the case where the iteration termination condition is not met, the second intermediate image sample that minimizes the loss function can be used as the image sample to be iterated, and then the second content feature of the optical image sample and the second style form feature of the radar image sample are extracted again by the second feature extraction module, and based on the second content feature of the optical image sample and the second style form feature of the radar image sample extracted again, a second intermediate image sample that minimizes the new loss function is generated. The new loss function is constructed based on the second content feature and the second style form feature of the newly generated second intermediate image sample, and the second content feature of the optical image sample and the second style form feature of the radar image sample extracted again.
[0097] Figure 4 A schematic diagram of the image iteration process is shown. The iteration process may include the following steps:
[0098] Step S21: specify the maximum number of iterations K and the iterative convergence error threshold ε;
[0099] Step S22: Obtain the image sample to be iterated and initialize it as Let k = 0, k is the current iteration number;
[0100] Step S23: Input and obtain The characteristics of each content and style and form characteristics Generate the second intermediate image, that is, get the new input image And set k:=k+1;
[0101] Step S24: If k>K, or Then the iteration stops, and the fused image sample output by the final model is Otherwise, return to step S23;
[0102] The fused image samples synthesized by homogeneous spatial transformation that are updated as the iteration termination output can generally be set to a maximum number of iterations K=100, ε=0.01.
[0103] The previous article introduced how the initial model generates image samples to be iterated based on optical image samples and radar image samples, and iterates the image samples to be iterated to obtain fused image samples; it can be understood that the method of the homogeneous variation image generation model to generate images to be iterated based on optical images and radar images, and to iterate the images to be iterated to obtain fused images is similar, except that there is no need to construct a loss function, and the image generation module can directly output the fused image.
[0104] After the heterogeneous remote sensing images are transformed into a fused image, the final fused image can be obtained by using fusion methods such as principal component analysis, additive fusion, and multiplicative fusion.
[0105] Since remote sensing image fusion generally undertakes tasks such as target detection, methods based on homogeneous transformation and existing fusion methods cannot effectively enhance the target proposal area in the fused image and suppress clutter, which can easily reduce the subsequent target detection performance. Therefore, based on the above technical solution, the target radar image can also be enhanced based on the fused image. Target enhancement of the target radar image can significantly improve the target-clutter ratio in the fused image, which helps to improve the subsequent target detection and recognition performance. Figure 5 The training process and the testing process of target enhancement are shown, and the training process and the testing process of target enhancement will be described in detail later.
[0106] Optionally, the common target area in the remote sensing image can be enhanced based on the target proposal, and the common clutter area in the remote sensing image can be suppressed based on the Kopra model.
[0107] Get the target radar image. The target radar image is an image from the same radar as the radar image and observes the same area as the optical image. The observed area includes the target proposal area. The target radar image and the optical image must be configured and have the same size. Otherwise, they need to be registered and the low-resolution image must be interpolated to make the two images the same size.
[0108] The fused image and the target radar image are input into a classifier to obtain respective target proposal regions of the fused image and the target radar image; a common target proposal region of the fused image and the target radar image is obtained according to the respective target proposal regions of the fused image and the target radar image; and the common target proposal region and the target radar image are fused to obtain an image with enhanced target proposal region.
[0109] The target radar image and optical image are images that observe the same area and are registered. The fused image is an image that retains the content characteristics of the optical image. Therefore, the target radar image and the fused image have a corresponding relationship. The structure, quantity, formation and other information of the target are contained in the target radar image and the fused image respectively. Therefore, there is an inherent correlation between the target radar image and the fused image, that is, there is joint sparsity of the data.
[0110] The target proposal regions of the fused image and the target radar image are identified based on the classifier. The target proposal region can be the target to be detected, such as a ship on the sea, a house somewhere, etc. Pixels outside the target proposal region can be considered as clutter. For example, when the target is a ship, clutter includes the sea surface, clouds, etc. The classifier is a model that learns how to identify the target proposal region based on the fused image samples and the radar image samples, where the radar image samples and the fused image samples used to train the classifier are also registered.
[0111] Optionally, inputting the fused image and the target radar image into a classifier to obtain target proposal regions of the fused image and the target radar image, respectively, includes: adjusting the scales of the fused image and the target radar image to obtain the fused image at multiple scales and the target radar image at multiple scales; inputting the fused image at multiple scales and the target radar image at multiple scales into the classifier to obtain target proposal regions respectively contained in the iterative images at multiple scales and target proposal regions respectively contained in the target radar images at multiple scales; integrating the target proposal regions respectively contained in the fused images at multiple scales to obtain the target proposal region of the fused image; integrating the target proposal regions respectively contained in the target radar images at multiple scales to obtain the target proposal of the target radar image.
[0112] The fused image and the target radar image can be adjusted to fused images and target radar images at multiple scales. The classifier and a sliding window (the size is the same as the size of the downsampled sample box used when training the initial classifier) are used to identify the target proposal area contained in the fused image and the target radar image at each scale. The classifier moves step by step along the sliding window to recognize the entire image. The classifier will determine whether the content included in the sliding window contains the target. By connecting all the sliding window areas that are judged to contain the target during the movement of the classifier, the target proposal area of an image can be obtained.
[0113] Then the fused image and target radar image at each scale are scaled to the original scale to obtain the target proposal region of the fused image. A pixel in the fused image at the original scale has a corresponding pixel in the fused image at each scale. If any of the corresponding pixels belongs to the target proposal region, then the pixel also belongs to the target proposal region in the fused image at the original scale. A pixel in the target radar image at the original scale has a corresponding pixel in the target radar image at each scale. If any of the corresponding pixels belongs to the target proposal region, then the pixel also belongs to the target proposal region in the target radar image at the original scale.
[0114] According to the target proposal regions of the fused image and the target radar image, the common target proposal region of the fused image and the target radar image can be obtained. Alternatively, the target proposal region of the fused image and the target proposal region of the target radar image can be represented by matrices respectively, and the Hadamard product of the two matrices can be taken to obtain the common target proposal region of the fused image and the target radar image.
[0115] Because the target proposal areas of the fused image and the target radar image include the target proposal areas of images at each scale, the missed alarm rate is effectively reduced at multiple scale angles.
[0116] Optionally, the classifier is trained by the following steps: obtaining a plurality of labeled box samples from radar image samples and fused image samples, the labels representing whether the box samples are the target proposal areas; obtaining a gradient norm map of each of the plurality of box samples; inputting the gradient norm map of each of the plurality of box samples into an initial classifier to obtain a classification prediction result of whether each of the plurality of box samples is the target proposal area; and updating the parameters of the initial classifier according to the labels of each of the plurality of box samples and the classification prediction results to obtain the classifier.
[0117] The downsampled sample box can be used to obtain multiple box samples from the radar image samples and the fused image samples, and a label can be given to each box sample. Optionally, the downsampled sample box size can be 8. Calculate the image gradient norm G(x,y) of each box sample:
[0118]
[0119] Among them, gx and gy represent the horizontal gradient and vertical gradient at the coordinate (x, y), respectively, x, y = 1, 2, .., D, and then the gradient norm map G of each downsampled sample box is obtained. In this way, unimportant detail information can be filtered out, and the salient edges of the image can be enhanced, and it has good robustness to objects of different scales.
[0120] The classifier can be a support vector machine (SVM). By outputting the gradient norm map of each box sample to the initial classifier, a binary classification prediction result of whether each box sample is clutter can be obtained. The label of each box sample is used as a constraint to update the parameters of the initial classifier to obtain a trained classifier.
[0121] Optionally, fusing the common target proposal region and the target radar image to obtain an image enhanced by the target proposal region may include: calculating a joint probability density function of background clutter in the target radar image and the fused image based on a Copra model, wherein the joint probability density function represents a distribution of background clutter common to the target radar image and the fused image; calculating a background clutter suppression matrix of the target radar image and the fused image according to the joint probability density function; acquiring a similarity between the target radar image and the fused image; and obtaining an image enhanced by the target proposal region according to the similarity among the common target proposal region, the target radar image, and the background clutter suppression matrix.
[0122] Copula is a concept in probability theory that can decompose the joint distribution of a multivariate random variable into a function of the marginal distribution of each variable and the correlation structure between the variables.
[0123] The background clutter in the image is useless to the target proposal area and will interfere with the recognition of the target proposal area, so the background clutter needs to be suppressed. The joint probability density function represents the common background clutter distribution in the target radar image and the fused image. Therefore, the common suppression coefficient distribution of the background clutter of the two images can be obtained based on the joint probability density function, that is, the background clutter suppression matrix. The larger the value of the joint probability density function of a certain area, the larger the common suppression coefficient of the two images in this area.
[0124] Calculate the similarity between the target radar image and the fused image. Since the two images are about the same target, the higher the similarity of a part at the same position in the two images, the greater the probability that the two images contain the same content at this position. For the image to be obtained, the two parts are more worthy of being fused together, so that the fused image contains more information. Therefore, the higher the similarity of a part is calculated, the larger the fusion weight should be marked. Calculate the similarity function of each part of the target radar image and the fused image, and further convert the obtained similarity function into a fusion weight function.
[0125] Alternatively, the similarity k can be calculated using the following formula:
[0126]
[0127] Among them, (m, n) represents the pixel point at each position, I SAR Characterize the target radar image, Represents the fused image, M and N are the sizes of the two images.
[0128] The fusion weight function α can be calculated by the following formula:
[0129] α=-logκ
[0130] The higher the similarity between the target radar image and the fusion image, the smaller the k value and the larger the α value.
[0131] According to the background clutter suppression matrix, the common target proposal region is corrected to obtain the corrected proposal region; according to the fusion weight function, the corrected proposal region and the target radar image are weighted summed to obtain an image enhanced by the target proposal region. The enhanced image of the target proposal region can be obtained by the following formula:
[0132] I F =I SAR +α(O⊙P)
[0133] Among them, I F For the image enhanced by the target proposal area, O represents the common target proposal area, and P represents the background clutter suppression matrix.
[0134] Optionally, the model parameters of the Copra model include Copra parameters and function forms, which can be obtained by the following steps: obtaining background clutter pixel points of the radar image sample and the fused image sample respectively; generating clutter pixel pairs according to the background clutter pixel points at the same position in the radar image sample and the fused image sample; substituting the clutter pixel pairs into the edge inference function to obtain the model parameters of the Copra model.
[0135] The model parameters of the Copra model can be obtained by solving the following optimization problem:
[0136]
[0137] in, is a pair of clutter pixels; c represents the Coppola density function, θ c represents the parameter in the Coppola density function, f S represents the probability density function of background clutter in the target radar image, f A represents the probability density function of background clutter in the fused image, represents the corresponding estimated value, Θ c Represents θ c The value range of represents the Kopra density dictionary, H0 represents the clutter occupying the current pixel pair, Represents the log-likelihood function:
[0138]
[0139] Among them, F S and F A Denote the probability distribution function of background clutter in the radar image and the fused image, respectively. Optionally, the Copra density dictionary is constructed according to the four Copra functions in Table 1 (the Copra density is the cross partial derivative of the Copra function).
[0140]
[0141] In this way, the model parameters and functions of the Copra model can be obtained
[0142] Optionally, calculating the joint probability density function of the target radar image and the background clutter in the fused image based on the Copra model includes: generating a plurality of pixel pairs according to each pixel point of the target radar image and the pixel points at corresponding positions in the fused image; and substituting the plurality of pixel pairs into the Copra model to obtain the joint probability density function of the target radar image and the background clutter in the fused image.
[0143] The joint probability density function of the target radar image and the background clutter in the fused image can be calculated by the following formula:
[0144]
[0145] Among them, (z S ,z A ) represents the pixel pairs in the target radar image and the fused image, F S and F A Can be and The joint probability density function represents the common background clutter distribution in the target radar image and the fused image.
[0146] Optionally, calculating the background clutter suppression matrix of the target radar image and the fused image according to the joint probability density function includes: generating a plurality of pixel pairs according to each pixel point of the target radar image and the pixel points at corresponding positions in the fused image; and obtaining the background clutter suppression matrix of the background clutter in the target radar image and the fused image according to the plurality of pixel pairs and the joint probability density function.
[0147] The background clutter suppression matrix of the target radar image and the fused image can be calculated by the following formula:
[0148]
[0149] Where m = 1, 2, ..., M, n = 1, 2, ..., N, we get the matrix
[0150] Optionally,
[0151] Figure 6 This is a flow chart of a heterogeneous remote sensing image fusion method based on homogeneous transformation and target enhancement.
[0152] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0154] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0155] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0158] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0160] The above is a detailed introduction to a method for fusing heterogeneous remote sensing images provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for fusing heterogeneous remote sensing images, characterized in that: The method comprises: Acquire optical and radar images; Inputting the optical image and the radar image into a homogeneous variation image generation model to obtain a fused image, wherein the fused image includes content features of the optical image and style features of the radar image; The homogeneous variation image generation model is a model that learns to generate image samples to be iterated based on optical image samples and radar image samples, and iterates the image samples to be iterated to generate fused image samples; The homogeneous variation image generation model is obtained by training an initial model, wherein the initial model includes an image generation module and a first feature extraction module with fixed weights; the steps of training the initial model at least include: Inputting the optical image samples and the radar image samples into the initial model; Using the first feature extraction module, extracting a first content feature of the optical image sample and a first style feature of the radar image sample; Inputting the first content feature and the first style form feature into the image generation module, and outputting a first intermediate image sample; Constructing a loss function according to the current input and output of the image generation module; Based on the loss function, the model parameters of the initial model are updated to obtain the homogeneous variation image generation model; The initial model also includes a second feature extraction module with random weights; and the step of training the initial model also includes: Acquire the to-be-iterated image sample generated by the image generation module, wherein the to-be-iterated image sample is: an image sample that minimizes the loss function when the input of the image generation module is the first content feature and the first style form feature; Using the second feature extraction module, extracting a second content feature of the optical image sample and a second style feature of the radar image sample; Acquire a second intermediate image sample generated by the image generation module, where the second intermediate image sample is: an image sample that minimizes the loss function when the input of the image generation module is the second content feature and the second style form feature; Iterating the image sample to be iterated according to the second intermediate image sample, comprising: When an iteration termination condition is met, the second intermediate image sample is used as the fused image sample output by the homogeneous variation image generation model.
2. The method according to claim 1, characterized in that The step of using the second intermediate image sample as the fused image sample output by the homogeneous variation image generation model when an iteration termination condition is satisfied includes: Calculating an iterative convergence error between the second intermediate image sample and the to-be-iterated image sample; When the iterative convergence error is less than an iterative convergence error threshold, or when the number of times the second intermediate image samples are generated reaches a preset number, the second intermediate image samples are used as the fused image samples output by the homogeneous variation image generation model.
3. The method according to claim 1, characterized in that: The iterating the image sample to be iterated according to the second intermediate image sample further includes: When the iteration termination condition is not met, the second intermediate image sample is used as the image sample to be iterated, and the process returns to the step of using the second feature extraction module to extract the second content feature of the optical image sample and the second style feature of the radar image sample.
4. The method according to any one of claims 1 to 3, characterized in that: In a case where the current input of the image generation module is the first content feature and the first style feature, constructing a loss function according to the current input and output of the image generation module includes: Using the first feature extraction module, extracting a first content feature and a first style feature of the first intermediate image sample; The loss function is constructed according to the difference between the first content feature of the first intermediate image sample and the first content feature of the optical image sample, and the difference between the first style form feature of the first intermediate image sample and the first style form feature of the radar image sample.
5. The method according to any one of claims 1 to 3, characterized in that: In a case where the current input of the image generation module is the second content feature and the second style feature, constructing a loss function according to the current input and output of the image generation module includes: Using the second feature extraction module, extracting a second content feature and a second style feature of the second intermediate image sample; The loss function is constructed according to the difference between the second content feature of the second intermediate image sample and the second content feature of the optical image sample, and the difference between the second style form feature of the second intermediate image sample and the second style form feature of the radar image sample.
6. The method according to claim 1, characterized in that After obtaining the fused image, the method further includes: Acquire a target radar image, where the target radar image is an image from the same radar as the radar image and observes the same area as the optical image, and the area includes a target proposal area; Inputting the fused image and the target radar image into a classifier to obtain target proposal regions of the fused image and the target radar image respectively; Obtaining a common target proposal region of the fused image and the target radar image according to respective target proposal regions of the fused image and the target radar image; The common target proposal region and the target radar image are fused to obtain an image with enhanced target proposal region.
7. The method according to claim 6, characterized in that The classifier is trained by the following steps: Acquire a plurality of labeled box samples from the radar image samples and the fused image samples, wherein the labels indicate whether the box samples are the target proposal areas; Obtaining a gradient norm map of each of the plurality of box samples; Inputting the gradient norm map of each of the plurality of box samples into an initial classifier to obtain a classification prediction result of whether each of the plurality of box samples is a target proposal area; According to the respective labels and classification prediction results of the plurality of box samples, the parameters of the initial classifier are updated to obtain the classifier.
8. The method according to claim 6, characterized in that The fusing the common target proposal region and the target radar image to obtain an image enhanced by the target proposal region includes: Based on the Copra model, a joint probability density function of background clutter in the target radar image and the fused image is calculated, wherein the joint probability density function represents the distribution of background clutter common to the target radar image and the fused image; Calculating background clutter suppression matrices of the target radar image and the fused image according to the joint probability density function; Acquiring the similarity between the target radar image and the fused image; An image enhanced by the target proposal region is obtained according to the common target proposal region, the target radar image, the background clutter suppression matrix and the similarity.
