A method, apparatus, device, and storage medium for training image super-resolution networks.

Abstract

This application belongs to the field of image processing technology in artificial intelligence, and relates to a training method, apparatus, computer device, and storage medium for an image super-resolution network based on optical image feature fusion. First, the proposed visible light feature guidance module extracts high-frequency and spatial information from a high-resolution visible light image to obtain a visible light image feature map, which is then mapped to a SAR image feature space through pixel multiplication. Next, the SAR image feature space is downsampled and encoded to extract deep features. Then, the proposed channel feature encoding module upsamples and decodes the extracted features to reconstruct image features. The reconstructed image features are compared with the extracted image features to calculate a reconstruction loss, thereby constraining the decoding module to generate more realistic SAR image features. The image super-resolution network improves the super-resolution performance of SAR images by introducing visible light image feature information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing technology in artificial intelligence, and particularly relates to an image super-resolution network training method and device based on optical image feature fusion, a computer device and a storage medium. BACKGROUND

[0002] Deep neural networks have achieved wide application in real life, such as image classification, text conversion and other applications.

[0003] Recently, there are more and more studies on image super-resolution based on deep learning. This method mainly learns a nonlinear mapping relationship from a large number of low-resolution image and high-resolution image pairs, and then reconstructs the corresponding high-resolution image from a low-resolution image according to the mapping relationship.

[0004] However, the applicant finds that the traditional deep learning-based image super-resolution method is proposed for natural images, that is, visible light images. For single-channel SAR images with less feature information, the image super-resolution network proposed by these methods is not applicable. Therefore, the traditional image super-resolution method has the problem of being unable to be applied to single-channel SAR images with less feature information. SUMMARY

[0005] The purpose of the embodiments of the present application is to propose an image super-resolution network training method and device based on optical image feature fusion, a computer device and a storage medium, to solve the problem that the traditional image super-resolution method cannot be applied to single-channel SAR images with less feature information.

[0006] To solve the above technical problems, the embodiments of the present application provide an image super-resolution network training method based on optical image feature fusion, which adopts the following technical solutions:

[0007] Obtain original data, wherein the original data includes a high-resolution SAR image and a high-resolution visible light image;

[0008] Downsample the high-resolution SAR image to obtain a low-resolution SAR image;

[0009] Input the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform a super-resolution conversion operation to obtain a super-resolution converted image;

[0010] Calculate the absolute value of the average pixel difference between the super-resolution converted image and the high-resolution SAR image;

[0011] The absolute value of the average pixel difference is taken as a loss function to perform a parameter tuning operation on the image super-resolution network, to obtain a target image super-resolution network;

[0012] The operation of inputting the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform a super-resolution conversion operation to obtain a super-resolution converted image specifically includes the following steps:

[0013] An image feature extraction operation is performed on the low-resolution SAR image to obtain a shallow SAR image feature;

[0014] A visible light feature fusion operation is performed on the shallow SAR image feature and the high-resolution visible light image according to a visible light feature guidance module to obtain a SAR image feature with enhanced visible light image information;

[0015] A feature encoding operation is performed on the SAR image feature with enhanced visible light image information according to a SAR image feature encoding module to obtain a deep SAR image feature;

[0016] A local feature reconstruction operation is performed on the deep SAR image feature according to a feature decoding module to obtain a local reconstruction feature;

[0017] The local reconstruction feature is input into a recovery convolution layer to perform a channel number recovery operation to obtain the super-resolution converted image.

[0018] Further, the absolute value L of the average pixel difference OFFN is represented as:

[0019]

[0020] wherein, is represented as the low-resolution SAR image; is represented as the high-resolution visible light image; is represented as the high-resolution SAR image; is represented as a reconstructed high-resolution SAR image output by an image super-resolution network mapping function; is represented as the value of the mth reconstruction feature loss; ||·||1 represents an L1 norm, i.e., an average absolute value error.

[0021] Further, the reconstruction feature loss Lrec is represented as: is represented as:

[0022]

[0023] wherein, is represented as a feature map output by the ith SAR image feature encoding module; The reconstructed feature map represented as the i-th reconstruction loss, and the sum of all reconstruction losses is 1. The weight represented as the i-th reconstruction loss, and the sum of all reconstruction losses is 1.

[0024] Further, the step of performing visible light feature fusion operation on the shallow SAR image feature and the high-resolution visible light image according to the visible light feature guiding module to obtain the SAR image feature with enhanced visible light image information includes the following steps:

[0025] Performing up-sampling operation on the low-resolution SAR image to obtain the original high-resolution SAR image;

[0026] Respectively inputting the original high-resolution SAR image and the high-resolution visible light image into the convolution layer to perform shallow feature extraction operation to obtain the SAR image shallow feature and the visible light shallow feature;

[0027] Inputting the visible light shallow feature into the OFGM module to perform visible light feature extraction operation to obtain the visible light feature map;

[0028] Performing pixel multiplication operation on the visible light feature map and the SAR image shallow feature to obtain the multiplied pixel feature;

[0029] Performing feature splicing operation on the multiplied pixel feature and the SAR image shallow feature to obtain the spliced feature;

[0030] Inputting the spliced feature into the restoration convolution layer to perform feature channel number restoration operation to obtain the SAR image feature with enhanced visible light image information.

[0031] To solve the above technical problems, the embodiment of the application further provides an image super-resolution network training device based on optical image feature fusion, which adopts the following technical scheme:

[0032] An original data acquisition module is configured to acquire original data, wherein the original data includes a high-resolution SAR image and a high-resolution visible light image;

[0033] A down-sampling module is configured to perform down-sampling operation on the high-resolution SAR image to obtain a low-resolution SAR image;

[0034] A super-resolution conversion module is configured to input the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform super-resolution conversion operation to obtain a super-resolution converted image;

[0035] an average pixel difference module configured to calculate an absolute value of an average pixel difference of the super-resolution converted image and the high-resolution SAR image;

[0036] a parameter tuning module configured to perform a parameter tuning operation on the image super-resolution network by taking the absolute value of the average pixel difference as a loss function, to obtain a target image super-resolution network;

[0037] The super-resolution conversion module includes an image feature extraction submodule, a visible light feature fusion submodule, a feature encoding submodule, a local feature reconstruction submodule, and a channel number recovery submodule, wherein:

[0038] The image feature extraction submodule is configured to perform an image feature extraction operation on the low-resolution SAR image to obtain a shallow SAR image feature.

[0039] The visible light feature fusion submodule is configured to perform a visible light feature fusion operation on the shallow SAR image feature and the high-resolution visible light image according to a visible light feature guidance module, to obtain a visible light image information enhanced SAR image feature.

[0040] The feature encoding submodule is configured to perform a feature encoding operation on the visible light image information enhanced SAR image feature according to a SAR image feature encoding module, to obtain a deep SAR image feature.

[0041] The local feature reconstruction submodule is configured to perform a local feature reconstruction operation on the deep SAR image feature according to a feature decoding module, to obtain a local reconstruction feature.

[0042] The channel number recovery submodule is configured to input the local reconstruction feature into a recovery convolution layer to perform a channel number recovery operation, to obtain the super-resolution converted image.

[0043] Further, the absolute value L of the average pixel difference is calculated as follows: OFFN is represented as:

[0044]

[0045] wherein, is represented as the low-resolution SAR image; is represented as the high-resolution visible light image; is represented as the high-resolution SAR image; is represented as a reconstructed high-resolution SAR image output by an image super-resolution network mapping function; is represented as a value of the mth reconstruction feature loss; ||·||1 represents an L1 norm, i.e., an average absolute value error.

[0046] Further, the reconstruction feature loss L is calculated as follows: is expressed as:

[0047]

[0048] wherein, is expressed as the feature map output by the i-th SAR image feature encoding module; is expressed as the feature map reconstructed by the log2r-i-th feature decoding module, which corresponds to the feature map of the i-th encoding module, so as to calculate the loss between the reconstructed feature and the original extracted feature; is expressed as the weight of the i-th reconstruction loss, and the sum of all reconstruction losses is 1.

[0049] Further, the visible light feature fusion sub-module comprises:

[0050] an up-sampling unit configured to perform an up-sampling operation on the low-resolution SAR image to obtain an original high-resolution SAR image;

[0051] a shallow feature extraction unit configured to input the original high-resolution SAR image and the high-resolution visible light image into a convolution layer respectively to perform a shallow feature extraction operation to obtain SAR image shallow features and visible light shallow features;

[0052] a visible light feature extraction unit configured to input the visible light shallow features into an OFGM module to perform a visible light feature extraction operation to obtain a visible light feature map;

[0053] a pixel multiplication unit configured to perform a pixel multiplication operation on the visible light feature map and the SAR image shallow features to obtain multiplied pixel features;

[0054] a feature concatenation unit configured to perform a feature concatenation operation on the multiplied pixel features and the SAR image shallow features to obtain concatenated features;

[0055] a feature channel number recovery unit configured to input the concatenated features into the recovery convolution layer to perform a feature channel number recovery operation to obtain visible light image information enhanced SAR image features.

[0056] To solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the technical scheme as follows:

[0057] comprising a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the image super-resolution network training method based on optical image feature fusion as described above.

[0058] To solve the above technical problems, the embodiment of the present application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0059] The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the image super-resolution network training method based on optical image feature fusion as described above.

[0060] The present application provides an image super-resolution network training method based on optical image feature fusion, comprising: obtaining original data, wherein the original data comprises a high-resolution SAR image and a high-resolution visible light image; performing downsampling operation on the high-resolution SAR image to obtain a low-resolution SAR image; inputting the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform super-resolution conversion operation to obtain a super-resolution converted image; calculating the absolute value of the average pixel difference of the super-resolution converted image and the high-resolution SAR image; taking the absolute value of the average pixel difference as a loss function to perform parameter tuning operation on the image super-resolution network to obtain a target image super-resolution network; the operation of inputting the low-resolution SAR image and the high-resolution visible light image into the image super-resolution network to perform super-resolution conversion operation to obtain the super-resolution converted image specifically comprises the following steps: performing image feature extraction operation on the low-resolution SAR image to obtain shallow SAR image features; performing visible light feature fusion operation on the shallow SAR image features and the high-resolution visible light image according to a visible light feature guiding module to obtain SAR image features with enhanced visible light image information; performing feature encoding operation on the SAR image features with enhanced visible light image information according to a SAR image feature encoding module to obtain deep SAR image features; performing local feature reconstruction operation on the deep SAR image features according to a feature decoding module to obtain local reconstruction features; inputting the local reconstruction features into a recovery convolution layer to perform channel number recovery operation to obtain the super-resolution converted image. Compared with the prior art, the present application first extracts high-frequency information and spatial information of the high-resolution visible light image through the proposed visible light feature guiding module to obtain a visible light image feature map, and maps it to the SAR image feature space through pixel multiplication. Then, the SAR image feature space is downsampled and encoded to extract deep features, and then the extracted features are upsampled and decoded through the proposed channel feature encoding module to reconstruct the image features, and the reconstructed image features and the extracted image features are calculated to obtain a reconstruction loss, so as to constrain the decoding module to generate more realistic SAR image features. The image super-resolution network improves the super-resolution performance of the SAR image by introducing visible light image feature information. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the solutions in the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0063] Figure 2 is an implementation flowchart of the image super-resolution network training method based on optical image feature fusion provided by Embodiment One of the present application;

[0064] Figure 3 is a structural schematic diagram of a specific implementation of the image super-resolution network provided by Embodiment One of the present application;

[0065] Figure 4 is a structural schematic diagram of a specific implementation of the visible light feature guiding module provided by Embodiment One of the present application;

[0066] Figure 5 is a structural schematic diagram of the image super-resolution network training device based on optical image feature fusion provided by Embodiment Two of the present application;

[0067] Figure 6 is a structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terminology used in the specification of the present application is only for the purpose of describing specific embodiments and is not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification of the present application and the claims and the above description of drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the specification and claims of the present application and the above description of drawings are used to distinguish different objects, not to describe a particular order.

[0069] Reference herein to "embodiments" means that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0070] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0071] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0072] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0074] The server 105 can be a server providing various services, such as a background server supporting the pages displayed on the terminal devices 101, 102, 103.

[0075] It should be noted that the image super-resolution network training method based on optical image feature fusion provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the image super-resolution network training device based on optical image feature fusion is generally provided in a server / terminal device.

[0076] It should be understood that Figure 1 the number of terminal devices, networks, and servers in is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0077] Embodiment One

[0078] Continuing to refer to Figure 2, and a flowchart of an implementation of the image super-resolution network training method based on optical image feature fusion provided in Embodiment One of the present application is shown. For ease of illustration, only the parts relevant to the present application are shown.

[0079] The image super-resolution network training method based on optical image feature fusion described above comprises the following steps:

[0080] Step S201: Obtain original data, wherein the original data comprises a high-resolution SAR image and a high-resolution visible light image.

[0081] In the embodiments of the present application, the image super-resolution network is trained using a dataset containing a high-resolution SAR image I HR and a high-resolution visible light image I OP under the same scene.

[0082] Step S202: Perform down-sampling operation on the high-resolution SAR image to obtain a low-resolution SAR image.

[0083] In the embodiments of the present application, the high-resolution SAR image in the dataset is obtained by bicubic method to obtain the corresponding low-resolution SAR image I LR .

[0084] Specifically, the bicubic bicubic interpolation function is used to up-sample the input low-resolution SAR image I LR to restore the original image size to obtain The process can be represented as:

[0085]

[0086] wherein f bicubic represents the bicubic bicubic interpolation function.

[0087] Step S203: Input the low-resolution SAR image and the high-resolution visible light image into the image super-resolution network to perform super-resolution conversion operation to obtain a super-resolution converted image.

[0088] In the embodiments of the present application, the SAR image super-resolution work of the present application aims to reconstruct and restore a high-resolution SAR image from a low-resolution SAR image. However, considering that it is difficult to obtain a high-resolution visible light image under the same scene as the SAR image in actual application scenarios, the present application only introduces visible light image information in the training process of the network to enhance the SAR image features.

[0089] In the embodiments of the present application, the overall network structure of the image super-resolution network is as follows: Figure 3The image super-resolution network described herein mainly consists of four parts: a visible light feature fusion module, a SAR image feature encoding part, a feature decoding part, and a reconstruction part.

[0090] In this embodiment of the application, the following is set and (C in C in HH, W W rr represents the number of channels, height, width, and scale factor of the input image, respectively. These are the low-resolution SAR image and the high-resolution visible light image inputs to the image super-resolution network of this application. The reconstructed high-resolution SAR image output by the network.

[0091] In this embodiment, the super-resolution conversion operation is combined with a high-resolution visible light image I from the same scene. OP Training an I LR and I OP To high-resolution SAR image I HR Nonlinear mapping network.

[0092] Step S204: Calculate the absolute value of the average pixel difference between the super-resolution converted image and the high-resolution SAR image.

[0093] In this embodiment of the application, the objective function of the above mapping is to make the reconstructed high-resolution SAR image I SR Compared with real high-resolution SAR images I HR The losses between them are smaller, specifically manifested as follows:

[0094]

[0095] Where Θ represents the parameter set of the trained image super-resolution network, i.e., the weights and bias matrices of each network layer; N represents the number of image pairs in the training set. Indicates the trained I LR and I OP To I HR The image super-resolution network mapping relationship model.

[0096] In some optional implementations of the embodiments of this application, the absolute value L of the average pixel difference OFFN Represented as:

[0097]

[0098] in, Represented as a low-resolution SAR image; Represented as a high-resolution visible light image; Represented as a high-resolution SAR image; This represents the reconstructed high-resolution SAR image output by the image super-resolution network mapping function; represents the value of the loss of the m-th reconstructed feature; ||·|1 represents the L1 norm, i.e., the mean absolute error.

[0099] In this embodiment, mean absolute error is used instead of mean squared error because mean absolute error is less sensitive to abnormal pixels. It avoids impacting the overall network training loss due to abnormal pixel values, thus making the network training more stable. This facilitates the recovery of more detailed SAR image information.

[0100] In some optional implementations of the embodiments of this application, the reconstruction feature loss Represented as:

[0101]

[0102] in, This is represented as the feature map output by the i-th SAR image feature coding module; Let log2r-i be the feature map reconstructed by the feature decoding module, which corresponds to the feature map of the i-th encoding module. Therefore, we calculate the loss between the reconstructed features and the original extracted features. Let represent the weight of the i-th reconstruction loss, and the sum of all reconstruction losses is 1.

[0103] In this embodiment of the application, the reconstruction feature loss This represents a weighted sum of the losses of all reconstructed feature maps and their corresponding extracted feature maps. Using this as part of the overall network loss, it constrains the feature decoding module to generate more realistic image feature information.

[0104] Step S205: Use the absolute value of the average pixel difference as the loss function to perform parameter tuning on the image super-resolution network to obtain the target image super-resolution network.

[0105] Step S203 above specifically includes the following steps:

[0106] Step S2031: Perform image feature extraction on the low-resolution SAR image to obtain shallow SAR image features;

[0107] Step S2032: Based on the visible light feature guidance module, perform visible light feature fusion operation on shallow SAR image features and high-resolution visible light image to obtain SAR image features with enhanced visible light image information.

[0108] In this embodiment, the visible light feature fusion operation can be as follows: Upsampling a low-resolution SAR image to obtain a raw high-resolution SAR image; inputting the raw high-resolution SAR image and the high-resolution visible light image into a convolutional layer for shallow feature extraction to obtain shallow features of the SAR image and shallow features of the visible light; inputting the shallow features of the visible light into an OFGM module for visible light feature extraction to obtain a visible light feature map; performing pixel multiplication on the visible light feature map and the shallow features of the SAR image to obtain multiplied pixel features; performing feature stitching on the multiplied pixel features and the shallow features of the SAR image to obtain stitched features; and inputting the stitched features into a recovery convolutional layer for feature channel number recovery to obtain SAR image features with enhanced visible light image information.

[0109] In the embodiments of this application, a 3×3 convolutional layer F is used respectively. head (·) Extract the upsampled SAR image And the original high-resolution visible light image I OP shallow features F OP ∈R C×rH×rW .

[0110]

[0111] F OP =F head (I OP )

[0112] In this embodiment, convolutional layers can stabilize network optimization and achieve better results. Furthermore, convolutional layers are a simple way to convert the input image matrix into a high-dimensional feature map. Then, the obtained shallow features of visible light are used to obtain the visible light feature weight map M through our proposed visible light feature guidance module. op Then, the feature weight map is mapped to the SAR image feature space through pixel multiplication. It is then concatenated with the initial shallow features of the SAR image, adding visible light feature weights to the original SAR image features. By extracting high-frequency information from visible light, corresponding important weights are provided to the SAR image features, enriching their feature information. Finally, a 3×3 convolutional layer fuses the concatenated channel features to obtain SAR image features with visible light image feature information. The entire visible light feature fusion module can be represented as:

[0113]

[0114] Among them, f fusion This is a 3×3 convolutional layer operation; Concat represents the concatenation operation along the channel dimension; Mop This refers to the visible light feature information extracted by the visible light feature fusion module.

[0115] In this embodiment, considering that different local areas of an optical image have varying guiding effects on SAR image super-resolution, a visible light feature guided module (OFGM) is proposed to fully extract spatial channel features and local high-frequency feature information from the optical image. This module aims to better leverage the guiding role of the visible light image, fully extract the rich high-frequency feature texture information in the visible light image, and thus improve the quality of the output high-resolution SAR image. Its network structure diagram is shown below. Figure 4 As shown.

[0116] In this embodiment of the application, the shallow features F of the visible light image are... OP First, a downsampling operation is performed using a convolutional layer with a stride of 2, simultaneously increasing the number of visible light image feature channels. Then, an AvgPooling operation smooths the image features. Compared to Maxpooling, which only considers the most important pixels within a single window, our AvgPooling avoids the loss of original texture features in the SAR image. This process can be represented as:

[0117] F` op =AvgPooling(Conv stride＝2 (F OP ))

[0118] Where AvgPooling represents the average pooling function; Conv stride＝2 This indicates a convolutional layer with a stride of 2. This process primarily aims to avoid the loss of high-frequency information in the feature map during downsampling. Then, F`... op The feature map extracts deep features through residual dense blocks. Here, we set the number of residual dense blocks to three. This module combines image features of different scales through a multi-scale residual network and dense connections. Then, the deep features are added to the original image features pixel by pixel to obtain multi-channel feature-weighted visible light image features. These features are then restored to their original feature size and added to the initial shallow feature pixels, effectively adding the extracted deep feature map to the initial shallow features, thus enhancing the original visible light image feature information. This process can be represented as:

[0119] F`` op =F OP +Conv 3×3 (Up(δ)F` op +α*RDB i (F` t)))),i=1,…,3

[0120] Among them, Conv 3×3 Let represent a 3×3 convolutional layer; Up represents the upsampling operation; δ represents the ReLU nonlinear activation function; and RDB represents the mapping function of the residual dense block. Then, a 1×1 convolutional layer is used to reduce the channel dimension and network parameters of the extracted visible light image features, and this is then passed through a Sigmoid activation function to obtain the weight map M of the visible light image to the super-resolution of the entire SAR image. op 。 is represented as:

[0121] M op =Sigmoid(Conv 1×1 (F`` op ))

[0122] Here, Sigmoid represents the Sigmoid activation function, whose output ranges between (0,1). Conv 1×1 This represents a 1×1 convolutional layer, whose purpose is to reduce network parameters. Then M... op The visible light image feature weight map is mapped to the SAR image feature space by pixel-wise multiplication with the shallow features of the SAR image. Different weights are applied to different local locations in the visible light image feature map to enhance the SAR image features to varying degrees. Then, the map is concatenated with the shallow features of the original SAR image along the channel dimension, adding SAR image features weighted by visible light characteristics to the original SAR image features. In simpler terms, important local features of the SAR image are first determined based on the visible light image, and then these local features are added to the original feature map to increase feature information.

[0123] This process can be represented as:

[0124]

[0125] Among them, f fusion This represents a 3×3 convolutional layer, whose function is to restore the image features after channel concatenation to the original number of feature channels; Concat represents the channel-dimensional concatenation operation. The final result is F`. sar It contains SAR image feature information that includes privileged weighting of visible light images.

[0126] Step S2033: Perform feature encoding operation on the SAR image features enhanced by visible light image information according to the SAR image feature encoding module to obtain deep SAR image features.

[0127] In this embodiment, to fully utilize the relationships between pixels in different channels of a visible light feature image to generate high-frequency texture information, we propose a Channel Feature Code Module (CFCM) using Channel Attention. For multi-channel feature maps, this module can efficiently capture the relationships between feature pixels in different channels. Its network structure diagram is shown below. Figure 3 As shown in the bottom right corner.

[0128] First, take the input feature map F CFCM First, a shallow feature is extracted using a 3×3 convolutional layer and a ReLU non-linear activation function, as follows:

[0129] F` CFCm =δ(Conv 3×3 (F CFCM ))

[0130] Among them, Conv 3×3 δ represents a 3×3 convolutional layer; δ represents the ReLU non-linear activation function. Then, a channel attention module is formed by combining one channel attention with a 3×3 convolutional layer. The shallow features above are processed through several channel attention modules to extract the feature weights of each channel in the feature map, obtaining the spatial information of the feature map. The output of each channel attention is then weighted by a learnable weight and summed with the initial feature map pixels to form a residual connection, increasing the spatial feature information of the feature map. Here, we set the number of channel attentions to 2. This process can be represented as:

[0131]

[0132] Among them, CALayer i This represents the attention module for the i-th channel; λ represents the output of the attention module in the (i-1)th channel; i This represents the weight of the output of the attention module in the i-th channel. This formula means summing the weighted pixels of the attention output of each channel with the initial feature map.

[0133] In this embodiment, after obtaining the SAR image feature map, the application further downsamples the feature map through encoding to extract deeper features. Each encoding module includes a convolutional layer with a stride of 2, a ReLU activation function, and a 3×3 convolutional layer. The use of a convolutional layer with a stride of 2 aims to perform convolutional downsampling on the feature map, increasing the number of channels in the feature map to obtain local features of the SAR image. The application varies the number of encoding modules according to different scale factors to extract deeper and more complete image features even at higher magnifications. Specifically, the number of encoding modules is log₂r of the scale factor r. For example, when the scale factor is 4, there are 2 encoding modules and 2 decoding modules; when the scale factor is 8, there are 3 encoding and 3 decoding modules. The SAR image feature map, after feature encoding, yields deeper features. This feature encoding module can be represented as:

[0134] Encoder = [Conv stride＝2 ,ReLu,Conv 3×3 ]

[0135]

[0136] Among them, Conv stride＝2 Represents a convolution with a stride of 2; ReLU represents the ReLU activation function; Conv 3×3 This represents a 3×3 convolutional layer. (Encoder) i Let represent the i-th feature encoding module. r represents the scaling factor.

[0137] Step S2034: Perform local feature reconstruction on the deep SAR image features according to the feature decoding module to obtain the local reconstructed features.

[0138] In this embodiment, after obtaining SAR image feature maps of different dimensions, feature decoding, i.e., feature upsampling, is performed on the deep features. In the feature decoding module, this application introduces a Channel Feature Coding (CFCM) module. This module is used to extract global feature information for each channel from the extracted SAR image feature maps, preserving the original channel feature information. The number of CFCMs in this decoding module is customizable; in this application, it is set to 20. After extracting channel features, an upsampling operation is performed to reconstruct part of the original image features. The overall decoding part can be represented as follows:

[0139] Decoder = [CFCM] m [Upsample], m = 1, ..., 20

[0140]

[0141]

[0142] in, This represents the feature map output after passing through the i-th feature decoding module; Concat represents the concatenation operation along the channel dimension. Here, the input to the first decoding module is the final feature map output by the feature encoding module. Each subsequent feature decoding module receives its input as the output of the previous module and the feature output of the corresponding feature encoding part. This application employs a skip connection approach to globally transmit the feature maps from the feature encoding modules, facilitating rapid transmission of image feature information within the network. For the output of each feature decoding module, this application also calculates a loss between it and the output of the corresponding channel encoding module, which is included as part of the network loss. Furthermore, different feature decoding losses are assigned different weights, with deeper features receiving larger weights. The aim is that deeper features possess richer local high-frequency information, and this application uses larger weights to constrain the corresponding feature decoding modules to reconstruct more realistic SAR image features. This feature reconstruction loss can be specifically expressed as:

[0143]

[0144] in This represents the normalization processing of the weights corresponding to each feature decoding module. For example, when the magnification is 4, the weight of the feature reconstruction loss of the first feature decoding module is 4 / (2+4), and the weight of the reconstruction loss of the second feature decoding module is 2 / (2+4).

[0145] Step S2035: Input the local reconstruction features into the recovery convolutional layer to perform channel number recovery operation to obtain the super-resolution transformed image.

[0146] In this embodiment, the final output of the feature decoding module is passed through a 3×3 convolutional layer to restore the original number of image channels, completing the image reconstruction operation, thus obtaining the final output I of the entire network. SR .

[0147]

[0148] Among them, Conv 3×3 This represents a 3×3 convolutional layer that restores the reconstructed image features to the original number of image channels, thus completing super-resolution reconstruction.

[0149] This application provides a method for training an image super-resolution network based on optical image feature fusion, comprising: acquiring raw data, wherein the raw data includes a high-resolution SAR image and a high-resolution visible light image; performing a downsampling operation on the high-resolution SAR image to obtain a low-resolution SAR image; inputting the low-resolution SAR image and the high-resolution visible light image into the image super-resolution network for super-resolution transformation to obtain a super-resolution transformed image; calculating the absolute value of the average pixel difference between the super-resolution transformed image and the high-resolution SAR image; using the absolute value of the average pixel difference as a loss function to perform parameter tuning on the image super-resolution network to obtain a target image super-resolution network; and then using the low-resolution SAR image as a loss function to train the image super-resolution network. The process of inputting an R-image and a high-resolution visible light image into an image super-resolution network for super-resolution transformation to obtain a super-resolution transformed image includes the following steps: First, a visible light feature fusion operation is performed on the shallow SAR image features and the high-resolution visible light image using a visible light feature guidance module to obtain SAR image features enhanced with visible light image information. Second, a feature encoding operation is performed on the SAR image features enhanced with visible light image information using a SAR image feature encoding module to obtain deep SAR image features. Third, a local feature reconstruction operation is performed on the deep SAR image features using a feature decoding module to obtain local reconstructed features. Finally, the local reconstructed features are input into a recovery convolutional layer for channel number recovery to obtain the super-resolution transformed image. Compared with existing technologies, this application first extracts high-frequency and spatial information from the high-resolution visible light image using the proposed visible light feature guidance module to obtain a visible light image feature map, and then maps it to the SAR image feature space through pixel multiplication. Next, downsampling encoding is performed on the SAR image feature space to extract deep features. Then, the extracted features are upsampled and decoded using the proposed channel feature encoding module to reconstruct image features. The reconstruction loss is calculated between the reconstructed image features and the extracted image features, thereby constraining the decoding module to generate more realistic SAR image features. Image super-resolution networks improve the super-resolution performance of SAR images by incorporating visible light image feature information.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed, they can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0151] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0152] Example 2

[0153] Further reference Figure 5 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an image super-resolution network training device based on optical image feature fusion. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0154] like Figure 5 As shown, the image super-resolution network training device 200 based on optical image feature fusion in this embodiment includes: a raw data acquisition module 201, a downsampling module 202, a super-resolution conversion module 203, an average pixel difference module 204, and a parameter tuning module 205. Wherein:

[0155] The raw data acquisition module 201 is used to acquire raw data, which includes high-resolution SAR images and high-resolution visible light images.

[0156] The downsampling module 202 is used to perform downsampling operations on high-resolution SAR images to obtain low-resolution SAR images.

[0157] The super-resolution conversion module 203 is used to input low-resolution SAR images and high-resolution visible light images into the image super-resolution network to perform super-resolution conversion operations and obtain super-resolution converted images.

[0158] The average pixel difference module 204 is used to calculate the absolute value of the average pixel difference between the super-resolution converted image and the high-resolution SAR image.

[0159] The parameter tuning module 205 is used to perform parameter tuning on the image super-resolution network by using the absolute value of the average pixel difference as the loss function, so as to obtain the target image super-resolution network.

[0160] The super-resolution conversion module includes: an image feature extraction submodule 2031, a visible light feature fusion submodule 2032, a feature encoding submodule 2033, a local feature reconstruction submodule 2034, and a channel number recovery submodule 2035, wherein:

[0161] The image feature extraction submodule 2031 is used to perform image feature extraction operations on low-resolution SAR images to obtain shallow SAR image features;

[0162] The visible light feature fusion submodule 2032 is used to perform visible light feature fusion operation on shallow SAR image features and high-resolution visible light images according to the visible light feature guidance module, so as to obtain SAR image features with enhanced visible light image information.

[0163] The feature coding submodule 2033 is used to perform feature coding operations on the SAR image features enhanced by the visible light image information based on the SAR image feature coding module to obtain deep SAR image features;

[0164] The local feature reconstruction submodule 2034 is used to perform local feature reconstruction operations on the deep SAR image features based on the feature decoding module to obtain local reconstructed features.

[0165] The channel number restoration submodule 2035 is used to input local reconstructed features into the restoration convolutional layer to perform channel number restoration operations and obtain a super-resolution transformed image.

[0166] In this embodiment, an image super-resolution network training device 200 based on optical image feature fusion is provided, comprising: a raw data acquisition module 201 for acquiring raw data, wherein the raw data includes a high-resolution SAR image and a high-resolution visible light image; a downsampling module 202 for performing a downsampling operation on the high-resolution SAR image to obtain a low-resolution SAR image; a super-resolution conversion module 203 for inputting the low-resolution SAR image and the high-resolution visible light image into the image super-resolution network for super-resolution conversion to obtain a super-resolution converted image; an average pixel difference module 204 for calculating the absolute value of the average pixel difference between the super-resolution converted image and the high-resolution SAR image; and a parameter tuning module 205 for using the absolute value of the average pixel difference as a loss function to perform parameter tuning on the image super-resolution network to obtain a target image super-resolution network; the super-resolution conversion module includes: an image feature extraction submodule 2031 and a visible light feature fusion submodule 203. 2. Feature encoding submodule 2033, local feature reconstruction submodule 2034, and channel number recovery submodule 2035, wherein: Image feature extraction submodule 2031 is used to perform image feature extraction operations on low-resolution SAR images to obtain shallow SAR image features; Visible light feature fusion submodule 2031 is used to perform visible light feature fusion operations on shallow SAR image features and high-resolution visible light images according to the visible light feature guidance module to obtain visible light image information enhanced SAR image features; Feature encoding submodule 2032 is used to perform feature encoding operations on visible light image information enhanced SAR image features according to the SAR image feature encoding module to obtain deep SAR image features; Local feature reconstruction submodule 2033 is used to perform local feature reconstruction operations on deep SAR image features according to the feature decoding module to obtain local reconstructed features; Channel number recovery submodule 2035 is used to input the local reconstructed features into the recovery convolutional layer to perform channel number recovery operations to obtain the super-resolution converted image. Compared with existing technologies, this application first extracts high-frequency and spatial information from high-resolution visible light images using a proposed visible light feature-guided module to obtain a visible light image feature map, and then maps it to the SAR image feature space through pixel multiplication. Next, it downsamples and encodes this SAR image feature space to extract deep features. Then, it upsamples and decodes the extracted features using a proposed channel feature encoding module to reconstruct image features. Finally, it calculates the reconstruction loss by comparing the reconstructed image features with the extracted image features, thereby constraining the decoding module to generate more realistic SAR image features. The image super-resolution network improves the super-resolution performance of SAR images by introducing visible light image feature information.

[0167] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 ,Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only the computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0169] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0170] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may also include both internal storage units and external storage devices of the computer device 300. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for image super-resolution network training methods based on optical image feature fusion. Furthermore, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0171] In some embodiments, the processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 320 is typically used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions stored in the memory 310 or to process data, for example, to execute computer-readable instructions for the image super-resolution network training method based on optical image feature fusion.

[0172] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0173] The computer device provided in this application first extracts high-frequency and spatial information from a high-resolution visible light image using the proposed visible light feature guidance module to obtain a visible light image feature map, and then maps it to the SAR image feature space through pixel multiplication. Next, it performs downsampling encoding on this SAR image feature space to extract deep features. Then, it uses the proposed channel feature encoding module to upsampling and decoding the extracted features to reconstruct the image features. Finally, it calculates the reconstruction loss by comparing the reconstructed image features with the extracted image features, thereby constraining the decoding module to generate more realistic SAR image features. The image super-resolution network improves the super-resolution performance of SAR images by introducing visible light image feature information.

[0174] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the image super-resolution network training method based on optical image feature fusion as described above.

[0175] The computer-readable storage medium provided in this application first extracts high-frequency and spatial information from a high-resolution visible light image using a proposed visible light feature-guided module to obtain a visible light image feature map, and then maps it to a SAR image feature space through pixel multiplication. Next, the SAR image feature space is downsampled and encoded to extract deep features. Then, the extracted features are upsampled and decoded using a proposed channel feature encoding module to reconstruct image features. The reconstructed image features are compared with the extracted image features to calculate a reconstruction loss, thereby constraining the decoding module to generate more realistic SAR image features. The image super-resolution network improves the super-resolution performance of SAR images by introducing visible light image feature information.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An image super-resolution network training method based on optical image feature fusion, characterized in that, The method comprises the following steps: obtaining original data, wherein the original data comprises a high-resolution SAR image and a high-resolution visible light image; performing a downsampling operation on the high-resolution SAR image to obtain a low-resolution SAR image; inputting the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform a super-resolution conversion operation to obtain a super-resolution converted image; calculating an absolute value of an average pixel difference between the super-resolution converted image and the high-resolution SAR image; performing parameter tuning on the image super-resolution network by taking the absolute value of the average pixel difference as a loss function to obtain a target image super-resolution network; the operation of inputting the low-resolution SAR image and the high-resolution visible light image into the image super-resolution network to perform the super-resolution conversion operation to obtain the super-resolution converted image specifically comprises the following steps: performing image feature extraction on the low-resolution SAR image to obtain a shallow SAR image feature; performing visible light feature fusion on the shallow SAR image feature and the high-resolution visible light image to obtain a SAR image feature with enhanced visible light image information; specifically comprising the following steps: performing an upsampling operation on the low-resolution SAR image to obtain an original high-resolution SAR image; inputting the original high-resolution SAR image and the high-resolution visible light image into a convolution layer respectively to perform shallow feature extraction to obtain a SAR image shallow feature and a visible light shallow feature; inputting the visible light shallow feature into a visible light feature guiding module to perform visible light feature extraction to obtain a visible light feature map; performing pixel multiplication on the visible light feature map and the SAR image shallow feature to obtain a multiplied pixel feature; performing feature splicing on the multiplied pixel feature and the SAR image shallow feature to obtain spliced features; inputting the spliced features into the restoration convolution layer to perform feature channel number restoration to obtain a SAR image feature with enhanced visible light image information; performing feature encoding on the SAR image feature with enhanced visible light image information according to a SAR image feature encoding module to obtain a deep SAR image feature; performing local feature reconstruction on the deep SAR image feature according to a feature decoding module to obtain a local reconstructed feature; inputting the local reconstructed feature into the restoration convolution layer to perform channel number restoration to obtain the super-resolution converted image.

2. The image super-resolution network training method based on optical image feature fusion according to claim 1, characterized in that, the absolute value of the average pixel difference is represented as: wherein, denotes the low resolution SAR image; denotes the high resolution visible light image; denotes the high resolution SAR image; denotes the reconstructed high resolution SAR image output by the image super-resolution network mapping function; denotes the value of the m-th reconstruction feature loss; denotes the L1 norm, i.e. the mean absolute error.

3. The image super-resolution network training method based on optical image feature fusion according to claim 2, characterized in that, the reconstruction feature loss is represented as: wherein, denotes the feature map output by the i-th SAR image feature encoding module; denotes the feature map reconstructed by the i-th feature decoding module, which corresponds to the feature map of the i-th encoding module, so as to calculate the loss between the reconstructed feature and the original extracted feature; denotes the feature map reconstructed by the i-th feature decoding module, which corresponds to the feature map of the i-th encoding module, so as to calculate the loss between the reconstructed feature and the original extracted feature; denotes the weight of the i-th reconstruction loss, and the sum of all reconstruction losses is 1.

4. An image super-resolution network training device based on optical image feature fusion, characterized in that, The device comprises: an original data acquisition module configured to acquire original data, wherein the original data comprises a high-resolution SAR image and a high-resolution visible light image; a downsampling module configured to perform a downsampling operation on the high-resolution SAR image to obtain a low-resolution SAR image; a super-resolution conversion module configured to input the low-resolution SAR image and the high-resolution visible light image into an image super-resolution network to perform a super-resolution conversion operation to obtain a super-resolution converted image; An average pixel difference module is configured to calculate absolute values of average pixel differences of the super-resolution converted image and the high-resolution SAR image; A parameter tuning module is configured to perform parameter tuning operation on the image super-resolution network by taking the absolute values of the average pixel differences as a loss function, to obtain a target image super-resolution network. The super-resolution conversion module comprises an image feature extraction submodule, a visible light feature fusion submodule, a feature encoding submodule, a local feature reconstruction submodule, and a channel number recovery submodule, wherein: The image feature extraction submodule is configured to perform image feature extraction operation on the low-resolution SAR image, to obtain shallow SAR image features. The visible light feature fusion submodule is configured to perform visible light feature fusion operation on the shallow SAR image features and the high-resolution visible light image, to obtain visible light image information enhanced SAR image features. The feature encoding submodule is configured to perform feature encoding operation on the visible light image information enhanced SAR image features according to a SAR image feature encoding module, to obtain deep SAR image features. The local feature reconstruction submodule is configured to perform local feature reconstruction operation on the deep SAR image features according to a feature decoding module, to obtain local reconstruction features. The channel number recovery submodule is configured to input the local reconstruction features into a recovery convolution layer to perform channel number recovery operation, to obtain the super-resolution converted image.

5. The image super-resolution network training device based on optical image feature fusion according to claim 4, characterized in that, the absolute value of the average pixel difference is represented as: wherein, denotes the low resolution SAR image; denotes the high resolution visible light image; denotes the high resolution SAR image; denotes the reconstructed high resolution SAR image output by the image super-resolution network mapping function; denotes the value of the mth reconstructed feature loss; denotes the L1 norm, i.e. the mean absolute error.

6. The image super-resolution network training device based on optical image feature fusion according to claim 4, characterized in that, the reconstruction feature loss is represented as: wherein, denotes the feature map output by the i-th SAR image feature encoding module; denotes the feature map reconstructed by the i-th feature decoding module, which corresponds to the feature map of the i-th encoding module, so as to calculate the loss between the reconstructed feature and the original extracted feature; denotes the feature map reconstructed by the i-th feature decoding module, which corresponds to the feature map of the i-th encoding module, so as to calculate the loss between the reconstructed feature and the original extracted feature; denotes the weight of the i-th reconstruction loss, and the sum of all reconstruction losses is 1.

7. A computer device, comprising: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the image super-resolution network training method based on optical image feature fusion according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the image super-resolution network training method based on optical image feature fusion according to any one of claims 1 to 3.

Images