Synthetic aperture optical image restoration method based on local-global feature enhancement

Abstract

The invention discloses a synthetic aperture optical image restoration method based on local-global feature enhancement, and the method comprises the steps: obtaining a synthetic aperture optical image data set, carrying out the simulation degradation, obtaining a simulation degradation data set, and carrying out the division to obtain a training set, a verification set and a test set; constructing a synthetic aperture optical image restoration network; inputting the training and verification set into a synthetic aperture optical image restoration network for training, calculating a loss function and performing back propagation, updating network parameters, and obtaining a trained optimal parameter model; and inputting the test set into the trained optimal parameter model to obtain a synthetic aperture optical restoration image. According to the method, global information attention is performed on an extracted feature sequence by using an LGA-Transform layer, the sequence length is reduced, advanced semantic information is extracted, local information of features is focused by using a GRM-Conversion layer and an ASFE layer, and the defect of local information capture capability of Transform is made up.

Description

Technical Field

[0001] The invention relates to a synthetic aperture optical image restoration method based on local-global feature enhancement, and belongs to the technical field of synthetic aperture optical system image restoration. Background Art

[0002] With the advancement of science and technology, human beings have increasingly higher requirements for the resolution of optical imaging systems such as atmospheric observation and environmental monitoring. According to the Rayleigh criterion, when the working wavelength of the optical system is constant, the system aperture is positively correlated with its resolution, but due to the limitations of processing technology, cost and other factors, it is difficult to increase the aperture indefinitely. Therefore, people have proposed the Optical Synthetic Aperture Imaging (OSAI) technology, which uses multiple sub-apertures to form an array synthetic large aperture system to achieve high-resolution imaging. Since several sub-apertures are filled together into a single large aperture, the light-passing area of ​​the entire system is reduced, and translation errors are inevitable between apertures, causing the point spread function (PSF) of the system to change, and the low and medium frequencies of the modulation transfer function (MTF) to decrease, resulting in degradation and blurring of the imaging results, and even in severe cases, imaging is impossible.

[0003] In order to solve the problem of imaging blur in synthetic aperture optical systems, traditional methods use image restoration techniques such as Wiener filtering, blind deconvolution method and maximum likelihood method to compensate for OSAI. The development of deep learning has greatly promoted the improvement of image restoration technology. For traditional blurred images, Tao et al. (X. Tao, H. Gao, J. Wang, et al., "Scale-Recurrent Network for Deep Image Deblurring," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 8174–8182 (2018)), Zamir et al. (SW Zamir, A. Arora, L. Khan, et al., "Multi-Stage Progressive Image Restoration," 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14816–14826 (2021)), Mou et al. (C. Mou, Q. Wang, and J. Zhang, "Deep Generalized Unfolding Networks for Image Restoration," 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 17378–17389 (2022) proposed the use of a multi-scale network, which is no longer limited by the blur kernel, and cross-fused features at different stages, optimizing the transmission of information between different scales. Hui et al. (M. Hui, Y. Wu, Y. Li, et al., “Image restoration for synthetic aperture systems with a non-blind deconvolution algorithm via a deep convolutional neural network,” Opt. Express 28(7), 9929–9943 (2020)) further combined the variational physical model and prior-based methods with deep learning, achieving better results, but still inseparable from the design of traditional methods. Unlike motion blur and jitter blur in traditional images, the imaging blur problem of synthetic aperture optical systems presents a specific situation described by the point spread function, and the blur shape is not uniformly distributed.In addition, due to the influence of the sub-aperture phase arrangement, the imaging of the synthetic aperture optical system is prone to ringing phenomenon. When using traditional methods for restoration, it is easily affected by the ringing phenomenon and produces artifacts, resulting in poor restoration effect. Therefore, how to effectively focus on the real structural information of the high-resolution image and restore it has become a key issue in the restoration of high-resolution images using synthetic aperture optics. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a synthetic aperture optical image restoration method based on local-global feature enhancement, which better focuses on the real structural information of the high-resolution image and significantly improves the synthetic aperture high-resolution optical image restoration effect.

[0005] The present invention adopts the following technical solutions to solve the above technical problems: The synthetic aperture optical image restoration method based on local-global feature enhancement includes the following steps: Step 1, obtaining a synthetic aperture optical image data set and performing simulated degradation to obtain a simulated degraded image data set, wherein the synthetic aperture optical image data set corresponds one-to-one with images in the simulated degraded image data set, and the corresponding image data set is randomly divided into a training set, a validation set, and a test set; Step 2, constructing a synthetic aperture optical image restoration network, including an encoding network and a decoding network, the encoding network includes a first module to a fifth module connected in sequence, an input mapping module is connected between the first module and the original input image, the decoding network includes a sixth module to a tenth module connected in sequence, the input end of the sixth module is connected to the output end of the fifth module, the output end of the tenth module is connected to the second gated residual convolution layer, and the output of the gated residual convolution layer is used as the final output of the synthetic aperture optical image restoration network; The structures of the first module to the tenth module are the same, and all include a linear gated Transformer layer, a first gated residual convolution layer and an adaptive scale feature enhancement module connected in sequence, the output of the input mapping module is jump-connected to the output of the second gated residual convolution layer, the output of the adaptive scale feature enhancement module in the first module is jump-connected to the output of the adaptive scale feature enhancement module in the tenth module, the output of the adaptive scale feature enhancement module in the second module is jump-connected to the output of the adaptive scale feature enhancement module in the ninth module, the output of the adaptive scale feature enhancement module in the third module is jump-connected to the output of the adaptive scale feature enhancement module in the eighth module, the output of the adaptive scale feature enhancement module in the fourth module is jump-connected to the output of the adaptive scale feature enhancement module in the seventh module, and the output of the adaptive scale feature enhancement module in the fifth module is jump-connected to the output of the adaptive scale feature enhancement module in the sixth module; The input mapping module maps the original input image to the original input feature sequence. The first to fifth modules respectively output feature sequences with lengths of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original input feature sequence. The sixth to tenth modules respectively output feature sequences with lengths of 1 / 32, 1 / 16, 1 / 8, 1 / 4 and 1 / 2 of the original input feature sequence. Step 3, input the training set and validation set obtained in step 1 into the synthetic aperture optical image restoration network constructed in step 2 for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the trained optimal parameter model; Step 4: Input the test set obtained in step 1 into the optimal parameter model trained in step 3, and output the restored image of the synthetic aperture optical system.

[0006] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: 1. The synthetic aperture optical image restoration network based on local-global feature enhancement proposed in the present invention consists of an encoding network and a decoding network, wherein the encoding and decoding network adopts the LAG-Transformer layer to replace the downsampling layer in the traditional U-Net network, and constructs the GRM-Convolution layer and ASFE layer to enhance the network's ability to pay attention to global information and local information.

[0007] 2. The synthetic aperture optical image restoration network based on local-global feature enhancement proposed in the present invention utilizes the LAG-Transformer layer to pay attention to the global information of the extracted feature sequence, while reducing the sequence length and extracting high-level semantic information. At the same time, it utilizes the GRM-Convolution layer and the ASFE layer to pay attention to the local information of the features, thereby making up for the lack of local information capture capability in the Transformer. After acquiring the high-level semantic information, the final restored image is obtained through the decoding area.

[0008] 3. The residual convolution layer based on the gating mechanism proposed in this invention uses The depth-separable convolution performs common extraction operations on feature information and uses The deep separable convolution controls the number of feature channels, and uses a simple gating mechanism to divide the features into two parts in the channel dimension, and then multiply them pixel by pixel, which can effectively replace the GELU activation function, help reduce the occurrence of gradient explosion, and has low computational complexity. At the same time, the deformable convolution input feature is used for deformation extraction operation. The deformable convolution introduces a learnable offset in the receptive field. Compared with traditional convolution, it deforms the convolution kernel of a fixed size, and the position of each convolution kernel is adaptively adjusted according to the learnable offset, which realizes more flexible extraction of local features, so that GRM-Convolution can better adapt to local changes in input features, improve the generalization ability of GRM-Convolution, and enable it to capture local information of different data features while also adapting to different data distributions. After two deformable convolution extractions, the residual connection is performed with the features after the gating mechanism, and then a simple gating operation is performed on it to improve the network's ability to express features. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of the synthetic aperture optical image restoration method based on local-global feature enhancement of the present invention; Figure 2 It is a structural diagram of a synthetic aperture optical image restoration network model constructed by the present invention; Figure 3 It is a structural diagram of the linear gated Transformer layer (LAG-Transformer) proposed in the present invention; Figure 4 It is a structural diagram of the gated residual convolution layer (GRM-Convolution) proposed in the present invention; Figure 5 is a structural diagram of the adaptive scale feature enhancement module proposed in the present invention; Figure 6 1 is a schematic diagram comparing the image restoration results of the synthetic aperture optical system by the method proposed in the present invention and other methods, wherein (a) is the original image, (b) is the degraded image, (c) is WF, (d) is DeblurGan, (e) is Uformer-T, (f) is Uformer-S, (g) is Stripformer, and (h) is the present invention; Figure 7 : It is a schematic diagram of the comparison of degraded images under the influence of different ringing gain factors and the restoration results of different methods, where (a) is the degraded image under the influence of different ringing gain factors, and (b) is the comparison of the restoration results of different methods under 40dB-50dB noise and the comparison of local magnification images; Figure 8: is a schematic diagram of the comparison of degraded images under the influence of different degrees of noise and the restoration results of different methods, where (a) is the degraded image under the influence of different degrees of noise, and (b) is the comparison of the restoration results of different methods; Fig. 9 The following are the detection comparisons of the YOLOv7 method on the unrestored and restored images, where (a) is the detection of a car and (b) is the detection of a ship. DETAILED DESCRIPTION

[0010] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0011] In view of the unique blur type and unique ringing phenomenon of high-resolution degraded images of synthetic aperture optical systems, traditional deep learning image restoration is not well applicable. The Transformer synthetic aperture optical system high-resolution image restoration method based on local-global feature enhancement proposed in this invention is Figure 1 As shown, the specific steps are as follows: Step 1: Obtain the DOTA remote sensing data set, segment it and select 7,000 images with obvious features. Use the point spread function of the system actually measured by the synthetic aperture optical system with an eight-aperture annular array structure to simulate the data set degradation experiment, obtain the degraded images based on the DOTA remote sensing data set, and divide them into training set, validation set and test set according to the set ratio.

[0012] Among them, the DOTA remote sensing dataset contains 2806 ultra-high-resolution remote sensing images with the highest pixel reaching , which contains 15 categories of common targets. After segmentation, 7,000 images with obvious features are selected.

[0013] The specific process of simulated degradation of the data set is as follows: measuring the actual synthetic aperture optical system parameters, obtaining the system point spread function, saving it in txt format, and then using pycharm software to read the point spread function and convolve it with the DOTA remote sensing data set to obtain a simulated degraded blurred image.

[0014] The data set preprocessing is as follows: the DOTA remote sensing data set and simulated degraded images are cropped into The images of different sizes are then mapped one by one to the input image and the target image.

[0015] The specific division of the data set is as follows: the corresponding images are randomly divided into three parts, of which the training set accounts for 80%, the validation set accounts for 10%, and the test set accounts for 10%.

[0016] Step 2: If Figure 2As shown in the figure, a synthetic aperture optical system high-resolution image restoration network (Transformer Networks with Local-Global Feature Augmentation, LGFAformer) based on local-global feature enhancement Transformer is built, including a linear gated Transformer layer (Transformer Based on Linear Attention and Gating Mechanism, LAG-Transformer), a gated residual convolution layer (Convolutional Layer Based on Gating and Residual Mechanism, GRM-Convolution) and an adaptive scale feature enhancement module (Adaptive Scale Feature Enhancement Module, ASFE).

[0017] Step 2.1: Input the blurred degraded image into the LGFAformer restoration network. The input image is first mapped into a feature sequence by the convolution layer, and then input into the LAG-Transformer layer to extract the image from the bottom-level detail features to the high-level features, and obtain feature images of different scales. Specifically, the input degraded image is first mapped into a feature sequence, and then input into the LAG-Transformer layer. After the feature sequence is normalized by the layer, it is divided into windows and input into the focused linear attention module (FLA). After the focused linear attention is calculated, the layer is normalized again, and then it is input into the forward propagation module FPBG based on the gating mechanism. The extracted global features are extracted again using the depth-separable convolution. Then, in order to reduce the interference of the windowing operation on the extraction of global information, the divided windows are shifted and reassembled into a new window, which is input into the FLA and FPBG modules again to extract the global information again. Finally, the residual structure is used to add and output a feature sequence of 1 / 4 of the original length.

[0018] Step 2.2: Input the feature sequence of 1 / 4 of the original length obtained in 2.1 into the GRM-Convolution layer. After converting the feature sequence into a feature map, first use deformable convolution to extract the feature information with spatial position offset and then perform layer normalization. Then perform deformable convolution again to further enhance the extraction of spatial position information. The depth of the separable convolution controls the number of channels, and then uses The depth-separable convolution is used to extract its local features and then used The depth-wise separable convolution adjusts the number of channels and adds the feature information extracted by the deformable convolution. The depth-wise separable convolution and simple gating mechanism are used to fuse the two into the final output while ensuring the number of channels.

[0019] Step 2.3: Input the feature map output from step 2.2 into the ASFE layer, where deformable convolution is used to obtain the feature weight map , and the other way uses depth-wise separable convolution to obtain the feature weight map After concatenating the feature information extracted from the two paths, the number of channels is readjusted and the feature weight map is obtained by using the average pooling operation. , and then three different feature weight maps The final feature weight map is obtained by fusion. Q , the original input feature map is converted into a weight factor M Using feature weight map under the control of Q Perform secondary learning to obtain a feature map with 1 / 2 resolution of the original input image.

[0020] Step 2.4: Convert the feature map with 1 / 2 resolution of the original input image obtained in step 2.3 back into a feature sequence, repeat steps 2.1-2.3 to obtain feature sequences with lengths of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input feature sequence, respectively, and finally input the extracted high-level semantic feature sequence into the decoder.

[0021] Step 2.5: The high-level semantic feature sequence of 1 / 32 length of the original input feature sequence obtained in step 2.1 is passed through the decoding area to re-obtain feature images of different scales. The decoding area and the encoding area together constitute the encoding and decoding area, and the up-sampling feature recovery is gradually completed through the LAG-Transformer layer to achieve the restoration of the synthetic aperture high-resolution optically degraded image. Specifically: first, the high-level semantic feature sequence of 1 / 32 length of the original input feature sequence output from the encoder is passed through the LAG-Transformer layer, and the length of its feature sequence is increased by layer normalization and FPBG module to make it a high-level semantic feature sequence of 1 / 16 length of the original input feature sequence, and then a simple skip connection operation is performed with the feature image of the corresponding size of the encoding layer, and then the GRM-Convolution layer and ASFE layer are input and similar operations are performed until the generated image with 3 channels and a size of 512×512 is finally restored.

[0022] The LGFAformer network is mainly composed of an encoding network and a decoding network. The encoding and decoding network uses the LAG-Transformer layer to replace the downsampling layer in the traditional U-Net network, and constructs the GRM-Convolution layer and ASFE module to enhance the network's ability to pay attention to global and local information.

[0023] like Figure 3 As shown, the LAG-Transformer layer proposed in the present invention is composed of a forward propagation module based on the gating mechanism (The Forward Propagation Module Based on The Gating Mechanism, FPBG), a focused linear attention module (FLA) and a normalization layer (LayerNorm). Among them, the focused linear attention module FLA is composed of a Relu activation function, a focused linear attention function and a fully connected layer; the forward propagation module FPBG based on the gating mechanism is composed of a depth-separable convolution, an effective channel attention module (Effectivechannel attention module, ECA) and a Gelu activation function; The aggregation mapping function is expressed as: , in, represents the similarity function, Respectively represent query vectors and key vectors of different dimensions, Relu represents the Relu activation function, express Element power of .

[0024] The LAG-Transformer layer takes the degraded image as input, normalizes the input features, divides the feature information into multiple windows, and calculates attention on each window, which reduces the amount of calculation on the high-resolution image and establishes long-distance dependencies within the window. In order to highlight the important feature information, multiple lightweight learnable tensors (size is , where W is the window size and C is the feature channel dimension), each tensor is added as a shared weight to the non-overlapping window before linear attention processing, which improves the performance significantly with the increase of slight parameters.

[0025] Focusing on the linear attention module FLA, a new linear calculation method is used to replace the Softmax in the original multi-head attention (MHA), reducing the computational complexity from Reduced to . First, the input tensor is linearly transformed to obtain the query vector Q, key vector K and value vector V. The query vector and key vector are activated respectively to ensure that they are non-zero and then scaled using a learnable factor. In order to prevent the lack of sharp attention distribution similar to Softmax in traditional linear attention, an aggregation mapping function is used to adjust the direction of each query and key vector feature, so that similar query vectors and key vector pairs are closer, and at the same time, dissimilar vector pairs are pushed apart, making the difference between the two more obvious, simulating the sharp attention distribution in Softmax. The key-value pair (Key-Value Pair) is linearly calculated for the key vector and value vector using the aggregation mapping function, and then linearly calculated with the query vector, and finally fused with the value vector extracted by the deep separable convolution. To prevent overfitting, the final result is passed through the linear layer, and the Dropout operation is used to set some elements of the tensor to zero.

[0026] The forward propagation module FPBG first passes the features through The convolution adjusts the number of channels and then uses two Depthwise separable convolutions, one for expanding channel features (expansion factor is 2), and the other for reducing the channels to the original input dimension. The depth-separable convolution extracts information from spatially adjacent pixel positions, which helps the LAG layer learn local image structures to restore the global image. One path is implemented using the GELU activation function, and the other uses a lightweight ECA attention to focus on the connection between feature channels. The two branches perform element-wise product operations and then use Convolution adjusts the number of channels to help the LAE module obtain more context information. The calculation of the element product between two branches is used to represent the implementation of the gating mechanism.

[0027] like Figure 4 As shown in FIG. 1 , the residual convolution layer based on the gating mechanism proposed in the present invention mainly consists of a deformable convolution, a depth-separable convolution, a normalization layer and a simple gating mechanism. The depth-separable convolution performs common extraction operations on feature information and uses The deep separable convolution controls the number of feature channels, and uses a simple gating mechanism to divide the features into two parts in the channel dimension, and then multiply them pixel by pixel. This can effectively replace the GELU activation function, help reduce the occurrence of gradient explosion, and has low computational complexity. At the same time, the deformable convolution input feature is used for deformation extraction operation. The deformable convolution introduces a learnable offset in the receptive field. Compared with traditional convolution, it deforms the convolution kernel of a fixed size, and the position of each convolution kernel is adaptively adjusted according to the learnable offset, which realizes more flexible extraction of local features, so that GRM-convolution can better adapt to local changes in input features, improves the generalization ability of GRM-convolution, and enables it to capture local information of different data features while also adapting to different data distributions. After two deformable convolution extractions, the residual connection is performed with the features after the gating mechanism, and then a simple gating operation is performed on it to improve the network's ability to express features.

[0028] like Figure 5 As shown in the figure, the adaptive scale feature enhancement module proposed in the present invention is mainly composed of depthwise separable convolution, deformable convolution, and average pooling operations. The output features of 2.2 are used to obtain the feature weight map based on the current features and then fused to sharpen the structural information of the extracted features and enhance the network's ability to restore the image structure. First, the input features are converted into feature tensors using two The deformable convolution extracts features, using its offset property to focus on information of different shapes on the features, and then uses The depth-wise separable convolution adjusts the number of channels to obtain the feature weight map , using two The feature weight map is obtained by depthwise separable convolution , then concatenate the features extracted by the depthwise separable convolution and the deformable convolution, and use the average pooling operation to obtain the feature weight map , and finally the weight map The final weight map is obtained after fusion Q , set a learnable weight parameter M (initial M = 0), the input features are combined with the weight map under the control of the weight factor Q Perform point multiplication to finally obtain features with sharper structural information.

[0029] Step 3: Input the training set and validation set that have been preprocessed and simulated in step 1 into the synthetic aperture optical image restoration network LGFAformer in step 2 for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the optimal parameter model. Specifically: The parameters of the synthetic aperture optical system image restoration network based on local-global feature enhancement are randomly initialized, and the preprocessed training set and validation set data in step one are input into the synthetic aperture optical system image restoration network based on local-global feature enhancement in step two. The restored optical image is output, and the loss is calculated with the original image. The minimization of the discrimination loss is taken as the optimization goal, and the network parameters are automatically updated to obtain the optimal parameter model and save it.

[0030] In step 3, Charbonnier loss is used as the loss function of the entire network, which is specifically defined as: , in, represents the loss function, Indicates the generated image, represents the original image, is a constant.

[0031] Step 4: Input the test set that has been simulated and degraded in step 1 into the optimal parameter model trained in step 3, and output the restored image of the synthetic aperture optical system.

[0032] In order to prove the effectiveness of the synthetic aperture optical image restoration method based on local-global feature enhancement proposed in this paper, the model is trained, verified and tested after simulation degradation using DOTA remote sensing data. Figure 6 As can be seen from (a)-(h) and Table 1, the results of this embodiment are basically higher than the compared methods in terms of peak signal-to-noise ratio, structural similarity and perceptual similarity, and the restoration result is closest to the original image. When it is weaker than a certain SOTA method, the model of the present invention is also much smaller than this method in terms of parameter quantity, proving that this model is more lightweight and is not much different from the SOTA. In addition, this embodiment has excellent performance in dealing with the ringing phenomenon that occurs in the imaging of synthetic aperture optical systems compared to traditional deep learning deblurring algorithms, such as Figure 7 As shown in (a) and (b) in Figure 2. Figure 8 (a) and (b) in Figure 3 and Table 3 show the comparison results of degraded images under different degrees of noise and the restoration results of different methods. In addition, this embodiment uses the YOLOv7 method to perform detection comparison on the unrestored and restored images, such as Fig. 9 As shown in (a) and (b), the results show that the restoration method helps the YOLOv7 method to improve the detection rate by about 40%, proving the practicality of the method of the present invention.

[0033] Table 1 Quantitative experimental results of different methods on 512×512 resolution images

[0034] Table 2 Quantitative experimental results of different methods on images affected by ringing factors

[0035] Table 3 Quantitative experimental results of different methods on images affected by 40dB-50dB noise

[0036] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned synthetic aperture optical image restoration method based on local-global feature enhancement are implemented.

[0037] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the aforementioned synthetic aperture optical image restoration method based on local-global feature enhancement are implemented.

[0038] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0040] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0041] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0042] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A synthetic aperture optical image restoration method based on local-global feature enhancement, characterized in that: The steps include: Step 1, obtaining a synthetic aperture optical image data set and performing simulated degradation to obtain a simulated degraded image data set, wherein the synthetic aperture optical image data set corresponds one-to-one with images in the simulated degraded image data set, and the corresponding image data set is randomly divided into a training set, a validation set, and a test set; Step 2, constructing a synthetic aperture optical image restoration network, including an encoding network and a decoding network, the encoding network includes a first module to a fifth module connected in sequence, an input mapping module is connected between the first module and the original input image, the decoding network includes a sixth module to a tenth module connected in sequence, the input end of the sixth module is connected to the output end of the fifth module, the output end of the tenth module is connected to the second gated residual convolution layer, and the output of the gated residual convolution layer is used as the final output of the synthetic aperture optical image restoration network; The structures of the first module to the tenth module are the same, and all include a linear gated Transformer layer, a first gated residual convolution layer and an adaptive scale feature enhancement module connected in sequence, the output of the input mapping module is jump-connected to the output of the second gated residual convolution layer, the output of the adaptive scale feature enhancement module in the first module is jump-connected to the output of the adaptive scale feature enhancement module in the tenth module, the output of the adaptive scale feature enhancement module in the second module is jump-connected to the output of the adaptive scale feature enhancement module in the ninth module, the output of the adaptive scale feature enhancement module in the third module is jump-connected to the output of the adaptive scale feature enhancement module in the eighth module, the output of the adaptive scale feature enhancement module in the fourth module is jump-connected to the output of the adaptive scale feature enhancement module in the seventh module, and the output of the adaptive scale feature enhancement module in the fifth module is jump-connected to the output of the adaptive scale feature enhancement module in the sixth module; The input mapping module maps the original input image to the original input feature sequence. The first to fifth modules respectively output feature sequences with lengths of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original input feature sequence. The sixth to tenth modules respectively output feature sequences with lengths of 1 / 32, 1 / 16, 1 / 8, 1 / 4 and 1 / 2 of the original input feature sequence. Step 3, input the training set and validation set obtained in step 1 into the synthetic aperture optical image restoration network constructed in step 2 for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the trained optimal parameter model; Step 4: Input the test set obtained in step 1 into the optimal parameter model trained in step 3, and output the restored image of the synthetic aperture optical system.

2. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 1, characterized in that: The specific process of step 1 is as follows: Obtain a synthetic aperture optical image data set, measure the actual synthetic aperture optical system parameters, obtain the system point spread function, save it in txt format, and then use pycharm software to read the point spread function and convolve it with the synthetic aperture optical image data set to obtain a simulated degraded image data set; The images in the synthetic aperture optical image dataset and the simulated degradation image dataset are cropped into Then, the images in the synthetic aperture optical image dataset and the simulated degraded image dataset are mapped one by one to the input image and the target image; The corresponding image dataset is randomly divided into three parts: 80% for training set, 10% for validation set, and 10% for test set.

3. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 1, characterized in that: The linear gated Transformer layer includes first to fourth normalization layers, first to second forward propagation modules, a window multi-head focused linear attention module, and a shifted window multi-head focused linear attention module; The input of the linear gated Transformer layer After passing through the first normalization layer and the window multi-head focused linear attention module in sequence, the output of the window multi-head focused linear attention module is obtained. The output of the window multi-head focused linear attention module is Add up to get the feature sequence ; After passing through the second normalization layer and the first forward propagation module in sequence, the output of the first forward propagation module is obtained. The output of the first forward propagation module is Add up to get the feature sequence ; After passing through the third normalization layer and the shift window multi-head focused linear attention module in sequence, the output of the shift window multi-head focused linear attention module is obtained. The output of the shift window multi-head focused linear attention module is Add up to get the feature sequence ; After passing through the fourth normalization layer and the second forward propagation module in sequence, the output of the second forward propagation module is obtained. The output of the second forward propagation module is The sum is used as the output of the linear gated Transformer layer.

4. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 3, characterized in that: The window multi-head focused linear attention module and the shifted window multi-head focused linear attention module both include a focused linear attention module; The window multi-head focused linear attention module divides the output of the first normalization layer into multiple windows, calculates the attention of each window using the focused linear attention module, and then concatenates the attention of all windows as the output of the window multi-head focused linear attention module; The shifted window multi-head focused linear attention module shifts the output of the third normalization layer and divides it into multiple windows. The focused linear attention module is used to calculate the attention of each window, and then the attention of all windows is concatenated as the output of the shifted window multi-head focused linear attention module. The input of the focused linear attention module is divided into a query vector, a key vector and a value vector through a linear transformation. The query vector is activated by the Relu function and scaled, and then solved by the first Euclidean norm; the key vector is activated by the Relu function and scaled, and then solved by the second Euclidean norm; the results of the first Euclidean norm and the second Euclidean norm enter the focusing function to obtain the focusing result, and the focusing result is transposed to obtain the transposed result; the transposed result is Einstein added to the value vector to obtain the KV key-value pair, and the KV key-value pair is Einstein added to the focusing result to obtain the first result; the value vector is The depthwise separable convolution is then added to the Einstein result to obtain the second result, which is then passed through the fully connected layer and the random dropout layer to obtain the output of the focused linear attention module. The focusing function formula is as follows: ， in, represents the similarity function, Represent query vectors and key vectors of different dimensions respectively, Express The activation function performs focused linear attention calculation, Express The activation function is used to perform focused linear attention calculation, Relu represents the Relu activation function, represents the focused linear attention calculation function, express Element power of .

5. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 3, characterized in that: The first forward propagation module and the second forward propagation module have the same structure, both of which include Convolutional layers, first to third Depthwise separable convolutional layer, first to second Depthwise separable convolutional layers, Gelu activation functions, and effective channel attention modules; The input of each forward propagation module is After the convolutional layer, The first feature sequence output by the convolutional layer is converted into a feature map; the feature map passes through two branches respectively. In the first branch, the feature map passes through the first Depthwise separable convolutional layer, first The depth-separable convolutional layer and the effective channel attention module are used to obtain the output of the first branch; in the second branch, the feature map is sequentially processed by the second Depthwise separable convolutional layer, second Depth-wise separable convolutional layer and Gelu activation function to obtain the second branch output; The second branch output is multiplied by the first branch output and the feature map, and the dot product result is passed through the third After the depth-wise separable convolutional layer, the third The feature map output by the depthwise separable convolutional layer is converted into a second feature sequence, and the second feature sequence is added to the first feature sequence as the output of each forward propagation module.

6. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 1, characterized in that: The first gated residual convolution layer and the second gated residual convolution layer have the same structure, both of which include the fifth to seventh normalization layers, the fourth to seventh Depthwise separable convolutional layer, third Depthwise separable convolutional layers, first to second deformable convolutional layers, and simple gating mechanism; The feature sequences input by each gated residual convolution layer are converted into feature maps, which are then passed through the fifth normalization layer, the fourth Depthwise separable convolutional layer, third Depthwise separable convolutional layer, fifth After the depth-separable convolutional layer and the sixth normalization layer, the output of the sixth normalization layer is obtained ; The feature map passes through the first deformable convolution layer, the seventh normalization layer and the second deformable convolution layer in sequence, and the output of the second deformable convolution layer is obtained ; and Add to get the feature map , The sixth Depthwise separable convolutional layers, simple gating mechanisms, and seventh The depth-wise separable convolutional layer obtains the seventh Output of a depthwise separable convolutional layer , and Add to get the feature map ,Will Converted into a feature sequence as the output of each gated residual convolution layer.

7. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 1, characterized in that: The adaptive scale feature enhancement module includes eighth to eleventh Depthwise separable convolutional layers, fourth to sixth Depthwise separable convolutional layer, third to fourth deformable convolutional layers, and average pooling layer; Convert the feature sequence input by the adaptive scale feature enhancement module into a feature map , After two branches, in the first branch, Sequentially through the third deformable convolution layer, the fourth deformable convolution layer and the eighth Depth-wise separable convolutional layer to obtain feature maps ; In the second branch, The fourth Depthwise separable convolutional layer, fifth Depthwise separable convolutional layer and the tenth Depth-wise separable convolutional layer to obtain feature maps ; and Concatenate to get feature map ; The Ninth Depthwise separable convolutional layer obtains feature weight map , Chapter 11 Depthwise separable convolutional layer obtains feature weight map , The sixth Depthwise separable convolutional layer and average pooling layer to obtain feature weight map ; and Add together to get the feature weight map , and Add together to get the feature weight map Q , Q and Point multiplication to get feature map N ,Will N The converted feature sequence is used as the output of the adaptive scale feature enhancement module.

8. The synthetic aperture optical image restoration method based on local-global feature enhancement according to claim 1, characterized in that: In step 3, the loss function is Charbonnier loss, and the formula is as follows: ， in, represents the loss function, Indicates the generated image, represents the original image, is a constant, .

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the synthetic aperture optical image restoration method based on local-global feature enhancement are implemented as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the synthetic aperture optical image restoration method based on local-global feature enhancement as described in any one of claims 1 to 8 are implemented.

Images