Advanced imaging system with variable-direction resolution image intelligent fusion restoration system and method
Through the combination of deep neural network, coordinate aggregation module and fuzzy direction attention module, the image restoration problem caused by aberration and pupil angle error in MAPF imaging system is solved, and high-quality image fusion restoration is achieved, adapting to complex systems and improving resolution Rate.
Patent Information
- Application Number
- CN202411191084.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The existing MAPF imaging system has degraded image quality caused by aberration during image restoration, and it is difficult to separate the impact of pupil angle measurement error and complex fuzzy core coupling, and it is difficult to achieve high-quality image restoration in existing methods.
Deep neural network is adopted, and the coordinate aggregation module and the fuzzy direction attention module are introduced. Image feature stitching and fusion are performed through sequential downsampling and upsampling, and image degradation mechanism and mapping relationship are learned to achieve image denoising and fusion restoration without prior information.
It improves image resolution and is adaptable, and can effectively handle system aberration, pupil motion and noise effects, reduces artificial decoupling errors, and improves work efficiency.
Smart Images

Figure CN118710520B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an advanced imaging system variable-direction resolution image intelligent fusion restoration system and method. Background Art
[0002] Advanced imaging system refers to a new type of large-aperture space optical payload system. The most common advanced imaging system is MAPF (Multi-Angle Pupil Fusion) imaging system, which uses a large aspect ratio rectangular or elliptical reflector as the main mirror. During the imaging process, the pupil of the MAPF imaging system is changed to rotate around the optical axis to obtain the degraded images collected in different directions of the observed target. Through image fusion restoration and other means, a full-range high-resolution restored image with the long side of the main mirror as the equivalent aperture is finally obtained. Its imaging mechanism is as follows: Figure 1 Affected by system aberrations, primary mirror motion around the optical axis, satellite platform vibration, detector sampling, random noise and other factors, the degradation effect of the image collected by this imaging system is relatively complex, and complex processing is required to obtain a relatively ideal high-resolution image.
[0003] For image restoration of MAPF imaging system, the process of traditional image restoration technology is as follows Figure 2 As shown, it mainly includes three steps: image registration, single-cycle image restoration and multi-cycle image super-resolution processing:
[0004] Step 1: Image registration: For the MAPF system, the movement of the pupil around the optical axis will cause the point spread function (PSF) to change with the pupil direction. The directional characteristics of the PSF are coupled with the vibration characteristics of the satellite platform, which will cause the image to shift geometrically under relatively good conditions. Existing methods extract the grayscale gradient information of the same scene in the image, and register multiple degraded images based on the grayscale gradient to restore their spatial correlation and eliminate the geometric shift;
[0005] Step 2: Single-cycle image restoration: After completing image registration, the traditional method obtains the relationship between the degraded image and the target scene in different pupil directions based on image priors, establishes a regularized single-cycle image restoration model, and iteratively solves the restored image;
[0006] Step 3: Multi-period image super-resolution processing: Establish constraints and interpolate and reconstruct multiple restored images of different periods to improve the resolution of the restored image.
[0007] The existing image restoration technology for MAPF imaging system has the following disadvantages:
[0008] 1) In the actual operation of the MAPF imaging system, the movement of the pupil around the optical axis will introduce large aberrations to the imaging system, and the aberrations will cause large unknown changes in the PSF of the system, which will reduce the quality of the degraded image and further affect the quality of the restored image. This change cannot be obtained from prior knowledge, and existing methods cannot solve this problem;
[0009] 2) In the image registration and image restoration technology based on prior knowledge, there are clear requirements for the pupil angle corresponding to the degraded image. However, in practical applications, there are errors in the measurement of the pupil angle, and existing methods are difficult to adapt;
[0010] 3) The blur kernel of image degradation in the MAPF imaging system is relatively complex. The coupling effects caused by system aberrations, pupil motion around the optical axis, platform vibration, random noise, and sampling process are difficult to separate. Existing methods decompose their effects based on prior knowledge, and the restoration effect achieved is very limited. Summary of the invention
[0011] In view of this, the invention aims to provide an advanced imaging system with variable-resolution image intelligent fusion restoration system and method, which uses a deep neural network to learn the degradation mechanism and mapping relationship of the image, denoises and fuses multiple target images taken at different angles at one time, and improves their image resolution. The whole process does not require any prior information or fuzzy estimation, and can better adapt to systems with complex imaging mechanisms.
[0012] To achieve the above object, the technical solution created by the present invention is implemented as follows:
[0013] An intelligent fusion restoration system for variable directional resolution images of an advanced imaging system comprises a coding branch, a coordinate aggregation module, a fuzzy directional attention module and a decoding branch; wherein, in the coding branch, a group of degraded image pairs input are successively downsampled for no less than N times to obtain N groups of degraded image pairs of different sizes, N>1; each group of degraded image pairs is input into the coordinate aggregation module to obtain the coordinate aggregation features corresponding to each group of degraded image pairs; each coordinate aggregation feature is input into the fuzzy directional attention module to obtain the corresponding fuzzy directional features; in the decoding branch, the fuzzy directional features corresponding to the last downsampled degraded image pair are upsampled; the upsampled features are spliced and fused with the fuzzy directional features corresponding to the last downsampled degraded image pair, and the spliced and fused features are upsampled; the spliced and fused and upsampling operations are repeated until the fuzzy directional features corresponding to the input degraded image pairs are spliced and fused, and the output fused restoration image and the corresponding spectrum diagram are obtained.
[0014] Furthermore, each pair of degraded images includes a degraded image and a frequency spectrum corresponding to the degraded image.
[0015] Furthermore, each group of degraded image pairs has a corresponding coordinate aggregation module; in each coordinate aggregation module, the input degraded image pairs are split according to the types of images to form a degraded image group and a spectrum graph group; for the degraded image group, two consecutive convolution operations are performed on the degraded image group to obtain no less than 2 spatial features, and all the spatial features are spliced and fused to obtain spatial feature weights; the spatial feature weights are multiplied by the corresponding elements of the degraded image group to obtain spatial coordinate features; the operation on the spectrum graph group is consistent with the operation on the degraded image group to obtain frequency domain coordinate features; the spatial coordinate features are spliced with the frequency domain coordinate features to obtain coordinate aggregation features.
[0016] Furthermore, each coordinate aggregation feature has a corresponding fuzzy direction attention module; in each fuzzy direction attention module, the input coordinate aggregation feature is split into a single feature map; after convolution operation on each feature map, the feature maps corresponding to the spatial coordinate features are summarized, and the summarized features are convolved to obtain the fuzzy direction spatial feature weights; the feature maps corresponding to the frequency domain coordinate features are summarized, and the summarized features are convolved to obtain the fuzzy direction frequency domain feature weights; the fuzzy direction spatial feature weights are multiplied by the corresponding elements of the spatial coordinate features to obtain spatial fuzzy features; the fuzzy direction frequency domain feature weights are multiplied by the corresponding elements of the frequency domain coordinate features to obtain frequency domain fuzzy features; the spatial fuzzy features and the frequency domain fuzzy features are concatenated to obtain fuzzy direction features.
[0017] Furthermore, in the decoding branch, when the up-sampled features are spliced and fused with the fuzzy direction features corresponding to the last down-sampled degraded image pair, the spatial fuzzy features of the two features are spliced and fused accordingly, and the frequency domain fuzzy features of the two features are spliced and fused accordingly.
[0018] An advanced imaging system variable-resolution image intelligent fusion restoration method, specifically comprising the following steps:
[0019] S1: Acquire image data, preprocess the image data, obtain degraded image pairs, and form a degraded data set;
[0020] S2: Constructing an advanced imaging system with variable-direction resolution image intelligent fusion restoration system as provided by the present invention;
[0021] S3: inputting the degraded data set into the fusion restoration system in step S2, and training the fusion restoration system to obtain a trained fusion restoration model;
[0022] S4: Use the fusion restoration model obtained in step S3 to restore the image to be restored.
[0023] Furthermore, the preprocessing in step S1 specifically includes the following steps:
[0024] S11: cropping the image data;
[0025] S12: obtaining a corresponding spectrogram according to the cropped image data, and using the cropped image data and the corresponding spectrogram as a true image pair;
[0026] S13: Convert the cropped image data into the first image intensity distribution collected by the MAPF imaging system:
[0027] S14: converting the first image intensity distribution into a second image intensity distribution at the image plane of the area array detector in the MAPF imaging system; and obtaining a degraded image according to the second image intensity distribution by the following formula:
[0028]
[0029] in, represents the second image intensity distribution, represents the sampling function of the area array detector, represents noise, represents a degraded image;
[0030] S15: obtaining a frequency spectrum corresponding to the degraded image according to the degraded image; the degraded image and the corresponding frequency spectrum constitute a degraded image pair;
[0031] S16: Repeat steps S11 to S15 to obtain real image pairs and degraded image pairs corresponding to different image data, and combine all degraded image pairs and all real image pairs to form a degraded data set.
[0032] Further, in step S13, the first image intensity distribution is obtained by the following formula:
[0033]
[0034] in, Represents the cropped image data, represents the point spread function of the MAPF imaging system, represents the first image intensity distribution, and t represents the shooting time of the MAPF imaging system.
[0035] Compared with the prior art, the invention can achieve the following beneficial effects:
[0036] (1) The advanced imaging system variable direction resolution image intelligent fusion restoration system created by the present invention improves the existing U-Net network and introduces a coordinate aggregation module and a fuzzy direction attention module. The coordinate aggregation module system can adapt to the multi-image fusion restoration task. The fuzzy direction attention module allows the system to learn the high-quality directional information of the fuzzy direction spectrum map features and retain its features, which is helpful to restore the spectrum map of the clear image, solving the problem that the existing method cannot realize the fusion restoration processing of the MAPF simulation image with aberration;
[0037] (2) The variable-resolution image intelligent fusion restoration method for an advanced imaging system created by the present invention does not require any prior knowledge and has strong adaptability. It processes the coupling effects caused by system aberrations, pupil movement around the optical axis, platform vibration, random noise, and the sampling process as a whole, and realizes image registration, image restoration, and super-resolution processing at one time, thus avoiding the errors caused by artificial decoupling and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:
[0039] Figure 1 Schematic diagram of the imaging mechanism of the traditional MAPF imaging system;
[0040] Figure 2 It is a flowchart of the image restoration technology of the traditional MAPF imaging system;
[0041] Figure 3 A schematic diagram of the structure of an advanced imaging system with variable-direction resolution image intelligent fusion restoration system according to an embodiment of the present invention;
[0042] Figure 4 A schematic diagram of a coordinate aggregation module according to an embodiment of the present invention;
[0043] Figure 5 A schematic diagram of a fuzzy direction attention module according to an embodiment of the present invention;
[0044] Figure 6 A schematic diagram of the process of the intelligent fusion restoration method of the image with variable direction resolution of the advanced imaging system according to the embodiment of the present invention;
[0045] Figure 7 This is a diagram of the fusion restoration result of the intelligent fusion restoration method for images with variable directional resolution in an advanced imaging system described in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.
[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0048] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.
[0049] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.
[0050] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.
[0051] like Figure 3 As shown, the advanced imaging system variable direction resolution image intelligent fusion restoration system described in the embodiment of the present invention includes an encoding branch, a coordinate aggregation module, a fuzzy direction attention module and a decoding branch.
[0052] In the encoding branch, a group of input degraded image pairs are successively downsampled for no less than N times to obtain N groups of degraded image pairs of different sizes, where N>1. Each group of degraded image pairs includes C pairs of degraded image pairs, each pair of degraded image pairs includes a degraded image and a spectrum graph corresponding to the degraded image, that is, each group of degraded image pairs includes C degraded images and C spectrum graphs, and each degraded image and each spectrum graph have C degraded images and C spectrum graphs. Figure 1 One-to-one correspondence. In the embodiment of the present invention, the number of downsampling N is preferably set to 2 times, corresponding to obtaining 2 groups of degraded image pairs with different sizes, and a total of 3 groups of degraded image pairs with different sizes plus the input group of degraded image pairs.
[0053] Each group of degraded image pairs has a corresponding coordinate aggregation module and a fuzzy direction attention module. Each group of degraded image pairs is input into the corresponding coordinate aggregation module to obtain corresponding coordinate aggregation features. The corresponding coordinate aggregation features of each group of degraded image pairs are then input into the corresponding fuzzy direction attention module to obtain corresponding fuzzy direction features. Since there are 3 groups of degraded image pairs of different sizes in the embodiment of the present invention, there are 3 corresponding coordinate aggregation modules and 3 fuzzy direction attention modules, and each group of degraded image pairs has its corresponding coordinate aggregation module and fuzzy direction attention module.
[0054] like Figure 4 As shown, in each coordinate aggregation module, the input degraded image pair is split into a degraded image group and a spectrum graph group according to the type of the image, where the size of the image in the degraded image pair is H×W, H represents the length of the image, and W represents the width of the image. The degraded image group includes C degraded images, and the spectrum graph group includes C spectrum graphs. For the degraded image group, two consecutive convolution operations are performed on the degraded image group to obtain no less than 2 spatial features. All spatial features are spliced and fused to obtain spatial feature weights.
[0055] In the embodiment of the present invention, the first convolution is to extract a single-layer channel to obtain a single-layer channel feature. Specifically, the degraded image group is convolved by a convolution kernel of size 1×1, and the result after the convolution is activated by a SELU activation function to obtain a single-layer channel feature of size H×W×C, that is, the number of channels of the feature is the same as the number of degraded images in the degraded image group, which is C, and the size of each feature map of the feature is the same as the size of the degraded image, which is H×W.
[0056] Among them, in the embodiment of the present invention, the SELU (Scaled Exponential Linear Unit) activation function is selected to complete the activation operation. Among them, the activation function introduces an automatic normalization mechanism so that the hidden layer of the neural network can automatically keep the mean and variance of the output close to 1 during the training process, which helps to solve the gradient explosion and gradient disappearance problems in the neural network, thereby making deep networks easier to train. The SELU activation function has an exponential growth when the input is a negative number, and is approximately linear when the input is a positive number. This nonlinear characteristic makes SELU perform better than activation functions such as ReLU. The SELU activation function is as follows:
[0057]
[0058] in, The parameter controls the scaling of the SELU function when the input is positive, which ensures that the output mean of the SELU function is 0 and the output variance is 1 when the input is positive. The parameter controls the behavior of the SELU function when a negative value is input, which can ensure that the SELU function has the expected gradient propagation characteristics when a negative value is input. In an embodiment of the present invention, , .
[0059] The single-layer channel features are then used to extract multi-channel spatial information. In the embodiment of the present invention, a 3×3 convolution kernel is used to convolve the single-layer channel features, and the result after the convolution is activated by a SELU activation function to obtain 9 spatial features of a size of H×W×C. The 9 spatial features are concatenated and fused to obtain the spatial feature weights.
[0060] It should be noted that all the “splicing and fusion” operations mentioned in the embodiments of the present invention are to first splice the multiple input features in the channel direction, then input the spliced features into the convolution layer for convolution operation, and activate the convolution result with the SELU activation function to obtain the spliced and fused features.
[0061] Therefore, when splicing and fusing the 9 spatial features, the 9 spatial features are first spliced by channels, and then the spliced features are convolved with a 1×1 convolution kernel, and the convolution result is activated by the SELU activation function to obtain a spatial feature weight of size H×W×C. After that, the spatial feature weight is multiplied by the corresponding element of the degraded image group to obtain a spatial coordinate feature of size H×W×C.
[0062] In each coordinate aggregation module, the above operation is repeated for the spectrum graph group to obtain the frequency domain feature weight corresponding to the spectrum graph group, and the frequency domain feature weight is multiplied by the corresponding element of the spectrum graph group to obtain a frequency domain coordinate feature of size H×W×C. The spatial coordinate feature and the frequency domain coordinate feature are concatenated to obtain a coordinate aggregation feature of size H×W×2C, that is, the coordinate aggregation feature of size H×W×2C contains a spatial coordinate feature of size H×W×C and a frequency domain coordinate feature of size H×W×C.
[0063] The essence of coordinate fusion is to transfer the key information near the key point coordinates to the same channel through dimensional changes, and then assign different weights to this channel. This is equivalent to assigning weights to the features within the range of each convolution according to different positions, rather than the shared parameterization of ordinary convolution. This can solve the problem of extracting the same features at different positions and achieve special attention to important information areas.
[0064] like Figure 5 As shown, in each fuzzy directional attention module, the input coordinate aggregation features are split into a single feature map of size H×W×1, that is, after the coordinate aggregation features of size H×W×2C are split, feature maps corresponding to C spatial coordinate features and feature maps corresponding to C frequency domain coordinate features are obtained. The sizes of the feature maps corresponding to the spatial coordinate features and the frequency domain coordinate features are both H×W×1.
[0065] After each feature map is convolved with a 1×1 convolution kernel, and the convolution result is activated by the SELU activation function, the feature maps corresponding to the spatial coordinate features are summarized, and the summarized features are convolved with a 3×3 convolution kernel, and the convolution result is activated by the SELU activation function to obtain the fuzzy direction spatial feature weight, and the size of the fuzzy direction spatial feature weight is H×W×C; the feature maps corresponding to the frequency domain coordinate features are summarized, and the summarized features are convolved with a 3×3 convolution kernel, and the convolution result is activated by the SELU activation function to obtain the fuzzy direction frequency domain feature weight, and the size of the fuzzy direction frequency domain feature weight is also H×W×C.
[0066] Multiply the weight of the fuzzy direction spatial feature with the corresponding element of the spatial coordinate feature to obtain a spatial fuzzy feature of size H×W×C; multiply the weight of the fuzzy direction frequency domain feature with the corresponding element of the frequency domain coordinate feature to obtain a frequency domain fuzzy feature of size H×W×C. Finally, concatenate the spatial fuzzy feature and the frequency domain fuzzy feature to obtain a fuzzy direction feature of size H×W×2C.
[0067] The essence of blur direction attention is that each channel in the image will change due to the blur direction. By convolving each channel separately, we can learn the changing trends from multiple channels at the same time, and then aggregate these trends, thereby strengthening the model's attention to the blur direction of the image.
[0068] In the decoding branch, the fuzzy direction features corresponding to the last downsampled degraded image pair are upsampled. The upsampled features are spliced and fused with the fuzzy direction features corresponding to the last downsampled degraded image pair, and the spliced and fused features are upsampled. The splicing, fusion and upsampling operations are repeated until the fuzzy direction features corresponding to the input degraded image pair are spliced and fused, and the output fused restored image and the corresponding spectrum are obtained.
[0069] In the process of splicing and fusing the fuzzy direction features corresponding to the features obtained by each upsampling and the degraded image pair obtained by the previous downsampling, the spatial fuzzy features of the two features are spliced and fused accordingly, and the frequency domain fuzzy features of the two features are spliced and fused accordingly.
[0070] In the embodiment of the present invention, since downsampling is performed twice in the encoding branch, upsampling is performed twice in the decoding branch. The first upsampling is to upsample the blur direction features corresponding to the degraded image pair of the second downsampling. The features obtained by the first upsampling are spliced and fused with the blur direction features corresponding to the degraded image pair of the first downsampling. Specifically, in the blur direction features corresponding to the features obtained by the first upsampling and the degraded image pair of the first downsampling, the spatial blur features of the two features are spliced and fused, and the frequency domain blur features of the two features are spliced and fused. The spliced and fused features are upsampled for the second time. The features obtained by the second upsampling are spliced and fused with the blur direction features corresponding to the input degraded image pair. Specifically, in the blur direction features corresponding to the features obtained by the second upsampling and the degraded image pair of the input, the spatial blur features of the two features are spliced and fused, and the frequency domain blur features of the two features are spliced and fused, so as to obtain the output fused restoration image and the corresponding spectrum diagram.
[0071] The embodiment of the present invention also provides an advanced imaging system variable direction resolution image intelligent fusion restoration method, which is used to fuse and restore the image acquired by the MAPF imaging system. Figure 6 As shown, the method for intelligent fusion restoration of images with variable directional resolution in an advanced imaging system described in the embodiment of the present invention specifically includes the following steps:
[0072] S1: Acquire image data, preprocess the image data, obtain degraded image pairs, and form a degraded data set.
[0073] In step S1, the image data comes from the public dataset remote sensing image military aircraft target recognition dataset MAR20 (Military Aircraft Recognition), which comes from the paper "MAR20: Remote Sensing Image Military Aircraft Target Recognition Dataset" published in Volume 12, Issue 27, 2023 of the Journal of Remote Sensing.
[0074] The preprocessing of image data specifically includes the following steps:
[0075] S11: cropping the image data. In the embodiment of the present invention, the image data is cropped to 600×600.
[0076] S12: Obtain a corresponding frequency spectrum according to the cropped image data, and use the cropped image data and the corresponding frequency spectrum as a real image pair. In the embodiment of the present invention, the cropped image data is subjected to Fourier transformation to obtain a corresponding frequency spectrum.
[0077] S13: Convert the cropped image data into a first image intensity distribution collected by the MAPF imaging system.
[0078] In step S13, the first image intensity distribution is obtained by the following formula:
[0079]
[0080] in, Represents the cropped image data, represents the point spread function of the MAPF imaging system, represents the first image intensity distribution, and t represents the shooting time of the MAPF imaging system.
[0081] Point spread function of MAPF imaging system It is obtained through the pupil function of the rectangular pupil in the MAPF imaging system, that is:
[0082]
[0083] in, represents the autocorrelation operation; The pupil function representing a rectangular pupil is obtained by the following formula:
[0084]
[0085]
[0086]
[0087] Where a and b represent the length and width of the rectangular pupil, respectively. Represents image data Center coordinate position The corresponding coordinate position after rotation, Indicates the length is , the width is The rectangular function of is the wavefront aberration of the system, is the angular velocity of the rectangular pupil moving around the optical axis, is the initial angle of the rectangular pupil.
[0088] S14: The first image intensity distribution is converted into a second image intensity distribution at the image plane of the array detector in the MAPF imaging system by the following formula:
[0089]
[0090] in, It represents the geometric displacement of the image caused by the object being photographed by the MAPF imaging system when it is shaken, that is, the geometric displacement of the image of the object in the x-direction and the y-direction at time t; It represents the second image intensity distribution generated at the image plane of the area array detector within the sampling time T of the area array detector.
[0091] According to the intensity distribution of the second image, the degraded image is obtained by the following formula:
[0092]
[0093] in, represents the degraded image, represents the second image intensity distribution, Represents noise, which is not limited to other noises such as Gaussian noise and Poisson noise; represents the sampling function of the area array detector, which is obtained by the following formula:
[0094]
[0095] in, Indicates the length is , the width is The rectangular function of Represents the number of pixels on the array detector, Indicates that the array detector is in the sampling interval The sampling function under Represents the size of a single pixel.
[0096] S15: Obtaining a frequency spectrum corresponding to the degraded image according to the degraded image. In the embodiment of the present invention, the degraded image is subjected to Fourier transformation to obtain a frequency spectrum corresponding to the degraded image. The degraded image and the corresponding frequency spectrum constitute a degraded image pair.
[0097] S16: Repeat steps S11 to S15 to obtain real image pairs and degraded image pairs corresponding to different image data, and combine all degraded image pairs and all real image pairs to form a degraded data set.
[0098] It can be seen that in the degradation process, factors such as the aberration of the MAPF imaging system, the movement of the pupil around the optical axis, the platform vibration, random noise, and the sampling process have a complex coupling relationship with the light intensity distribution of the collected image. The existing methods decouple and eliminate these influencing factors based on prior knowledge, but the effect is limited.
[0099] Although the above effects cannot be decoupled, they can be roughly divided into the following categories:
[0100] (1) Related to the pupil angle. This part of the impact is reflected in the image, causing the grayscale / frequency characteristics of the image corresponding to each pupil angle to change in a way related to the pupil angle;
[0101] (2) It is relatively random and has no statistical regularity. This part is mainly caused by system aberrations, which cannot be handled by existing methods at all, but deep learning can reduce this part of the impact to a certain extent;
[0102] (3) Random, but with statistical rules. This part mainly includes the sampling process, random noise, and image geometric offset. Existing methods have relatively mature calculation methods for this, but due to the use of a large amount of prior knowledge, when the impact of this part is small, it can be effectively eliminated. However, when the impact of this part is large, existing methods are difficult to adapt, while deep learning has strong adaptability and can easily eliminate it.
[0103] S2: Construct an advanced imaging system variable directional resolution image intelligent fusion restoration system as provided in the present invention, that is, construct an advanced imaging system variable directional resolution image intelligent fusion restoration system with a coordinate aggregation module and a fuzzy direction attention module based on the U-Net network, so that the fusion restoration system can adapt to the multi-image fusion restoration task, and at the same time let the fusion restoration system learn the high-quality directional information of the characteristics of the fuzzy direction spectrum map and retain its characteristics, which is helpful for the network to restore the spectrum map of the clear image, thereby realizing the fusion restoration of the image collected by the MAPF imaging system and eliminating the above-mentioned influence.
[0104] S3: Input the degraded data set into the fusion restoration system in step S2, and train the fusion restoration system to obtain a trained fusion restoration model.
[0105] In step S3, the degraded data set obtained in step S1 is divided into a training set and a validation set, and the ratio of the training set to the validation set is 7:3. The training set is input into the fusion restoration system in step S2 for training, wherein the degraded image pairs in the training set are used as the input of the fusion restoration system, and the real image pairs are used as the real values of the fusion restoration system, and the fusion restoration system is trained. The Adam optimizer is used in the training process, the learning rate is set to 0.001, the number of training iterations is 1000, and the number of pairs trained in each iteration is 8.
[0106] After each training, the degraded image pairs in the validation set are input into the trained fusion restoration model to obtain the corresponding fusion restoration image and the corresponding spectrum map. The MSE (mean square error) loss function is used to calculate the difference between the fusion restoration image and the corresponding spectrum map output each time and the corresponding real image pair in the validation set, that is:
[0107]
[0108]
[0109] in, Represents the MSE loss function between the fused restored image output by the system and the corresponding true value, represents the fused restored image obtained by inputting the fusion restoration model into the i-th pair of degraded images, represents the real image corresponding to the i-th pair of degraded images, and n represents the number of degraded image pairs in the validation set; Represents the MSE loss function between the spectrum output by the system and the corresponding true value, represents the spectrum obtained from the input fusion restoration model of the i-th pair of degraded images, Represents the true spectrum value corresponding to the i-th pair of degraded images.
[0110] S4: Use the fusion restoration model obtained in step S3 to restore the image to be restored.
[0111] In the embodiment of the present invention, an image is selected from the MAR20 data set, and the image is subjected to the degradation operation as described in step S1 to obtain a test image to be restored. The test image to be restored is input into the trained fusion restoration model to obtain a fusion restoration image and a corresponding spectrum diagram.
[0112] Existing public methods cannot restore MAPF images with aberrations:
[0113] For degraded images with large aberrations, directly fusing multi-directional images without de-aberration processing is equivalent to performing interpolation with aberrations, and the result is even worse than a single degraded image without processing.
[0114] If the degraded image is first subjected to a de-aberration operation, its directional characteristics will be destroyed, making it impossible to perform subsequent image fusion steps in various directions.
[0115] Therefore, the existing public methods have no way to perform fusion restoration processing on the "MAPF image with aberration" mentioned in the present invention.
[0116] In order to clearly demonstrate the improvement of the image intelligent fusion restoration method provided by the embodiment of the present invention, the embodiment of the present invention uses PSNR (peak signal-to-noise ratio) and SSIM (structural similarity ratio) as evaluation indicators to evaluate the quality of the fused image. The evaluation results are shown in Table 1:
[0117] Table 1:
[0118]
[0119] Table 1 shows the SNR and SSIM indicators of a single degraded image and the SNR and SSIM indicators of the restored image of the present invention under the conditions of signal-to-noise ratio SNR=30dB, pupil aspect ratio P=4, and aberration size of 0.6λrms. It can be seen that the method of the present invention can improve the image quality in all aspects and make it closer to the ideal image.
[0120] Figure 7 It shows the restoration effect described in the embodiment of the present invention under the conditions of signal-to-noise ratio SNR=30dB, pupil aspect ratio P=4, and aberration size of 0.6λrms. Figure 7 The left side is the original image, the middle is the simulated image corresponding to the MAPF pupil in four different directions, and the right side is the image obtained by fusion restoration based on the four degraded images using the image intelligent fusion restoration method provided by the embodiment of the present invention. It can be seen that Figure 7 There is significant directional blur and blur caused by aberration in a single simulated degraded image. After the fusion restoration method provided by the present invention, the blur in the image is well suppressed, and a better restored image is obtained (such as Figure 7 on the right).
[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.
[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An advanced imaging system with variable-direction resolution image intelligent fusion restoration system, characterized by: It includes an encoding branch, a coordinate aggregation module, a fuzzy direction attention module and a decoding branch; wherein, in the encoding branch, a group of degraded image pairs input are successively downsampled for no less than N times to obtain N groups of degraded image pairs of different sizes, N>1; each group of degraded image pairs is input into the coordinate aggregation module to obtain the coordinate aggregation features corresponding to each group of degraded image pairs; each coordinate aggregation feature is input into the fuzzy direction attention module to obtain the corresponding fuzzy direction feature; in the decoding branch, the fuzzy direction features corresponding to the last downsampled degraded image pair are upsampled; the upsampled features are spliced and fused with the fuzzy direction features corresponding to the last downsampled degraded image pair, and the spliced and fused features are upsampled; the spliced and fused operations are repeated until the fuzzy direction features corresponding to the input degraded image pairs are spliced and fused, and the output fused restored image and the corresponding spectrum diagram are obtained; Each group of degraded image pairs has a corresponding coordinate aggregation module; in each coordinate aggregation module, the input degraded image pairs are split according to the types of images to form a degraded image group and a spectrum graph group; for the degraded image group, two consecutive convolution operations are performed on the degraded image group to obtain no less than 2 spatial features, and all spatial features are spliced and fused to obtain spatial feature weights; the spatial feature weights are multiplied by the corresponding elements of the degraded image group to obtain spatial coordinate features; the operation on the spectrum graph group is consistent with the operation on the degraded image group to obtain frequency domain coordinate features; the spatial coordinate features are spliced with the frequency domain coordinate features to obtain the coordinate aggregation features; Each coordinate aggregation feature has a corresponding fuzzy direction attention module; in each fuzzy direction attention module, the input coordinate aggregation feature is split into a single feature map; after convolution operation is performed on each feature map, the feature maps corresponding to the spatial coordinate features are summarized, and the summarized features are convolved to obtain the fuzzy direction spatial feature weights; the feature maps corresponding to the frequency domain coordinate features are summarized, and the summarized features are convolved to obtain the fuzzy direction frequency domain feature weights; the fuzzy direction spatial feature weights are multiplied by the corresponding elements of the spatial coordinate features to obtain spatial fuzzy features; the fuzzy direction frequency domain feature weights are multiplied by the corresponding elements of the frequency domain coordinate features to obtain frequency domain fuzzy features; the spatial fuzzy features and the frequency domain fuzzy features are concatenated to obtain fuzzy direction features.
2. The advanced imaging system variable direction resolution image intelligent fusion restoration system according to claim 1, characterized in that: Each pair of degraded images includes a degraded image and a frequency spectrum corresponding to the degraded image.
3. The advanced imaging system variable direction resolution image intelligent fusion restoration system according to claim 1, characterized in that: In the decoding branch, when the up-sampled features are spliced and fused with the fuzzy direction features corresponding to the degraded image pair of the last down-sampled, the spatial fuzzy features of the two features are spliced and fused accordingly, and the frequency domain fuzzy features of the two features are spliced and fused accordingly.
4. An intelligent fusion restoration method for variable-direction resolution images of an advanced imaging system, characterized in that: The specific steps include: S1: Acquire image data, preprocess the image data, obtain degraded image pairs, and form a degraded data set; S2: Construct an advanced imaging system variable direction resolution image intelligent fusion restoration system as described in any one of claims 1 to 3; S3: inputting the degraded data set into the fusion restoration system in step S2, and training the fusion restoration system to obtain a trained fusion restoration model; S4: Restoring the image to be restored using the fused restoration model obtained in step S3.
5. The method for intelligent fusion restoration of images with variable directional resolution in an advanced imaging system according to claim 4, characterized in that: The preprocessing in step S1 specifically includes the following steps: S11: cropping the image data; S12: obtaining a corresponding spectrogram according to the cropped image data, and using the cropped image data and the corresponding spectrogram as a true image pair; S13: Convert the cropped image data into the first image intensity distribution collected by the MAPF imaging system: S14: converting the first image intensity distribution into a second image intensity distribution at the image plane of the area array detector in the MAPF imaging system; and obtaining a degraded image according to the second image intensity distribution by the following formula: in, represents the second image intensity distribution, represents the sampling function of the area array detector, represents noise, represents the degraded image; S15: obtaining a frequency spectrum corresponding to the degraded image according to the degraded image; the degraded image and the corresponding frequency spectrum constitute the degraded image pair; S16: Repeat steps S11 to S15 to obtain real image pairs and degraded image pairs corresponding to different image data, and combine all degraded image pairs and all real image pairs to form the degraded data set.
6. The method for intelligent fusion restoration of images with variable directional resolution in an advanced imaging system according to claim 5, characterized in that: In the step S13, the first image intensity distribution is obtained by the following formula: in, Represents the cropped image data, represents the point spread function of the MAPF imaging system, represents the first image intensity distribution, and t represents the shooting time of the MAPF imaging system.
