Infrared light field image super-resolution method and device based on intra-frequency correlation

Through the infrared light field image super-resolution method based on in-frequency correlation, using the interaction of units within and between frequency and triple attention mechanism, the infrared light field image super-resolution method in the prior art is solved during the reconstruction of parallax structure in the disparity structure. The oversmoothing and artifact problems achieve clearer structural reconstruction and non-uniform noise suppression effects.

CN119090719AActive Publication Date: 2024-12-06WUHAN UNIV
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411059953.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-06
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

The existing infrared light field image super-resolution method ignores angle information when reconstructing the parallax structure, resulting in problems of oversmoothing and artifacts, and cannot effectively decouple non-uniform noise and structure.

Method used

Using the infrared light field image super-resolution method based on in-frequency correlation, frequency features are extracted through the IRLF-FSR network, cascading feature interactions within and between frequency and between frequency units, and learning phases using in-frequency units and inter-frequency units. The dependence of neighbor pixels and content correlations are combined with the triple attention mechanism to generate high-resolution images.

Benefits of technology

A clearer structural reconstruction is achieved, reducing artifacts, and is better than other methods in reconstructing parallax structures and suppressing non-uniform noise.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090719A_ABST
    Figure CN119090719A_ABST
Patent Text Reader

Abstract

The invention relates to an image processing technology, in particular to an infrared light field image super-resolution method and device based on intra-frequency correlation, and the method proposes that in an IRLF FSR network, SAI is decomposed into high-frequency and low-frequency components so as to decouple non-uniform noise and structure; and the correlation between the high-frequency component and the low-frequency component is learned through an intra-frequency unit and an inter-frequency unit. In-frequency units model the dependency of adjacent pixels to roughly assist in reconstructing the structure and suppress non-uniform noise, while inter-frequency units are used to learn content dependencies between two frequency components to assist in reconstructing finely authorized structures and suppress residual noise. In the two units, a triple attention mechanism is embedded so as to make full use of space, angle and frequency information, and spatial angle, spatial frequency and angular frequency correlation are jointly modeled through self-attention. According to the method, the problems of over-smoothness and artifacts caused by non-uniform noise and structural coupling are solved, the structure is clearer, and the artifacts are fewer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an infrared light field image super-resolution method and device based on intra-frequency correlation. Background Art

[0002] An infrared light field (IRLF) device consisting of multiple infrared cameras captures sub-aperture infrared images (SAI) in one shot, which record the intensity of light emitted from different positions and along different angles. The spatial information (i.e., contextual information within a sub-aperture image) and angular information (i.e., parallax information between sub-aperture images) of IRLF sub-aperture images have broad prospects in various industrial applications, including 3D reconstruction, deocclusion, and depth estimation. However, the resolution of sub-aperture images is limited by the infrared imaging sensor, which has a low resolution, hindering the application of infrared light field devices and their images.

[0003] In order to reconstruct a high-resolution light field image from a low-resolution light field image, the intuitive approach is to super-resolve each SAI individually, which is the well-known single infrared image super-resolution (IRSR) method. IRSR methods tend to interpolate adjacent pixels or learn end-to-end mappings to restore high-resolution images, but can only utilize the spatial information within each SAI and ignore the angular information between SAIs. Therefore, they are ineffective in reconstructing the parallax structure. Another more reasonable approach is light field image super-resolution (LFSR), which super-resolves all SAIs simultaneously. Existing LFSR methods mainly focus on the visible band, referred to as VILF-SR. VILF-SR methods include traditional methods and learning-based methods. Learning-based VILF-SR methods have been shown to outperform traditional methods. In the learning-based VILF-SR method, techniques such as 4D convolution on SAI, cascaded 2D convolution on stacked SAI, dilated 2D convolution on macropixels, and deformable convolution between center view and side view are used to exploit spatial and angular information. Some VILF-SR methods can only reconstruct over-smoothed edges, while others can reconstruct slightly clear edges but show artifacts in the transition area between foreground and background. However, VILF-SR methods directly extract features from the image space without decomposing non-uniform noise and structure, causing the structure to be mistakenly regarded as non-uniform noise and suppressed, while the non-uniform noise is mistakenly identified as structure and restored. Summary of the invention

[0004] The present invention provides an infrared light field image super-resolution method and device based on intra-frequency correlation, which are used to solve the defects of over-smoothing and artifacts caused by non-uniform noise and structural coupling in the prior art, and realize decoupling non-uniform noise and structure.

[0005] The present invention provides an infrared light field image super-resolution method based on intra-frequency correlation, comprising: an IRLF-FSR network is used to obtain the LR image of the IRLF imaging system; As input, frequency feature extraction, cascade intra-frequency and inter-frequency feature interaction and feature reconstruction are performed in sequence to generate output ; The steps are as follows: Extract frequency features; perform cascade convolution on each frequency component to extract frequency features; Perform cascaded intra-frequency and inter-frequency feature interactions; after extracting frequency features, model the dependency of adjacent pixels in each component, learn the content correlation between the two components, and obtain information within and between frequency components as the basis for structural reconstruction and non-uniform noise suppression; Perform feature reconstruction; use the output features of the last intra-frequency interaction group for reconstruction, perform bicubic interpolation upsampling on SAI, add the reconstruction result to the bicubic interpolation result to generate the final SR image , ,in, R For the image set, A is the LR image, is the spatial resolution; is the number of rows of the infrared image array, For each row of the infrared image array, the sub-aperture image The number of α is the pixel super-resolution coefficient; Using loss function The network is trained and optimized using the ADAM optimizer.

[0006] According to an infrared light field image super-resolution method based on intra-frequency correlation provided by the present invention, the cascade convolution of each frequency component to extract frequency features includes: 1.1. Decompose SAI using 5×5 convolution kernel; Perform a 5×5 convolution on the input SAI to obtain the low-frequency component, and subtract the input SAI from the obtained low-frequency component to obtain the corresponding high-frequency component; 1.2. Capturing high-frequency features from each frequency component through ResBlock and low frequency characteristics , ResBlock sequentially performs 3×3 convolution, leaky ReLU activation, and 3×3 convolution after the input, and combines the result with the input; High frequency features The expression is:

[0007] Low frequency characteristics The expression is:

[0008] in represents a 5×5 convolution, ResBlock; and The size is ,in, is the angular resolution, is the horizontal coordinate of the angle space model, is the ordinate of the angle space model; Represents the channel dimension.

[0009] According to an infrared light field image super-resolution method based on intra-frequency correlation provided by the present invention, the cascading intra-frequency and inter-frequency feature interactions includes: 2.1. Assume a pair of intra-frequency and inter-frequency units to model the dependency and content correlation of adjacent pixels respectively; pack a pair of intra-frequency and inter-frequency units into a group, namely, intra-frequency and inter-frequency feature interaction group, to obtain information within and between frequency components, and cascade L groups, L=5; 2.2. High-frequency features of input and low frequency characteristics It is processed by the intra-frequency unit; it is input to the inter-frequency unit to model the content correlation between different frequency components; the output features of the inter-frequency unit are sent to the next group, and the expression is as follows:

[0010] in, and Respectively represent the input and output frequency characteristics of the lth group; and Respectively represent the operation of intra-frequency unit and inter-frequency unit; 2.3. Both the intra-frequency and inter-frequency units include the transformers BasicTrans and ResBlock; residual connections are added to obtain output features; set X Represents the input features, within the unit The formula is:

[0011] in, Denotes BasicTrnas, BasicTrans mines information from the input features, including the inherent spatial and angular information of IRLF, as well as the implicit frequency information behind the frequency decoy position; BasicTrans replaces MHSA with triple attention TA, calculates and fuses HU , HC andUC Attention in different dimensions; jointly mining spatial, angular and frequency information through TA; 2.4. The structure of BasicTrans is: given the input features, a linear layer is applied to generate query Q , Keywords K Sum V ; Then perform layer norm to normalize the query Q and keywords K ; Set the value V , Query Q and keywords K Input TA to model attention; perform multi-layer perception and layer norm along the channel dimension, combining its correlation; set X Represents the input features, and the process of BasicTrans is expressed as:

[0012] in, , , and Represents linear layer operations, layer norms, multi-layer perception based on channel dimension, and triple attention; 2.5. TA Simulate spatial angle attention, spatial frequency attention and angular frequency attention in parallel; respectively HU , HC and UC right SAA , SFA and AFA Modeling; TA is expressed as:

[0013] in, Indicates a connection operation; , and They represent SAA, SFA and AFA respectively; exist SAA , SFA and AFA Enter the same Q , K and V ;enter SAA of Q , K and V Reshaped into Q ′∈ R V W×HU×C , K ′∈ R V W×C×HU , V ′∈RV W×HU×C ;enter SFA of Q , K and V Reshaped into Q ′∈ R V W×HU×U , K ′∈ R V W×C×HC , V ′∈ R V W×HC×U ;enter AFA of Q , K and V Reshaped into Q ′∈ R V W×UC×H , K ′∈ R V W×H×UC , V ′∈ R V W×UC×H ; Through matrix multiplication and softmax function along HU Dimensional attention; V ′ to obtain the output features; perform a reverse reshape operation on the output features to keep the same shape as the input features; The expression of spatial angle attention is as follows:

[0014] The expression of spatial frequency attention is as follows:

[0015] The expression of angular frequency attention (AFA) is as follows:

[0016] in, and represents reshape and softmax function; 2.5. After the intra-frequency unit learns the dependencies of neighboring pixels, the inter-frequency unit instead models the content correlation; the inter-frequency unit uses features from two frequency components as a query key pair; the expression of the inter-frequency unit is,

[0017] in, X 1 represents the input feature of a frequency component, X 2 represents the characteristics of another frequency component; Represents BasicTrans in inter-frequency units, is the inter-frequency unit; Along W V , W C and V C Dimension calculation TA Attention in inter-frequency units; covering all spatial and angular dimensions by combining intra-frequency units and inter-frequency units.

[0018] According to an infrared light field image super-resolution method based on intra-frequency correlation provided by the present invention, the feature reconstruction comprises: The feature reconstruction expression is:

[0019] in represents the upsampling block, Represents the interpolation operation; the upsampling block includes sequential 1×1 convolution, PixelShuffle method and 1×1 convolution; sequential 1×1 convolution is used to increase the channel to , the PixelShuffle method is used to reshape features from channels to spatial dimensions, that is, from arrive ; 1×1 convolution is used to compress the channel dimension of the final output.

[0020] According to the infrared light field image super-resolution method based on intra-frequency correlation provided by the present invention, the loss function is used Training the network and optimizing it with the ADAM optimizer involves: set up , , , the learning rate is initialized with 2e-4 and halved after every 20 iterations; the number of cascade groups, batch size, and epoch are set to 5, 8, and 60, respectively; the training data is augmented by random flipping and rotation; the model is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

[0021] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an infrared light field image super-resolution method based on intra-frequency correlation as described above is implemented.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the infrared light field image super-resolution method based on intra-frequency correlation as described in any one of the above is implemented.

[0023] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned infrared light field image super-resolution methods based on intra-frequency correlation.

[0024] Compared with the prior art, the present invention has the following beneficial effects: (1) This paper proposes a SR method for IRLF, which aims to learn the correlation between frequency components to help reconstruct the structure and suppress non-uniform noise. It is a method that focuses on improving IRLF SAI.

[0025] (2) The intra-frequency unit and inter-frequency unit of the present invention are used to learn the correlation within the frequency component and between the frequency components respectively. The former aims to learn the correlation between adjacent pixels in each frequency component, while the latter aims to model the correlation between the contents of the components.

[0026] (3) The triple attention mechanism is used to jointly utilize spatial, angular and frequency information. The correlation of SAI is modeled along the spatial angle, spatial frequency and angular frequency dimensions through self-attention. The proposed method is superior to other methods, with clearer structure and fewer artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a structural block diagram of an infrared light field image super-resolution method based on intra-frequency correlation provided by an embodiment of the present invention; Figure 2 This is the SR result on the ×4 validation set provided by the embodiment of the present invention; Figure 3 It is the ×4 SR result of all methods on the real-world test set provided by the embodiments of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] This embodiment proposes an infrared light field image super-resolution method based on intra-frequency correlation. This method aims to solve the problems of over-smoothing and artifacts caused by non-uniform noise and structural coupling. This method designs an IRLF-FSR network, in which SAI is decomposed into high-frequency and low-frequency components to decouple non-uniform noise and structure. The network includes a pair of units to learn the correlation of SAI. The intra-frequency unit is designed to simulate the dependency of adjacent pixels to roughly help reconstruct the structure and suppress non-uniform noise; the inter-frequency unit is used to learn the content correlation between the two frequency components to help reconstruct the fine authorized structure and suppress residual noise. In both units, a triple attention mechanism is designed and embedded to make full use of spatial, angular and frequency information, and to jointly model spatial angle, spatial frequency and angular frequency correlation through self-attention.

[0031] This embodiment is implemented by the following technical solution: an infrared light field image super-resolution method based on intra-frequency correlation. In an IRLF imaging system based on a camera array, given a set of low-resolution (LR) SAIs, represented as , ,in SR of IRLF images aims to reconstruct their high-resolution (HR) objects. , .in, is the amplification factor, represents the angular resolution, Indicates the spatial resolution.

[0032] This embodiment designs the overall architecture of the IRLF-FSR network, such as Figure 1 As shown. The IRLF-FSR network is based on the LR image of the IRLF imaging system. As input, frequency feature extraction, cascade intra-frequency-inter-frequency feature interaction and feature reconstruction are performed in sequence to generate the final output .

[0033] 1. Frequency feature extraction: Feature extraction aims to map the input from the image space to the feature space. The main motivation is to decompose SAI into high-frequency and low-frequency components to decouple non-uniform noise and structure, so we perform cascade convolution on each frequency component to extract frequency features.

[0034] 1.1. Use 5×5 convolution kernel to decompose SAI.

[0035] First, a 5×5 convolution is performed on the input SAI to obtain the low-frequency component, and then the input SAI is subtracted from the obtained low-frequency component to obtain the corresponding high-frequency component.

[0036] 1.2. Capture high-frequency features from each frequency component through ResBlock and low frequency characteristics , ResBlock sequentially performs 3×3 convolution after the input, Leaky ReLU activation and 3×3 convolution, and then combines the result with the input. High-frequency features and low frequency characteristics The expression is as follows:

[0037] in represents a 5×5 convolution, ResBlock. and The size is ,in, is the angular resolution, is the horizontal coordinate of the angle space model, is the ordinate of the angle space model; Represents the channel dimension.

[0038] 2. Cascade intra-frequency and inter-frequency feature interactions: After frequency feature extraction, the focus is on modeling the dependency of adjacent pixels within each component and learning the content correlation between the two components, thereby mining the information within and between frequency components to help structure reconstruction and non-uniform noise suppression.

[0039] 2.1 A pair of intra-frequency and inter-frequency units are designed to model the dependency and content relevance of adjacent pixels, respectively. These two units are then packaged into a group (i.e., intra-frequency-inter-frequency feature interaction group) to jointly mine the information within and between frequency components. Finally, L groups (L=5) are cascaded to make full use of this information.

[0040] 2.2 High-frequency features of input and low frequency characteristics First, they are processed individually by the intra-frequency units. Then, they are input into the inter-frequency units to model the content correlation between different frequency components. Finally, the output features of the inter-frequency units are sent to the next group, and the expression is as follows:

[0041] in, and Represent the input and output frequency characteristics of the lth group respectively. and Represent the operations of intra-frequency unit and inter-frequency unit respectively.

[0042] 2.3 Both intra-frequency and inter-frequency units consist of transformers BasicTrans and ResBlock. Residual connections are added to obtain output features. Let X denote the input features, and For example, the formula is:

[0043] in, Denotes BasicTrnas. BasicTrans aims to mine information from the input features. This information includes not only the intrinsic spatial and angular information of IRLF, but also the implicit frequency information behind the frequency decoy position. BasicTrans replaces MHSA with triple attention (TA), which computes and fuses HU , HC and UC In this way, TA can jointly mine spatial, angular and frequency information.

[0044] 2.4. The structure of BasicTrans is: given the input features, first apply a linear layer to generate query Q , Keywords K Sum V ; Then perform layer norm to normalize Q and K Next, V , Q and K TA is input to model the attention. Afterwards, multi-layer perception (channel MLP

[56] ) and layer norm are performed along the channel dimension to further incorporate these correlations. X Represents the input features, the entire process of BasicTrans can be expressed as:

[0045] in, , , and Represents linear layer operations, layer norms, MLP-based multi-layer perception, and triple attention.

[0046] 2.5、 TA In parallel, spatial angle attention is simulated ( SAA )、spatial-frequency attention( SFA ) and angular frequency attention ( AFA ).along HUV or W UV dimensional attention map, which transforms the space (i.e. H and W ) and angle (i.e. U and V) dimensions, thereby utilizing both spatial and angular information. To include more dimensions, we HU , HC and UC right SAA , SFA and AFA In this way, TA Spatial, angular and frequency information can be jointly utilized. T A It can be expressed as:

[0047] in, Represents a join operation. , and express SAA , SFA and AFA .

[0048] exist SAA , SFA and AFA Enter the same Q , K and V ;enter SAA of Q , K and V Reshaped into Q ′∈ R V W×HU×C , K ′∈ R V W×C×HU , V ′∈R V W×HU×C ;enter SFA of Q , K and V Reshaped into Q ′∈ R V W×HU×U , K ′∈ R V W×C×HC , V ′∈ R V W×HC×U ;enter AFA of Q , K and V Reshaped into Q ′∈ R V W×UC×H , K ′∈ R V W×H×UC , V ′∈ RV W×UC×H .

[0049] Then, the attention along the HU dimension is obtained by matrix multiplication and softmax function. The attention score is applied on V′ to obtain the output features. Finally, the output features are reshaped inversely to keep the same shape as the input features.

[0050] The expression of spatial angle attention is as follows:

[0051] The expression of spatial frequency attention is as follows:

[0052] The expression of angular frequency attention (AFA) is as follows:

[0053] in, and represents the reshape and softmax function.

[0054] 2.5. After the intra-frequency unit has learned the dependencies of neighboring pixels, the inter-frequency unit is designed to model content correlation. Unlike the dependencies of neighboring pixels, content correlation refers to the relationship between two frequency components. Therefore, unlike the intra-frequency unit that uses features from one frequency component as the query key pair in its transformer, the inter-frequency unit uses features from two frequency components as the query key pair. The expression of the inter-frequency unit is expressed as follows:

[0055] Among them, X1 represents the input feature of one frequency component, and X2 represents the feature of another frequency component. Represents BasicTrans in inter-frequency units, It is the frequency unit.

[0056] To cover the dimensions not covered by the intra-frequency unit (i.e., W and V), we compute the attention of TA in the inter-frequency unit along the dimensions of WV, WC, and VC. By combining the intra-frequency unit and the inter-frequency unit, all spatial and angular dimensions can be covered, thus making full use of the spatial, angular, and frequency information.

[0057] 3. Feature reconstruction: Feature reconstruction aims to generate the final result from the feature space. Use the output features of the last frequency interaction group to reconstruct; then perform bicubic interpolation upsampling on SAI, and finally, add the reconstruction result to the bicubic interpolation result to generate the final SR image. , The reconstructed expression is as follows:

[0058] in Represents an upsampling block. Represents an interpolation operation. The upsampling block includes a sequential 1×1 convolution, a PixelShuffle method, and a 1×1 convolution. The former convolution is used to increase the channel to , PixelShuffle is used to reshape features from channels to spatial dimensions, that is, from arrive The latter convolution is used to compress the channel dimension of the final output.

[0059] 4. Loss Function .use L 1 loss function trains the network and optimizes it with the ADAM optimizer (default setting , , ). During training, we initialize the learning rate with 2e-4 and halve it after every 20 epochs. The number of cascade groups, batch size, and epoch are set to 5, 8, and 60, respectively. The training data is augmented by random flipping and rotation. The model is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

[0060] Example We first introduce the experimental data and implementation details. Then, we qualitatively and quantitatively compare the results of IRLF FSRNet with existing state-of-the-art methods.

[0061] The compared methods are trained and validated using synthetic datasets, and tested on real datasets captured by the camera array-based IRLF device. The publicly available IRLF dataset IRLF-WHU is used to create the training and validation datasets, which consists of 1132 scenes with a resolution of 3×3×384×280. We split the IRLF WHU dataset into training and validation sets in a ratio of 7:3. For both training and validation datasets, HR images are generated by randomly cropping 16 patches (128×128 for the ×4 task) from each IRLF data. The corresponding LR images are obtained by bicubic downsampling and adding non-uniform noise. The non-uniform noise level in the training and validation sets is slightly different. Specifically, in the training set, the non-uniform noise level is randomly set in the range of [0.01, 0.05]. In the validation set, the noise levels are set to 0, 0.01, 0.03, and 0.05, respectively. Real-world test data are collected using 9 uncooled infrared cameras arranged in a 3x3 grid. A total of 58 sets of data with significant non-uniformity are collected for testing.

[0062] Comparison with state-of-the-art SR methods: IRLF SRNet is the first SR network for IRLF images, therefore, the comparison is limited to the most comparable VILF-SR and IRSR methods. In particular, two IRSR methods, namely TherISuRNet and ChaSNet, and five VILF-SR methods, namely LF-ATO, LF IINet, DistgSSR, DPT, and EPIT, are selected. For a fair comparison, all the above methods are retrained on the same dataset.

[0063] Table 1 Comparison results of PSNR and SSIM of various methods on the validation set

[0064] 1) Quantitative comparison: Table 1 lists the comparison results of PSNR and SSIM on the validation set. It can be observed that IRLFSRNet achieves the best performance on both ×2 and ×4 tasks. Specifically, IRLF SRNet is at least 2.85 dB (0.0231) higher than the IRSR methods (i.e., TherISuRNet and ChaSNet) in terms of average PSNR (SSIM) for ×2 tasks, and at least 4.20 dB (0.0629) higher in terms of average peak signal-to-noise ratio (SSIM) for ×4 tasks.

[0065] Compared with the VILF-SR method, the average PSNR (SSIM) performance is at least 0.66 dB higher in the x2 task (0.0054) and at least 0.51 dB higher in the x4 task (0.0069), demonstrating that the proposed method outperforms existing SR methods. Note that at noise levels of 0.01, 0.03, and 0.05, the proposed method is 0.45 dB, 0.50 dB, and 0.64 dB stronger than EPIT, respectively, indicating that the proposed method is more robust.

[0066] 2) Qualitative comparison: qualitative results of ×4; The SR task on the validation set is as follows Figure 2 As shown in Figure 2. For the more challenging ×4 task, compared with the VILF-SR method, our method can maintain clearer edges and fewer artifacts. This further confirms the robustness of our method. In the experiment, we also compared the ×4 SR results of all methods on the real-world test set, as shown in Figure 2. Figure 3As shown. It can be observed that in real scenes, the results of the VILF-SR method cannot reconstruct certain structures (as shown in the red box), while the IRSR method and the method of the present invention can still reconstruct them. This shows that it is crucial to exploit the information within the frequency components. In addition, the VILF-SR and IRSR methods present artifacts around the transition area between the edge and the background (as shown in the green box), indicating that it is crucial to mine the information of the intra-frequency components.

[0067] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute the infrared light field image super-resolution method based on intra-frequency correlation.

[0068] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0069] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the infrared light field image super-resolution method based on intra-frequency correlation provided by the above methods.

[0070] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the infrared light field image super-resolution method based on intra-frequency correlation provided by the above methods.

[0071] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A super-resolution method for infrared light field images based on intra-frequency correlation, It is characterized in that include: The IRLF-FSR network is based on the LR image of the IRLF imaging system. As input, frequency feature extraction, cascade intra-frequency and inter-frequency feature interaction and feature reconstruction are performed in sequence to generate output ; The steps are as follows: Extract frequency features; Perform cascade convolution on each frequency component to extract frequency features; Perform cascaded intra-frequency and inter-frequency feature interactions; after extracting frequency features, model the dependency of adjacent pixels in each component, learn the content correlation between the two components, and obtain information within and between frequency components as the basis for structural reconstruction and non-uniform noise suppression; Perform feature reconstruction; use the output features of the last intra-frequency interaction group for reconstruction, perform bicubic interpolation upsampling on SAI, add the reconstruction result to the bicubic interpolation result to generate the final SR image , ,in, R For the image set, A is the LR image, is the spatial resolution; is the number of rows of the infrared image array, For each row of the infrared image array, the sub-aperture image The number of α is the pixel super-resolution coefficient; Using loss function The network is trained and optimized using the ADAM optimizer.

2. According to the infrared light field image super-resolution method based on intra-frequency correlation according to claim 1, It is characterized in that The performing cascade convolution on each frequency component to extract frequency features comprises: 1.

1. Decompose SAI using 5×5 convolution kernel; Perform a 5×5 convolution on the input SAI to obtain the low-frequency component, and subtract the input SAI from the obtained low-frequency component to obtain the corresponding high-frequency component; 1.

2. Capturing high-frequency features from each frequency component through ResBlock and low frequency characteristics , ResBlock sequentially performs 3×3 convolution, leaky ReLU activation, and 3×3 convolution after the input, and combines the result with the input; High frequency features The expression is: Low frequency characteristics The expression is: in represents a 5×5 convolution, ResBlock; and The size is ,in, is the angular resolution, is the horizontal coordinate of the angle space model, is the ordinate of the angle space model; Represents the channel dimension.

3. According to the infrared light field image super-resolution method based on intra-frequency correlation according to claim 1, It is characterized in that The cascading intra-frequency and inter-frequency feature interactions comprises: 2.

1. Assume a pair of intra-frequency and inter-frequency units to model the dependency and content correlation of adjacent pixels respectively; pack a pair of intra-frequency and inter-frequency units into a group, namely, intra-frequency and inter-frequency feature interaction group, to obtain information within and between frequency components, and cascade L groups, L=5; 2.

2. High-frequency features of input and low frequency characteristics It is processed by the intra-frequency unit; it is input to the inter-frequency unit to model the content correlation between different frequency components; the output features of the inter-frequency unit are sent to the next group, and the expression is as follows: in, and Respectively represent the input and output frequency characteristics of the lth group; and Respectively represent the operation of intra-frequency unit and inter-frequency unit; 2.

3. Both the intra-frequency and inter-frequency units include the transformers BasicTrans and ResBlock; residual connections are added to obtain output features; set X Represents the input features, within the unit The formula is: in, Denotes BasicTrnas, BasicTrans mines information from the input features, including the inherent spatial and angular information of IRLF, as well as the implicit frequency information behind the frequency decoy position; BasicTrans replaces MHSA with triple attention TA, calculates and fuses HU , HC and UC Attention in different dimensions; jointly mining spatial, angular and frequency information through TA; 2.

4. The structure of BasicTrans is: given the input features, a linear layer is applied to generate query Q , Keywords K Sum V ; Then perform layer norm to normalize the query Q and keywords K ; Set the value V , Query Q and keywords K enter TA Model attention; perform multi-layer perception and layer norm along the channel dimension, combining their correlations; set X Represents the input features, and the process of BasicTrans is expressed as: in, , , and Represents linear layer operations, layer norms, multi-layer perception based on channel dimension, and triple attention; 2.

5. TA Simulate spatial angle attention, spatial frequency attention and angular frequency attention in parallel; respectively HU , HC and UC right SAA , SFA and AFA Modeling; TA It is expressed as: in, Indicates a connection operation; , and Respectively SAA , SFA and AFA ; exist SAA , SFA and AFA Enter the same Q , K and V ;enter SAA of Q , K and V Reshaped into Q ′∈ R V W×HU×C , K ′∈ R V W×C×HU , V ′∈R V W×HU×C ;enter SFA of Q , K and V Reshaped into Q ′∈ R V W×HU×U , K ′∈ R V W×C×HC , V ′∈ R V W ×HC×U ;enter AFA of Q , K and V Reshaped into Q ′∈ R V W×UC×H , K ′∈ R V W×H×UC , V ′∈ R V W×UC×H ; Attention along the HU dimension via matrix multiplication and softmax function; V ′ to obtain the output features; perform a reverse reshape operation on the output features to keep the same shape as the input features; The expression of spatial angle attention is as follows: The expression of spatial frequency attention is as follows: The expression of angular frequency attention (AFA) is as follows: in, and represents reshape and softmax function; 2.

5. After the intra-frequency unit learns the dependencies of neighboring pixels, the inter-frequency unit instead models the content correlation; the inter-frequency unit uses features from two frequency components as a query key pair; the expression of the inter-frequency unit is, in, X 1 represents the input feature of a frequency component, X 2 Indicates the characteristics of another frequency component; Represents BasicTrans in inter-frequency units, is the inter-frequency unit; Along WV , WC and VC The dimension of TA is used to calculate the attention of TA in the inter-frequency unit; by merging the intra-frequency unit and the inter-frequency unit, all spatial and angular dimensions are covered.

4. According to the infrared light field image super-resolution method based on intra-frequency correlation according to claim 1, It is characterized in that The feature reconstruction comprises: The feature reconstruction expression is: in represents the upsampling block, Represents the interpolation operation; the upsampling block includes sequential 1×1 convolution, PixelShuffle method and 1×1 convolution; sequential 1×1 convolution is used to increase the channel to , the PixelShuffle method is used to reshape features from channels to spatial dimensions, that is, from arrive ; 1×1 convolution is used to compress the channel dimension of the final output.

5. According to the infrared light field image super-resolution method based on intra-frequency correlation according to claim 1, It is characterized in that The loss function used Training the network and optimizing it with the ADAM optimizer involves: set up , , , the learning rate is initialized with 2e-4 and halved after every 20 iterations; the number of cascade groups, batch size, and epoch are set to 5, 8, and 60, respectively; the training data is augmented by random flipping and rotation; the model is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the infrared light field image super-resolution method based on intra-frequency correlation as described in any one of claims 1 to 5 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the infrared light field image super-resolution method based on intra-frequency correlation as claimed in any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the infrared light field image super-resolution method based on intra-frequency correlation as claimed in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Light field image super-resolution reconstruction method based on frequency domain analysis and deep learning

    CN113139898A

  • Multi-infrared image enhancement method based on spatial parallax prior network

    CN113628125A

  • Infrared image super-resolution method based on light field

    CN116523740A

  • Image restoration method for optical synthetic aperture imaging (OSAI)

    CN116664426A

  • Remote sensing image super-resolution reconstruction method based on multi-path fusion and attention

    CN117173022A