Infrared light field image super-resolution method and device based on intra-frequency correlation

Abstract

The present application relates to image processing technology, specifically relates to a kind of infrared light field image super-resolution method and equipment based on frequency correlation, the method proposes IRLF FSR network, SAI is decomposed into high frequency and low frequency component, to decouple non-uniform noise and structure;Again, the correlation between these high, low frequency components is learned by frequency unit and frequency unit.The dependence of adjacent pixels is modeled in frequency unit, to help roughly reconstruct structure and suppress non-uniform noise, while frequency unit is used to learn the content correlation between two frequency components, help reconstruct fine authorized structure and suppress residual noise.In two units, triple attention mechanism is embedded to make full use of spatial, angle and frequency information, and the spatial angle, spatial frequency and angular frequency correlation are jointly modeled by self-attention.The method solves the problem of over-smoothing and artifact caused by non-uniform noise and structure coupling, and the structure is clearer and the artifact is less.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an infrared light field image super-resolution method and device based on intra-frequency correlation. BACKGROUND

[0002] An infrared light field (IRLF) device composed of multiple infrared cameras captures sub-aperture infrared images (SAIs) in one shot, which record the intensity of light rays emitted from different positions and along different angles. The spatial information (i.e. contextual information within one SAI) and angular information (i.e. disparity information between SAIs) of the sub-aperture images of the IRLF have broad prospects in various industrial applications, including 3D reconstruction, de-occlusion and depth estimation. However, the resolution of the sub-aperture images is limited by the infrared imaging sensor, and the resolution is not high, which hinders the application of the infrared light field device and its images.

[0003] In order to reconstruct a high-resolution light field image from a low-resolution light field image, the intuitive method is to super-resolve each SAI separately, which is a well-known single infrared image super-resolution (IRSR) method. The IRSR method tends to interpolate adjacent pixels or learn an end-to-end mapping to recover a high-resolution image, but can only utilize the spatial information within each SAI, while ignoring the angular information between SAIs. Therefore, they are ineffective in reconstructing the disparity structure. Another more reasonable method is light field image super-resolution (LFSR), which simultaneously super-resolves all SAIs. Existing LFSR methods mainly focus on the visible band, referred to as VILF-SR. VILF-SR methods include traditional methods and learning-based methods, and learning-based VILF-SR methods have been proven to be superior to traditional methods. In learning-based VILF-SR methods, techniques such as 4D convolution on SAIs, cascaded 2D convolution on stacked SAIs, extended 2D convolution on macro-pixels, and deformable convolution between center and side views are used to utilize spatial and angular information. Some VILF-SR methods can only reconstruct over-smoothed edges, and some methods can reconstruct slightly clear edges, but show artifacts in the transition area between foreground and background. However, VILF-SR methods directly extract features from image space without decomposing non-uniform noise and structure, resulting in the structure being incorrectly considered as non-uniform noise and suppressed, and the non-uniform noise being incorrectly identified as structure and recovered. SUMMARY

[0004] The application provides an infrared light field image super-resolution method and device based on intra-frequency correlation, which solves the defects of over-smoothing and artifacts caused by the coupling of non-uniform noise and structure in the prior art, and realizes the decoupling of non-uniform noise and structure.

[0005] The application provides an infrared light field image super-resolution method based on intra-frequency correlation, comprising: an IRLF-FSR network for LR images of an IRLF imaging system For input, frequency feature extraction, cascaded intra-frequency and inter-frequency feature interaction and feature reconstruction are sequentially performed to generate output ; The steps are as follows:

[0006] Extracting frequency features; performing cascaded convolution on each frequency component to extract frequency features;

[0007] Performing cascaded intra-frequency and inter-frequency feature interaction; after extracting frequency features, the dependency of adjacent pixels in each component is modeled, and the content correlation between two components is learned to obtain intra-frequency component and inter-frequency component information as the basis for structure reconstruction and non-uniform noise suppression;

[0008] Performing feature reconstruction; using the output features of the last intra-frequency interaction group for reconstruction, adding the reconstruction result to the bicubic interpolation result by bicubic interpolation upsampling on SAI to generate the final SR image , , wherein, R is an image set, A is an LR image, is a spatial resolution; is the number of rows of an infrared image array, is the number of sub-aperture images in each row of the infrared image array , α is a pixel super-resolution coefficient;

[0009] The network is trained using a loss function , and is optimized by an ADAM optimizer.

[0010] According to the infrared light field image super-resolution method based on intra-frequency correlation provided by the application, the cascaded convolution on each frequency component to extract frequency features comprises:

[0011] 1.1. decomposing SAI using a 5x5 convolution kernel;

[0012] 5x5 convolution is performed on the input SAI to obtain a low-frequency component, and the corresponding high-frequency component is obtained by subtracting the input SAI from the obtained low-frequency component;

[0013] 1.2. capturing high-frequency features and low-frequency features from each frequency component through a ResBlock, the ResBlock sequentially performs 3x3 convolution, Leaky ReLU activation and 3x3 convolution on the input, and combines the results with the input;

[0014] high-frequency features

[0015]

[0016] low-frequency features

[0017]

[0018] wherein represents a 5x5 convolution, represents a ResBlock; and has a size of wherein is an angular resolution, is a horizontal coordinate of the angular space model, is a vertical coordinate of the angular space model; represents a channel dimension.

[0019] According to the infrared light field image super-resolution method based on intra-frequency correlation provided by the application, the intra-frequency and inter-frequency feature interaction includes:

[0020] 2.1. A pair of intra-frequency and inter-frequency units are provided to model the dependency and content correlation of adjacent pixels, respectively; a pair of intra-frequency and inter-frequency units are packaged into a group, i.e. an intra-frequency and inter-frequency feature interaction group, to obtain intra-frequency component and inter-frequency component information, and L groups are cascaded, L=5;

[0021] 2.2. The input high-frequency features and low-frequency features are processed by the intra-frequency unit; they are input into the inter-frequency unit to model the content correlation between different frequency components; the output features of the inter-frequency unit are sent to the next group, and the expression is as follows:

[0022]

[0023] wherein, and represent the input and output frequency features of the lth group, respectively; and represent the operations of the intra-frequency unit and the inter-frequency unit, respectively;

[0024] 2.3. The intra-frequency and inter-frequency units both include a converter BasicTrans and a ResBlock; a residual connection is added to obtain the output features; let X represent the input features, and the intra-frequency unit The formula is:

[0025] ​​

[0026] where, BasicTrans extracts information from input features, including inherent spatial and angular information of IRLF, and also includes implicit frequency information after the position of frequency decoy; BasicTrans replaces MHSA with triple attention TA, which calculates and fuses attention of different dimensions including HU 、 HC and UC ; spatial, angular and frequency information are jointly mined through TA;

[0027] 2.4. The structure of BasicTrans is: given input features, a linear layer is applied to generate query Q , key K and value V respectively; then layer norm is performed to normalize query Q and key K ; value V , query Q and key K are input into TA to model attention; multi-layer perception and layer norm along the channel dimension are combined with their relevance; let X represent input features, the process of BasicTrans is represented as:

[0028]

[0029] where, 、 、 and represent linear layer operation, layer norm, multi-layer perception based on channel dimension and triple attention;

[0030] 2.5. TA Parallelly simulate spatial angular attention, spatial frequency attention and angular frequency attention; model HU 、 HC and UC on SAA 、 SFA and AFA respectively; TA is represented as:

[0031]

[0032] where, represents connection operation; 、 and represent SAA, SFA and AFA respectively;

[0033] InSAA 、 SFA and AFA input the same Q , K and V ; input SAA 's Q , K and V are reshaped to Q '∈ R V W×HU×C , K '∈ R V W×C×HU , V '∈R V W×HU×C ; input SFA 's Q , K and V are reshaped to Q '∈ R V W×HU×U , K '∈ R V W×C×HC , V '∈ R V W×HC×U ; input AFA 's Q , K and V are reshaped to Q '∈ R V W×UC×H , K '∈ R V W×H×UC , V '∈ R V W×UC×H ;

[0034] Attention along the HU dimension by matrix multiplication and softmax function; apply the attention score on V ' to get the output feature; perform reverse reshaping operation on the output feature to keep the same shape as the input feature;

[0035] The expression of spatial angle attention is as follows:

[0036]

[0037] The expression of spatial frequency attention is as follows:

[0038]

[0039] The expression of angle frequency attention (AFA) is as follows:

[0040]

[0041] wherein, and denote reshaping and softmax function;

[0042] 2.5. After the intra-frequency unit learns the dependency of neighboring pixels, the inter-frequency unit is changed to model the content correlation; the inter-frequency unit uses features from two frequency components as query key pairs; the expression of the inter-frequency unit is,

[0043]

[0044] wherein, X 1 denotes the input features of one frequency component, X 2 denotes the features of another frequency component; denotes BasicTrans in the inter-frequency unit, is the inter-frequency unit;

[0045] is calculated along the dimensions of W V , W C and V C TA attention in the inter-frequency unit; by merging the intra-frequency unit and the inter-frequency unit, all spatial and angular dimensions are covered.

[0046] According to the infrared light field image super-resolution method based on intra-frequency correlation provided by the application, the feature reconstruction comprises:

[0047] The feature reconstruction expression is,

[0048]

[0049] wherein denotes an up-sampling block, denotes an interpolation operation; the up-sampling block comprises a sequential 1x1 convolution, a PixelShuffle method and a 1x1 convolution; the sequential 1x1 convolution is used to increase the channel to , the PixelShuffle method is used to reshape the features from the channel to the spatial dimension, i.e., from to ; the 1x1 convolution is used to compress the channel dimension of the final output.

[0050] According to the infrared light field image super-resolution method based on intra-frequency correlation provided by the application, the network is trained using a loss function and is optimized by an ADAM optimizer, which comprises:

[0051] Let , ​, The learning rate is initialized with 2e-4 and halved after every 20 iteration cycles; the number of cascaded groups, batch size, and epoch are set to 5, 8, and 60, respectively; the training data is enhanced by random flipping and rotation; the model is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

[0052] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of the above description when executing the program.

[0053] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of the above description.

[0054] The application further provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of the above description.

[0055] Compared with the prior art, the application has the following beneficial effects:

[0056] (1) The application proposes a SR method for IRLF, which aims to learn the correlation between frequency components to help reconstruct the structure and suppress non-uniform noise, and is a method focusing on improving the IRLF SAI.

[0057] (2) The intra-frequency unit and the inter-frequency unit are respectively used to learn the correlation within and between frequency components. The former aims to learn the correlation between adjacent pixels within each frequency component, while the latter aims to model the correlation between the content of the components.

[0058] (3) The triple attention mechanism is used to jointly utilize spatial, angular, and frequency information. The correlation of SAI is modeled along the spatial angle, spatial frequency, and angular frequency dimensions through self-attention. The method of the application is superior to other methods, and has a clearer structure and fewer artifacts. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 is a structural block diagram of an infrared light field image super-resolution method based on intra-frequency correlation provided by an embodiment of the present application;

[0061] Figure 2 is the SR result of the x4 verification set provided by an embodiment of the present application;

[0062] Figure 3 is the x4 SR result of all methods on the real-world test set provided by an embodiment of the present application;

[0063] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0065] The present embodiment proposes an infrared light field image super-resolution method based on intra-frequency correlation. The method aims to solve the problems of oversmoothing and artifacts caused by non-uniform noise and structure coupling. The method designs an IRLF-FSR network, in which SAI is decomposed into high-frequency and low-frequency components to decouple non-uniform noise and structure. The network includes a pair of units to learn the correlation of SAI. Through the design of intra-frequency units to simulate the dependence of adjacent pixels, to roughly help reconstruct the structure and suppress non-uniform noise; through the inter-frequency unit to learn the content correlation between the two frequency components, to help reconstruct the fine authorized structure and suppress residual noise. In these two units, a triple attention mechanism is designed and embedded to fully utilize the spatial, angular and frequency information, and to jointly model the spatial angle, spatial frequency and angular frequency correlation through self-attention.

[0066] The present embodiment is implemented by the following technical solution, an infrared light field image super-resolution method based on intra-frequency correlation, in a camera array-based IRLF imaging system, given a set of low-resolution (LR) SAIs, denoted as , wherein . The SR of the IRLF image aims to reconstruct its high-resolution (HR) object , . wherein, is a magnification factor, denotes the angular resolution, denotes the spatial resolution.

[0067] The overall architecture of the IRLF-FSR network is designed as shown in Figure 1 The IRLF-FSR network takes the LR image of the IRLF imaging system as input, and sequentially performs frequency feature extraction, cascaded intra-frequency and inter-frequency feature interaction and feature reconstruction to generate the final output .

[0068] 1. Frequency feature extraction: Feature extraction aims to map the input from image space to feature space. The main motivation is to decompose the SAI into high-frequency and low-frequency components to decouple non-uniform noise and structure, so we perform cascaded convolution on each frequency component to extract frequency features.

[0069] 1.1. Use a 5x5 convolution kernel to decompose the SAI.

[0070] First, perform 5x5 convolution on the input SAI to obtain the low-frequency component, and then subtract the input SAI with the obtained low-frequency component to obtain the corresponding high-frequency component.

[0071] 1.2. Capture high-frequency features and low-frequency features from each frequency component through a ResBlock, which sequentially performs 3x3 convolution on the input, Leaky ReLU activation and 3x3 convolution, and then combines the results with the input. The expressions of high-frequency features and low-frequency features are as follows:

[0072]

[0073] where denotes 5x5 convolution, denotes ResBlock. and have dimensions , where is the angular resolution, is the horizontal coordinate of the angular space model, is the vertical coordinate of the angular space model; denotes the channel dimension.

[0074] 2. Cascaded intra-frequency and inter-frequency feature interaction: After frequency feature extraction, the focus is on modeling the dependency of adjacent pixels within each component and learning the content correlation between two components, thereby mining the intra-frequency component and inter-frequency component information to help structure reconstruction and non-uniform noise suppression.

[0075] ​2.1 A pair of intra- and inter-frequency units are designed to model the dependency of neighboring pixels and the content correlation, respectively. Then, these two units are packed into a group (i.e., intra- inter-frequency feature interaction group) to jointly exploit the information within and across frequency components. Finally, L groups (L = 5) are concatenated to fully utilize these information.

[0076] 2.2 High-frequency features and low-frequency features are first processed by the intra-frequency unit individually. Then, they are input into the inter-frequency unit to model the content correlation between different frequency components. Finally, the output features of the inter-frequency unit are sent to the next group, which is expressed as follows:

[0077]

[0078] where and denote the input and output frequency features of the l-th group, respectively. and denote the operations of the intra- and inter-frequency units, respectively.

[0079] 2.3 Both the intra- and inter-frequency units are composed of the transformer BasicTrans and ResBlock. The residual connection is added to obtain the output features. Let X denote the input features, and take the intra-frequency unit as an example, the formula is as follows:

[0080]

[0081] where denotes BasicTrans. BasicTrans aims to exploit the information from the input features. This information not only includes the inherent spatial and angular information of IRLF, but also the implicit frequency information after the frequency lure position. BasicTrans replaces MHSA with triple attention (TA), which calculates and fuses the attention of different dimensions including HU , HC and UC . In this way, TA can jointly exploit spatial, angular, and frequency information.

[0082] 2.4 The structure of BasicTrans is as follows: given the input features, a linear layer is first applied to generate the query Q , key K and value V respectively; then, layer norm is performed to normalize Q and K . Next, V , Q and K ​Input TA to model attention. Then, multilayer perception (channel MLP

[56] ) and layer specification are performed along the channel dimension to further incorporate these correlations. Let X Representing the input features, the entire process of BasicTrans can be represented as follows:

[0083]

[0084] in, , , and It represents linear layer operations, layer canonicalization, MLP-based multilayer perception, and triple attention.

[0085] 2.5 TA Parallel simulation of spatial angle attention ( SAA Spatial frequency attention ( SFA ) and angular frequency attention ( AFA ).along HUV or W UV A dimensional attention graph that divides space (i.e., H and W ) and angle (i.e. U and V ) Dimensions are combined to utilize spatial and angular information. To include more dimensions, we separately... HU , HC and UC right SAA , SFA and AFA Modeling is performed. In this way... TA Spatial, angular, and frequency information can be used in combination. T A It can be represented as:

[0086]

[0087] in, This indicates a join operation. , and express SAA , SFA and AFA .

[0088] exist SAA , SFA and AFA The same input Q , K and V ;enter SAA of Q , K and V Reshaped intoQ ′∈ R V W×HU×C , K ′∈ R V W×C×HU , V ′∈R V W×HU×C ; input SFA of Q , K and V are reshaped to Q ′∈ R V W×HU×U , K ′∈ R V W×C×HC , V ′∈ R V W×HC×U ; input AFA of Q , K and V are reshaped to Q ′∈ R V W×UC×H , K ′∈ R V W×H×UC , V ′∈ R V W×UC×H .

[0089] Attention along the HU dimension is again performed through matrix multiplication and a softmax function. The attention scores are applied on V' to obtain the output features. Finally, an inverse reshaping operation is performed on the output features to maintain the same shape as the input features.

[0090] The expression of spatial angle attention is as follows:

[0091]

[0092] The expression of spatial frequency attention is as follows:

[0093]

[0094] The expression of angle-frequency attention (AFA) is as follows:

[0095]

[0096] where and denote reshaping and softmax function.

[0097] 2.5、After the intra-frequency unit learns the dependency of neighboring pixels, the inter-frequency unit is designed to model the content correlation. Unlike the dependency of neighboring pixels, the content correlation refers to the relationship between two frequency components. Therefore, unlike the intra-frequency unit that uses features from one frequency component as the query key pair in its transformer, the inter-frequency unit uses features from two frequency components as the query key pair. The expression of the inter-frequency unit is represented as follows:

[0098]

[0099] where X1 represents the input features of one frequency component and X2 represents the features of another frequency component. represents the BasicTrans in the inter-frequency unit, is the inter-frequency unit.

[0100] To cover the dimensions that the intra-frequency unit does not involve (i.e., W and V), we calculate the attention of TA in the inter-frequency unit along the dimensions of W V, W C, and V C. By merging the intra-frequency unit and the inter-frequency unit, all spatial and angular dimensions can be covered, so as to fully utilize the spatial, angular, and frequency information.

[0101] 3、Feature reconstruction: Feature reconstruction aims to generate the final result from the feature space. The output features of the last intra-frequency interaction group are used for reconstruction; then, the SAI is up-sampled by bicubic interpolation, and finally, the reconstructed result is added to the bicubic interpolation result to generate the final SR image , . The expression of the reconstruction is as follows:

[0102]

[0103] where represents the up-sampling block. represents the interpolation operation. The up-sampling block includes a sequential 1x1 convolution, a PixelShuffle method, and a 1x1 convolution. The former convolution is used to increase the channel to , the PixelShuffle is used to reshape the features from the channel to the spatial dimension, i.e., from to . The latter convolution is used to compress the channel dimension of the final output.

[0104] 4、Loss function . The network is trained using the 1 loss function and is optimized by the ADAM optimizer (the default setting L , , , ). During training, we initialize the learning rate with 2e-4 and halve it after every 20 iteration cycles. The number of cascaded groups, batch size, and epoch are set to 5, 8, and 60, respectively. The training data is augmented by random flipping and rotation. The model is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

[0105] Embodiments

[0106] First, the experimental data and implementation details are introduced. Then, the results of IRLF FSRNet are compared with the state-of-the-art methods both qualitatively and quantitatively.

[0107] The comparative methods are trained and validated using a synthetic dataset and tested on a real dataset captured by a camera array-based IRLF device. The publicly available IRLF dataset IRLF-WHU is used to create the training and validation datasets, which consists of 1132 scenes with a resolution of 3x3x384x280. We split the IRLF WHU dataset into training and validation sets with a 7:3 ratio. For both training and validation datasets, the HR images are generated by randomly cropping 16 patches (128x128 for x4 task) from each IRLF data. The corresponding LR images are obtained by bicubic down-sampling and adding non-uniform noise. The non-uniform noise level is slightly different in the training and validation sets. Specifically, in the training set, the non-uniform noise level is randomly set in the range of [0.01, 0.05]. In the validation set, the noise levels are set to 0, 0.01, 0.03, and 0.05, respectively. Real-world test data is collected using nine non-cooled infrared cameras arranged in a 3x3 grid. A total of 58 groups of data with significant non-uniformity are collected for testing.

[0108] Comparison with state-of-the-art SR methods:

[0109] IRLF SRNet is the first SR network for IRLF images, therefore, the comparison is limited to the most comparable VILF-SR and IRSR methods. In particular, two IRSR methods, TherISuRNet and ChaSNet, and five VILF-SR methods, LF-ATO, LF IINet, DistgSSR, DPT, and EPIT, are selected. To make a fair comparison, all the above methods are retrained on the same dataset.

[0110] Table 1. Comparison results of PSNR and SSIM on the validation set for each method

[0111]

[0112] 1) Quantitative comparison: Table 1 lists the comparison results of PSNR and SSIM on the validation set. It can be observed that IRLF SRNet achieves the best performance on both x2 and x4 tasks. Specifically, IRLF SRNet is at least 2.85 dB (0.0231) higher in average PSNR (SSIM) on the x2 task and 4.20 dB (0.0629) higher in average PSNR (SSIM) on the x4 task than IRSR methods (i.e., TherISuRNet and ChaSNet).

[0113] Compared with VILF-SR method, the average PSNR (SSIM) performance is at least 0.66 dB (0.0054) higher in x2 task and 0.51 dB (0.0069) higher in x4 task, which demonstrates that the present method is superior to existing SR methods. It is noted that the present method is 0.45 dB, 0.50 dB and 0.64 dB stronger than EPIT at noise levels of 0.01, 0.03 and 0.05, respectively, which indicates that the present method is more robust.

[0114] 2) Qualitative comparison: Qualitative results of x4;

[0115] SR tasks on the validation set are shown in Figure 2 For the more challenging x4 task, the present method can maintain sharper edges and fewer artifacts compared with VILF-SR method. This further confirms the robustness of the present method. In the experiment, x4 SR results of all methods on the real-world test set are also compared, as shown in Figure 3 It can be observed that in real scenes, the results of VILF-SR method cannot reconstruct certain structures (as shown in the red box), while IRSR method and the present method can still reconstruct them. This indicates that it is crucial to exploit the information within the frequency components. In addition, VILF-SR and IRSR methods exhibit artifacts around the transition area between edges and background (as shown in the green box), indicating that it is crucial to exploit the information of the intra-frequency component.

[0116] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown in Figure 4 The electronic device can include a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory complete mutual communication through the communications bus. The processor can invoke the logic instructions in the memory to execute the infrared light field image super-resolution method based on intra-frequency correlation.

[0117] In addition, the logic instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0118] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the infrared light field image super-resolution method based on intra-frequency correlation provided by the above-mentioned methods.

[0119] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the infrared light field image super-resolution method based on intra-frequency correlation provided by the above-mentioned methods.

[0120] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0121] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary general hardware platform, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or the parts that contribute to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intra-frequency correlation based infrared light field image super-resolution method, characterized in that, comprising: an IRLF-FSR network taking LR images of an IRLF imaging system as input, sequentially performing frequency feature extraction, cascaded intra- and inter-frequency feature interaction and feature reconstruction to generate output. ; the steps are as follows: extracting frequency features; performing cascaded convolution on each frequency component to extract frequency features, the steps are as follows: 1.

1. Decompose SAI using 5x5 convolution kernel; perform 5x5 convolution on input SAI to obtain low-frequency components, subtract input SAI with obtained low-frequency components to obtain corresponding high-frequency components; 1.

2. Capture high-frequency features from each frequency component by ResBlock and low-frequency features , ResBlock performs 3x3 convolution, Leaky ReLU activation and 3x3 convolution in order on the input, and combines the result with the input; High frequency features Expression: Low frequency features Expression: wherein denotes a 5x5 convolution, denotes a ResBlock; and has a size of wherein, is the angular resolution, is the horizontal coordinate of the angular space model, is the vertical coordinate of the angular space model; denotes the channel dimension; perform cascaded intra-frequency and inter-frequency feature interaction; After extracting frequency features, model the dependency of adjacent pixels within each component and learn the content correlation between two components to obtain intra-frequency component and inter-frequency component information as the basis for structure reconstruction and non-uniform noise suppression; performing cascaded intra-frequency and inter-frequency feature interaction includes: 2.

1. Set a pair of intra-frequency and inter-frequency units to model the dependency of adjacent pixels and content correlation respectively; package a pair of intra-frequency and inter-frequency units into a group, i.e. intra-frequency and inter-frequency feature interaction group, to obtain intra-frequency component and inter-frequency component information, and cascade L groups, L=5; 2.

2. High-frequency features of the input and low-frequency features are processed by intra-frequency units; their outputs are fed into inter-frequency units that model content correlations between different frequency components; the output features of the inter-frequency units are sent to the next group, expressed as follows: wherein, and represent the input and output frequency characteristics of the lth group, respectively; and represent the intra-frequency and inter-frequency cell operations, respectively. 2.

3. Both intra- and inter-frequency units include a converter BasicTrans and ResBlock; add a residual connection to obtain the output feature; set X represents the input feature, the intra- and inter-frequency units The formula is: wherein, BasicTrans, which mines information from input features, including inherent spatial and angular information of IRLF, and also includes implicit frequency information after the frequency decoy position; BasicTrans replaces MHSA with triple attention TA, which calculates and fuses attention of different dimensions including HU , HC and UC ; jointly mines spatial, angular and frequency information through TA; performing feature reconstruction; using the output features of the last intra-frequency interaction group to perform reconstruction, up-sampling the reconstruction result by bi-cubic interpolation on the SAI, adding the up-sampled result to the bi-cubic interpolation result, and generating a final SR image , wherein, R is a set of images, A is a LR image, is a spatial resolution; is a number of rows of an infrared image array, is a number of sub-aperture images in each row of the infrared image array , α is a pixel super-resolution coefficient; Using a loss function The network is trained and optimized by an ADAM optimizer.

2. The method of claim 1, wherein the method is an intra- frequency correlation based infrared light field image super-resolution method. The performing of cascaded intra-frequency and inter-frequency feature interaction further includes: 2.

4. The structure of BasicTrans is as follows: given input features, a linear layer is applied to generate queries respectively. Q Keywords K Sum V Then perform level norm normalization to normalize the query. Q and keywords K ; value V Query Q and keywords K enter TA Model attention; perform multi-layer perception and layer specification along the channel dimension, and combine their correlations; set X Representing the input features, the BasicTrans process is expressed as follows: wherein, , , and represent linear layer operations, layer specifications, channel-dimension-based multi-layer perception, and triple attention; 2.

5. TA Parallel modeling of spatial angular attention, spatial frequency attention, and angular frequency attention; along HU , HC and UC modeling of SAA , SFA and AFA ; TA is represented as: wherein represents a connection operation; , and respectively represent SAA , SFA and AFA ; In SAA , SFA and AFA the same Q , K and V are entered; the SAA Q , K and V are reshaped to Q ′∈ R V W×HU×C , K ′∈ R V W×C×HU , V ′∈R V W×HU×C ; the SFA Q , K and V are entered are reshaped to Q ′∈ R V W×HU×U , K ′∈ R V W×C×HC , V ′∈ R V W ×HC×U ; the AFA Q , K and V are entered are reshaped to Q ′∈ R V W×UC×H , K ′∈ R V W×H×UC , V ′∈ R V W×UC×H ;​​​ Attention along the HU dimension via matrix multiplication and the softmax function, in V Attention scores are applied to obtain output features; inverse shaping is performed on the output features to maintain the same shape as the input features. The expression of spatial angle attention is as follows: The expression of spatial frequency attention is as follows: ​ The expression of angular frequency attention (AFA) is as follows: wherein, and denotes the reshaping and softmax function; 2.

6. After the intra-frequency unit learns the dependency of adjacent pixels, the inter-frequency unit is changed to model the content correlation; the inter-frequency unit uses features from two frequency components as query key pairs; the expression of the inter-frequency unit is, wherein X 1 represents an input feature of one frequency component, X 2 represents a feature of another frequency component; BasicTrans represents a BasicTrans in inter-frequency unit, is an inter-frequency unit; Along W V , W C and V C dimensions to compute the TA interest in inter-frequency cells; by merging intra- and inter- frequency cells, all spatial and angular dimensions are covered.

3. The method of claim 1, wherein the method is based on intra- frequency correlation. The performing of feature reconstruction includes: The expression of feature reconstruction is, wherein denotes an up-sampling block, denotes an interpolation operation; the up-sampling block comprises sequential 1x1 convolutions, a PixelShuffle method and a 1x1 convolution; the sequential 1x1 convolutions are used to increase the channels to , the PixelShuffle method is used to reshape the features from channels to spatial dimensions, i.e. from to ; the 1x1 convolution is used to compress the channel dimension of the final output.

4. The method of claim 1, wherein, The use of the loss function Training the network and optimizing it through an ADAM optimizer includes: Set , , , the learning rate is initialized with 2e-4 and halved after every 20 iteration cycles; the number of cascaded groups, batch size, and epochs are set to 5, 8, and 60, respectively; the training data is augmented by random flipping and rotation; the network is implemented in PyTorch and trained using an NVIDIA RTX 4090 GPU.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of claims 1-4 when executing the program.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of claims 1-4 when executed by the processor.

7. A computer program product comprising a computer program, characterized in that, The computer program implements the infrared light field image super-resolution method based on intra-frequency correlation according to any one of claims 1-4 when executed by the processor.

Images