Multi-scale super-resolution reconstruction method based on intensity guidance

Abstract

The application discloses a kind of multi-scale super-resolution reconstruction methods based on intensity guidance.The method is aimed at the low spatial resolution of depth information collected by most low-cost depth devices, and designs a guided residual depth super-resolution network combining local information and global information based on the depth network UNet3+ with encoder-decoder architecture. The network uses intensity map and multi-scale feature map extracted from depth histogram to guide the upsampling of depth map. And based on attention mechanism, the dependence between intensity map and depth map is established to improve the utilization rate of feature information. The application can successfully upsample the low spatial resolution depth map by 4 times, and prove the robustness of the algorithm by adding noise and verify the generalization ability of the proposed algorithm on simulated data and real data.

Description

Technical Field

[0001] This invention belongs to the field of depth super-resolution reconstruction technology, and is a multi-scale super-resolution reconstruction method based on intensity guidance. Background Technology

[0002] In recent years, the rapid development of depth imaging devices capable of capturing target depth information has made acquiring depth information much easier, and its use has become increasingly common in our lives. Furthermore, due to the miniaturization of depth devices, they have even been incorporated into mobile phones and tablets for augmented reality applications. However, the small size of most low-cost depth devices remains a major reason for the low spatial resolution of depth information.

[0003] Compared to traditional cameras, depth maps captured by low-cost depth sensors typically suffer from low spatial resolution and are subject to degradation due to missing values, noise, and quantization issues. This limits their potential applications. Image Super Resolution (SR) technology offers a new approach to overcoming this limitation.

[0004] Generally speaking, the super-resolution problem of images is an ill-conditioned inverse problem, which has no fixed solution. This is because there are always multiple high-resolution (HR) images corresponding to a single low-resolution (LR) image. Some classic SR methods have laid the foundation for the development of this technique: R. Keys (1. R. Keys, "Cubic convolution interpolation for digital image processing," IEEE Transactions on Acoustics, Speech, and Signal Processing 29, 1153-1160 (1981)) proposed Cubic convolution interpolation as a new discrete data reconstruction sampling technique based on prediction. Sun et al. (2. J. Sun, J. Sun, ZBXu, and HYShum, "Image super-resolution using gradient profile prior," Proc Cvpr IEEE, 2471-+ (2008)) used an edge-based method to constrain the image gradient when estimating HR images using gradient profiles learned from a large number of natural images. Xiong et al. (3. ZWXiong, XYSun, and F.Wu, "Robust Web Image / Video Super-Resolution," IEEE T Image Process 19, 2017-2028 (2010).) proposed a method combining adaptive regularization and learning-based super-resolution to improve image resolution and perceptual quality. Yang et al. (4. JCYang, J.Wright, T.S.Huang, and Y.Ma, "Image Super-Resolution Via Sparse Representation," IEEE T Image Process 19, 2861-2873 (2010).) proved, based on sparse representation methods, that image patch representations are sparse linear combinations of appropriately selected overcomplete dictionaries. Thanks to the excellent performance of CNNs on color image SR, SRCNN demonstrated its potential.More and more researchers are beginning to try to apply it to depth images and have made progress: Hui et al. (5.TWHui,CCLoy,andX.O.Tang, "Depth Map Super-Resolution by Deep Multi-Scale Guidance",ComputerVision-Eccv 2016,Pt Iii 9907,353-369(2016).) proposed an intensity-guided depth super-resolution network MSG-Net, which uses HR intensity features to solve the ambiguity problem in the upsampling process of depth maps at different levels. Guo et al. (6.CLGuo,CYLi,JCGuo,RMCong,HZFu,andP.Han, "Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution,"IEEE T ImageProcess28,2545-2557(2019).) proposed a super-resolution network called DepthSR-Net built on the residual U-Net deep network architecture. The final HR map is obtained by adding the learned residual map to the interpolated HR image. Ruget et al. (7.A.Ruget,S.McLaughlin,RKHenderson,I.Gyongy,A.Halimi,andJ.Leach, "Robust super-resolution depth imaging via a multi-feature fusion deep network,"Opt Express 29,11917-11937(2021).) utilizes first and second depth maps extracted from the original histogram to achieve robust super-resolution imaging in low-photon-level, low-signal-to-background-ratio environments through multi-scale feature fusion. Lindell's method employs a linear SPAD detector, requiring scene scanning and imaging, often taking several hours to acquire scene depth information, failing to meet the increasingly widespread real-time requirements. Sun improved Lindell's network, resulting in better depth estimation, but did not address the long acquisition time for scene information. Ruget's method breaks down the feature extraction process into a preprocessing stage, which reduces the computational burden of network training, but the preprocessing stage is lengthy and time-consuming. Summary of the Invention

[0005] The purpose of this invention is to propose an intensity-guided multi-scale super-resolution reconstruction method to achieve robust estimation of scene depth images and overcome the shortcomings of low spatial resolution in low-cost depth devices.

[0006] The technical solution for achieving this invention is: a deep super-resolution reconstruction method based on intensity-guided multi-scale feature fusion, the specific steps of which are as follows:

[0007] Step 1: Construct a depth map imaging system to obtain 3D depth information and 2D intensity images from the SPAD array and camera, respectively; project the 3D depth information into 2D, and use checkerboard feature detection and feature transformation to align it with the 2D intensity image; perform signal-to-noise ratio improvement processing on the registered 3D depth data; calculate the peak center mass of the data cube using the maximum likelihood estimation method to obtain the 2D depth map, and enlarge it by 4x, 2x, and 1x in the spatial dimension using the Kronecker product, respectively, as the input depth map, the first depth feature map, and the second depth feature map of the network; reduce the 3D data by 2x and 4x in the spatial dimension using the downsampling method, respectively, and obtain the 2D third and fourth depth feature maps by calculating the center mass;

[0008] Step 2: Construct and train a deep super-resolution network based on intensity-guided multi-scale feature fusion. The super-resolution network includes a U-shaped network, an intensity feature extraction branch, and a denoising branch. One side of the U-shaped network is the encoder, consisting of encoder layers 1, 2, 3, 4, and 5 from top to bottom; the other side is the decoder, consisting of 5 layers, consisting of decoder layers 1, 2, 3, 4, and 5 from top to bottom. The denoising branch includes a multi-scale convolution module and a non-local convolution module. The multi-scale convolution module is used to extract multi-scale deep feature information from the stitched encoder output image; the non-local convolution module is used to fuse depth feature information at different scales; the intensity feature extraction branch is used to extract corresponding intensity features with the same size as the depth features.

[0009] Step 3: Input the preprocessed depth data obtained in Step 1 into the trained model to obtain the reconstructed final depth map; wherein, the preprocessed input depth map is input into the first layer of the encoder of the network, and the first, second, third, fourth and fifth layers of the encoder are respectively input into the first depth feature map, the second depth feature map, the third depth feature map and the fourth depth feature map; the data output from the denoising branch is input into each layer of the decoder, and the intensity features extracted from the intensity feature extraction branch at the corresponding scale are inserted into each layer.

[0010] Preferably, the two-dimensional depth map D is calculated using the maximum likelihood estimation method for depth.i,j The specific formula is as follows:

[0011]

[0012] Among them, H i,j,t For noisy photon counting, D max D is the location of the maximum peak value. max =arg max(H i,j,t The median value of each pixel (i,j) in the real data is used as the background level of the scene, i.e., B. i,j,t =median(H i,j,t ) is H i,j,t Background level; D max -2σ and D max Between +2σ Integrate over a time period, where σ is a set value.

[0013] Preferably, the specific process of extracting intensity features by the intensity feature extraction branch is as follows:

[0014] (1) Input the intensity map I into the intensity feature extraction branch;

[0015] (2) The intensity feature extraction branch performs convolution and pooling on the intensity map I to obtain four intensity features I'1, I'2, I'3 and I'4 with different sizes and different numbers of channels, as shown below;

[0016] I'1=σ[conv 3×3 (I)+b]

[0017] pool1_f = max pool 2×2 (I'1)

[0018] I'2=σ[conv 3×3 (pool1_f)+b]

[0019] pool2_f = max pool 2×2 (I'2)

[0020] I'3=σ[conv 3×3 (pool2_f)+b]

[0021] pool3_f = max pool 2×2 (I'3)

[0022] I'4=σ[conv 3×3 (pool3_f)+b]

[0023] Where I is the input, σ represents the ReLU activation function, b represents the bias, and maxpool...2×2 conv represents a pooling operation that is reduced by a factor of 2. 3×3 This refers to a 3×3 convolution operation.

[0024] Preferably, the specific process by which the multi-scale convolution module extracts multi-scale deep feature information from the stitched encoder output image is as follows:

[0025] (1) The data after each layer of the encoder convolution is concatenated together using pooling, changing the number of channels, or upsampling to obtain the input data K;

[0026] (2) The multi-scale convolution module uses a 3×3 convolutional layer to convolve the input data K, resulting in the image K1 = σ(conv). 3×3 (K)+b); where K is the input, σ represents the ReLU activation function, and conv 3×3 This refers to a 3×3 convolution operation, where b represents the bias.

[0027] (3) Process image K1 using 1×3 and 3×1 convolutional layers respectively to obtain image K2 = σ(conv 1×3 (σ(conv 3×3 (K)+b))+b) and the image K3=σ(conv) 3×1 (σ(conv 3×3 (K)+b))+b); where, conv 1×3 and conv 3×1 These represent 1×3 and 3×1 convolution operations, respectively.

[0028] (4) After merging K1, K2 and K3, the output image K' is obtained:

[0029] K' = concat(K1, K2, K3)

[0030] Where K' is the output image, and concat(·) represents the merging operation.

[0031] Preferably, the nonlocal convolutional module adopts a cascaded residual dense structure, and the specific process of feature fusion is as follows:

[0032] (1) The output K' of the multi-scale convolution module is used as the input of the non-local fusion module. The shape of K' is (B,H,W,C), where B is the batch size, H is the height of the feature, W is the width of the feature, and C is the number of channels.

[0033] (2) The nonlocal convolution module uses a 1×1 convolutional layer and a reshape operation to extract features at different scales from the input data, denoted as theta = reshape[conv]. 1×1 (K')]、phi=reshape[conv 1×1(K')] and g = reshape[conv 1×1 [(K')], where the shapes of theta, phi, and g are (B,H*W,C / / 2), (B,C / / 2,H*W), and (B,H*W,C / / 2), respectively, where W is the input, reshape[·] represents the operation to change the shape, and conv 1×1 (·) represents a 1×1 convolution operation;

[0034] (3) Multiply the two features theta and phi element by element to obtain The shape of f is (B, H*W, H*W). Multiplying f and g element-wise yields... The shape of y is (B, H*W, C / / 2), where, Indicates element-wise multiplication;

[0035] (4) Use the convolutional attention mechanism module CBAM to perform adaptive feature optimization of y in space and channel to obtain z, the shape of z is (B,H,W,C);

[0036] (5) Add z to the input K' element by element to get the final result K.

[0037] Preferably, the final output of the nonlocal convolution module is as follows:

[0038]

[0039] Where K” is the output, softmax(·) is the normalized exponential function, and CBAM(·) represents the attention mechanism CBAM module. This indicates multiplication and addition between elements.

[0040] Compared with existing technologies, this invention has significant advantages: (1) It simply and efficiently utilizes SPAD array data to decompose the extraction of multi-scale features from the network into preprocessing, and then provides it directly to the network during the training phase, reducing the training time of the network and the computational pressure on the computer. (2) It solves the misalignment problem between histogram and intensity image. (3) Compared with other networks, the proposed network performs well in upsampling, especially for data with low signal-to-noise ratio and low photon level.

[0041] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0042] Figure 1 This is a system diagram of the present invention used for data acquisition.

[0043] Figure 2 This refers to the specific registration process.

[0044] Figure 3 This is a diagram of a network architecture.

[0045] Figure 4 This is a schematic diagram of the structure of a multi-scale convolution module.

[0046] Figure 5 This is a schematic diagram of the structure of a nonlocal convolution module.

[0047] Figure 6 The method involves using a trained model to reconstruct the target depth map on simulation data with different noise levels.

[0048] Figure 7 It involves using a trained model on real data to reconstruct the target depth map. Detailed Implementation

[0049] A super-resolution reconstruction method based on intensity-guided multi-scale feature fusion is proposed, suitable for post-processing of 3D data acquired by single-photon lidar to obtain magnified, high-quality depth maps. The specific steps are as follows:

[0050] Step 1: Build a depth map imaging system to obtain 3D low-resolution depth information and 2D high-resolution intensity images from the SPAD array and camera, respectively;

[0051] like Figure 1 As shown, the depth map imaging system uses a coaxial arrangement, which can provide a complete overlap between the receiver's field of view and the laser beam. The depth map imaging system is divided into two modules: (1) Light source module. The light source is a semiconductor pulsed laser with a working wavelength of 635nm and a repetition frequency of 20MHz. The signal generator generates a synchronization signal to trigger the pulsed laser and the SPAD array. The laser receives the pulse signal from the signal generator to generate laser light. The point light source is shaped into a surface light source by a diffuser, covering the detection range of the array SPAD detector. (2) Detector module. It consists of a SPAD array, a conventional camera, and a bandpass filter. The SPAD array outputs a time-correlated single-photon count histogram of the detected target, and the conventional camera records the time-integrated photon flux of the scene. The specific operation is to obtain three-dimensional low-resolution depth information and two-dimensional high-resolution intensity image from the SPAD array and the camera, respectively. After the data is acquired, the three-dimensional depth information is projected into two dimensions. Through checkerboard feature detection, feature transformation is applied to achieve the alignment of the projected two-dimensional image and the two-dimensional high-resolution intensity image obtained by the camera. The detailed process is as follows. Figure 2 As shown.

[0052] The registered 3D depth data undergoes signal-to-noise ratio (SNR) enhancement processing. Using the maximum likelihood estimation method for depth, a 2D depth map is calculated by determining the center quality of the 3D data peaks. This map is then spatially magnified by a factor of 4, 2, and 1 using the Kronecker product, serving as the network's input depth map, first depth feature map, and second depth feature map. In the spatial dimension, the 3D data is downsampled by a factor of 2 and 4, respectively, and the center quality is calculated to obtain the 2D third and fourth depth feature maps. The specific formula for calculating the 2D depth map using the maximum likelihood estimation method is as follows:

[0053]

[0054] Among them, the noisy photon count H i,j,t Maximum peak position D max It can be made by D max =arg max(H i,j,t This is obtained by taking the median value of each pixel (i,j) from the real data as the background level of the scene, since the noise level of the real scene is unknown. That is, B. i,j,t =median(H i,j,t ) is H i,j,t The background level. We are in D max -2σ and D max Between +2σ The time interval is integrated. Finally, normalization is performed within the range of 0 to 1 to obtain the depth D. i,j .

[0055] Step 2: As Figure 3 As shown, a deep super-resolution network based on intensity-guided multi-scale feature fusion is constructed and trained. The basic framework of this network is a U-shaped network, which is approximately symmetrical in its overall structure. The left side consists of an encoder with five layers (layers 1-5 from top to bottom); the right side consists of a decoder with five layers (layers 1-5 from top to bottom). The network includes a denoising branch and an intensity feature extraction branch. These two branches primarily denoise and upsample the input low-resolution depth image to the required resolution, guided by the high-resolution intensity image. The denoising branch includes multi-scale convolutional modules and non-local convolutional modules.

[0056] The intensity feature extraction branch extracts the corresponding intensity features of the same size as the depth features; the intensity branch consists of convolutional operations and 2×2 max pooling. The number of filters in each convolutional operation is 64, 128, 256, and 512, respectively. The output size of the convolutional operation at each layer of the branch is the intensity feature of the same size as the depth features. These outputs are integrated along the decoder of the network by cascading along the filter dimensions and corresponding sizes. A detailed description follows:

[0057] (1) Use intensity map I as the input to the intensity feature extraction branch;

[0058] (2) Convolution and pooling of I yields four intensity features I'1, I'2, I'3 and I'4 with different sizes and number of channels, as shown below;

[0059] I'1=σ[conv 3×3 (I)+b]

[0060] pool1_f = maxpool 2×2 (I'1)

[0061] I'2=σ[conv 3×3 (pool1_f)+b]

[0062] pool2_f = maxpool 2×2 (I'2)

[0063] I'3=σ[conv 3×3 (pool2_f)+b]

[0064] pool3_f = maxpool 2×2 (I'3)

[0065] I'4=σ[conv 3×3 (pool3_f)+b]

[0066] Where I is the input, σ represents the ReLU activation function, b represents the bias, and maxpool... 2×2 This indicates a pooling operation that is reduced by a factor of 2.

[0067] like Figure 4 As shown, the multi-scale convolution module in the denoising branch is used to extract deep feature information to mine more effective feature information. The specific process of extracting deep feature information is as follows:

[0068] (1) The data after each layer of convolution of the encoder is concatenated together after pooling, changing the number of channels or upsampling to obtain the input data K;

[0069] (2) The multi-scale convolution module first uses a 3×3 convolutional layer to convolve the input data K, resulting in K1 = σ(conv). 3×3 (K)+b); where K is the input, σ represents the ReLU activation function, and conv 3×3 This refers to a 3×3 convolution operation, where b represents the bias.

[0070] (3) Process K1 with 1×3 and 3×1 convolutional layers respectively to obtain K2=σ(conv 1×3 (σ(conv 3×3 (K)+b))+b) and K3=σ(conv) 3×1 (σ(conv 3×3 (K)+b))+b); where, conv 1×3 and conv 3×1 These represent 1×3 and 3×1 convolution operations, respectively.

[0071] (4) Combine and output K1, K2 and K3;

[0072] The entire module can be represented by the following formula:

[0073] K' = concat(K1, K2, K3)

[0074] Here, K' is the output, and concat(·) represents the merge operation.

[0075] like Figure 5 As shown, the nonlocal convolution module is used to fuse feature information at different scales to improve the utilization rate of feature information. The nonlocal convolution module adopts a cascaded residual dense structure, and the specific process of feature fusion is as follows:

[0076] (1) The output K' of the multi-scale convolution module is used as the input of the non-local fusion module. The shape of K' is (B,H,W,C), where B is the batch size, H is the height of the feature, W is the width of the feature, and C is the number of channels.

[0077] (2) The nonlocal convolution module first uses a 1×1 convolutional layer and a reshape operation to extract features of different scales from the input data, denoted as theta = reshape[conv]. 1×1 (K')]、phi=reshape[conv 1×1 (K')] and g = reshape[conv 1×1 [(K')], where the shapes of theta, phi, and g are (B,H*W,C / / 2), (B,C / / 2,H*W), and (B,H*W,C / / 2), respectively, where W is the input, reshape[·] represents the operation to change the shape, and conv 1×1(·) represents a 1×1 convolution operation;

[0078] (3) Multiply the two features theta and phi element by element to obtain The shape of f is (B, H*W, H*W). Multiplying f and g element-wise yields... The shape of y is (B, H*W, C / / 2), where, Indicates element-wise multiplication;

[0079] (4) Use the convolutional attention mechanism module CBAM to perform adaptive feature optimization of y in space and channel to obtain z, the shape of z is (B,H,W,C);

[0080] (5) Add z to the initial input K' element by element to get the final result K”; insert K” into each layer of the corresponding decoder and concatenate it with the intensity feature and the output data of the next layer after upsampling of the current decoder layer.

[0081] The entire module can be represented by the following formula:

[0082]

[0083] Where K” is the output, softmax(·) is the normalized exponential function, and CBAM(·) represents the attention mechanism CBAM module. This indicates multiplication and addition between elements.

[0084] In addition, the network selects ADAM as the optimizer and L1 loss function as the loss function.

[0085] Step 3: Input the preprocessed depth data obtained in Step 1 into the trained model to obtain the reconstructed final depth map; wherein, the preprocessed input depth map is input into the first layer of the encoder of the network, and the first, second, third, fourth and fifth layers of the encoder are respectively input into the first depth feature map, the second depth feature map, the third depth feature map and the fourth depth feature map; the data output from the denoising branch is input into each layer of the decoder, and the intensity features extracted from the intensity feature extraction branch at the corresponding scale are inserted into each layer.

[0086] like Figure 6The figure shows a comparison of the reconstructed target depth maps obtained using nearest neighbor interpolation, the direct intensity fusion method proposed by Lindell et al., and the method of this invention on simulation data. (a) shows the target depth reconstruction maps of each method at a noise level of ppp = 5 / SBR = 0.5. (b) shows the target depth reconstruction maps of each method at a noise level of ppp = 2 / SBR = 0.04. (c) The table presents the numerical representation of the reconstruction performance of each method at two different noise levels, using root mean square error (RMSE) sensitive to outliers. To compare reconstructed images The difference between the actual depth Y on the ground and the actual depth Y is used to quantify the network's performance on simulated data.

[0087] like Figure 7 The image shows a comparison of the reconstructed target depth maps obtained using nearest neighbor interpolation, the intensity direct fusion method proposed by Lindell et al., and the method of this invention on real data. Here, "Captured" refers to the raw data after simple preprocessing.

Claims

1. A multi-scale super-resolution reconstruction method based on intensity guidance, characterized in that, The specific steps are as follows: Step 1: Construct a depth map imaging system to obtain 3D depth information and 2D intensity images from the SPAD array and camera, respectively; project the 3D depth information into 2D, and use checkerboard feature detection and feature transformation to align it with the 2D intensity image; perform signal-to-noise ratio improvement processing on the registered 3D depth data; calculate the peak center mass of the data cube using the maximum likelihood estimation method to obtain the 2D depth map, and enlarge it by 4x, 2x, and 1x in the spatial dimension using the Kronecker product, respectively, as the input depth map, the first depth feature map, and the second depth feature map of the network; reduce the 3D data by 2x and 4x in the spatial dimension using the downsampling method, respectively, and obtain the 2D third and fourth depth feature maps by calculating the center mass; Step 2: Construct and train a deep super-resolution network based on intensity-guided multi-scale feature fusion; the super-resolution network includes a U-shaped network, an intensity feature extraction branch and a denoising branch, one side of the U-shaped network is an encoder, and from top to bottom are encoder layer 1, encoder layer 2, encoder layer 3, encoder layer 4 and encoder layer 5; On the other side is the decoder, which has five layers in total, from top to bottom: decoder layer 1, decoder layer 2, decoder layer 3, decoder layer 4, and decoder layer 5. The denoising branch includes a multi-scale convolution module and a non-local convolution module. The multi-scale convolution module is used to extract multi-scale deep feature information of the stitched encoder output image; the non-local convolution module is used to fuse depth feature information of different scales; the intensity feature extraction branch is used to extract the corresponding intensity feature with the same size as the depth feature. Step 3: Input the preprocessed depth data obtained in Step 1 into the trained model to obtain the reconstructed final depth map; wherein, the preprocessed input depth map is input into the first layer of the encoder of the network, and the first, second, third, fourth and fifth layers of the encoder are respectively input into the first depth feature map, the second depth feature map, the third depth feature map and the fourth depth feature map; the data output from the denoising branch is input into each layer of the decoder, and the intensity features extracted from the intensity feature extraction branch at the corresponding scale are inserted into each layer.

2. The intensity-guided multi-scale super-resolution reconstruction method according to claim 1, characterized in that, The two-dimensional depth map D is calculated using the maximum likelihood estimation method for depth. i,j The specific formula is as follows: Among them, H i,j,t For noisy photon counting, D max D is the location of the maximum peak value. max =arg max(H i,j,t The median value of each pixel (i,j) in the real data is used as the background level of the scene, i.e., B. i,j,t =median(H i,j,t ) is H i,j,t Background level; D max -2σ and D max Between +2σ Integrate over a time period, where σ is a set value.

3. The intensity-guided multi-scale super-resolution reconstruction method according to claim 1, characterized in that, The specific process of extracting intensity features using the intensity feature extraction branch is as follows: (1) Input the intensity map I into the intensity feature extraction branch; (2) The intensity feature extraction branch performs convolution and pooling on the intensity map I to obtain four intensity features I′1, I′2, I′3 and I′4 with different sizes and different numbers of channels, as shown below; I′1=σ[conv 3×3 (I)+b] pool1_f=max pool 2×2 (I′1) I'2=σ[conv 3×3 (pool1_f)+b] pool2_f=max pool 2×2 (I'2) I′3=σ[conv 3×3 (pool2_f)+b] pool3_f=max pool 2×2 (I′3) I'4=σ[conv 3×3 (pool3_f)+b] Where I is the input, σ represents the ReLU activation function, b represents the bias, and max pooling... 2×2 conv represents a pooling operation that is reduced by a factor of 2. 3×3 This refers to a 3×3 convolution operation.

4. The intensity-guided multi-scale super-resolution reconstruction method according to claim 1, characterized in that, The specific process by which the multi-scale convolution module extracts multi-scale deep feature information from the stitched encoder output image is as follows: (1) The data after each layer of the encoder convolution is concatenated together using pooling, changing the number of channels, or upsampling to obtain the input data K; (2) The multi-scale convolution module uses a 3×3 convolutional layer to convolve the input data K, resulting in the image K1 = σ(conv). 3×3 (K)+b); where K is the input, σ represents the ReLU activation function, and conv 3×3 This refers to a 3×3 convolution operation, where b represents the bias. (3) Process image K1 using 1×3 and 3×1 convolutional layers respectively to obtain image K2 = σ(conv 1×3 (σ(conv 3×3 (K)+b))+b) and the image K3=σ(conv) 3×1 (σ(conv 3×3 (K)+b))+b); where, conv 1×3 and conv 3×1 These represent 1×3 and 3×1 convolution operations, respectively. (4) After merging K1, K2 and K3, the output image K' is obtained: K' = concat(K1, K2, K3) Where K' is the output image, and concat(·) represents the merging operation.

5. The intensity-guided multi-scale super-resolution reconstruction method according to claim 2, characterized in that, The nonlocal convolutional module employs a cascaded residual dense structure, and the specific process of feature fusion is as follows: (1) The output K' of the multi-scale convolution module is used as the input of the non-local fusion module. The shape of K' is (B,H,W,C), where B is the batch size, H is the height of the feature, W is the width of the feature, and C is the number of channels. (2) The nonlocal convolution module uses a 1×1 convolutional layer and a reshape operation to extract features at different scales from the input data, denoted as theta = reshape[conv]. 1×1 (K')]、phi=reshape[conv 1×1 (K')] and g = reshape[conv 1×1 [(K')], where the shapes of theta, phi, and g are (B,H*W,C / / 2), (B,C / / 2,H*W), and (B,H*W,C / / 2), respectively, where W is the input, reshape[·] represents the operation to change the shape, and conv 1×1 (·) represents a 1×1 convolution operation; (3) Multiply the two features theta and phi element by element to obtain The shape of f is (B, H*W, H*W). Multiplying f and g element-wise yields... The shape of y is (B, H*W, C / / 2), where, Indicates element-wise multiplication; (4) Use the convolutional attention mechanism module CBAM to perform adaptive feature optimization of y in space and channel to obtain z, the shape of z is (B,H,W,C); (5) Add z to the input K' element by element to get the final result K.

6. The intensity-guided multi-scale super-resolution reconstruction method according to claim 5, characterized in that, The final output of the nonlocal convolution module is as follows: Where K” is the output, softmax(·) is the normalized exponential function, and CBAM(·) represents the attention mechanism CBAM module. This indicates multiplication and addition between elements.

Images