A parallax-guided binocular super-resolution image reconstruction method and system

Abstract

The application provides a parallax-guided binocular super-resolution image reconstruction method and system, and relates to the technical field of computer vision binocular super-resolution; the method comprises the following steps: carrying out artifact removal processing on the input binocular low-resolution image through a constructed parallax-guided binocular super-resolution model to obtain a binocular artifact-removed image, carrying out iterative denoising processing on the binocular artifact-removed image to obtain a binocular super-resolution reconstruction image, estimating the binocular estimated parallax of the binocular super-resolution reconstruction image, respectively calculating the loss values corresponding to the binocular estimated parallax and the binocular super-resolution reconstruction image, optimizing the model according to different loss values, and using the optimized model to reconstruct a target binocular low-resolution image into a target binocular super-resolution reconstruction image. The model is constrained by parallax loss and path loss, the reconstruction accuracy of the model is improved, and a binocular high-resolution image with high fidelity and gathered high-frequency information is reconstructed.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of binocular super-resolution in computer vision, and in particular to a parallax-guided binocular super-resolution image reconstruction method and system. Background Art

[0002] In recent years, intelligence and informatization have become core drivers of urban development. Binocular surveillance video plays a crucial role in urban security management and the advancement of smart cities. Binocular surveillance cameras integrate two cameras into a single device, simulating left and right eye vision, respectively, to achieve depth perception and acquire three-dimensional spatial information. However, in practical applications, binocular surveillance often suffers from low video resolution and poor image quality. Therefore, improving the quality of binocular surveillance video has become a top priority. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a parallax-guided binocular super-resolution image reconstruction method and system in response to the deficiencies of the existing technology.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] A parallax-guided binocular super-resolution image reconstruction method comprises the following steps:

[0006] Importing a binocular super-resolution image, and performing downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image;

[0007] Constructing a disparity-guided binocular super-resolution model, the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructing a multi-task mutual feedback constraint mechanism and a path total loss function;

[0008] Performing de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, performing forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, and performing reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and performing disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity;

[0009] performing path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, performing disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimizing the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model;

[0010] The target binocular low-resolution image is reconstructed by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

[0011] Another technical solution of the present invention to solve the above technical problems is as follows:

[0012] A disparity-guided binocular super-resolution image reconstruction system, comprising:

[0013] An import module is used to import a binocular super-resolution image and perform downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image;

[0014] A construction module is used to construct a disparity-guided binocular super-resolution model, wherein the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructs a multi-task mutual feedback constraint mechanism and a path total loss function;

[0015] a computing module, configured to perform de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, perform forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, perform reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and perform disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity;

[0016] an optimization module, configured to perform path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, perform disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimize the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model;

[0017] The application module is used to reconstruct the target binocular low-resolution image by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

[0018] The beneficial effects of the present invention are as follows: the input image is de-artifacted to remove images that do not conform to the true form of the image, thereby obtaining a clearer image. During the model training process, the de-artifacted image is subjected to forward iterative denoising, using noise interference to simulate the information that would be lost in a low-resolution image, and then reverse iterative derivation is used to remove the noise, learning the information attenuation caused by noise, so as to effectively remove noise when processing low-resolution images and extract texture detail feature information. The depth information between the binocular images is extracted from the generated reconstructed image to calculate the disparity loss, and the disparity of the binocular images during the binocular image reconstruction process is calculated to calculate the path loss. The model is constrained by the disparity loss and path loss to improve the model reconstruction accuracy, so as to reconstruct a binocular high-resolution image with high fidelity and concentrated high-frequency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a disparity-guided binocular super-resolution image reconstruction method provided by an embodiment of the present invention;

[0020] Figure 2 A structural diagram of a disparity-guided binocular super-resolution image reconstruction method provided by an embodiment of the present invention;

[0021] Figure 3 This is a module block diagram of a disparity-guided binocular super-resolution image reconstruction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0023] Stereo image super-resolution reconstruction technology is used to mine the complementary information between the left and right perspectives of the image. It can reconstruct a high-resolution binocular image from the input low-resolution binocular image, and then obtain parallax to reconstruct three-dimensional information.

[0024] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a disparity-guided binocular super-resolution image reconstruction method, comprising the following steps:

[0025] Importing a binocular super-resolution image, and performing downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image;

[0026] Constructing a disparity-guided binocular super-resolution model, the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructing a multi-task mutual feedback constraint mechanism and a path total loss function;

[0027] Performing de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, performing forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, and performing reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and performing disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity;

[0028] performing path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, performing disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimizing the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model;

[0029] The target binocular low-resolution image is reconstructed by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

[0030] It should be understood that a pair of binocular low-resolution images is formed by a left-eye low-resolution image and a right-eye low-resolution image. When each of the above networks processes the binocular low-resolution images, they analyze and process the pair of binocular low-resolution images. Each network includes two networks with the same structure, namely a left-eye network and a right-eye network. The left-eye network processes the left-eye image in the same way as the right-eye network processes the right-eye image. The generated binocular data (i.e., binocular estimated disparity, binocular de-artifacted image, binocular noise image, and binocular super-resolution reconstructed image) includes left-eye data and right-eye data. For example, the binocular de-artifacted image is composed of a left-eye de-artifacted image and a right-eye de-artifacted image. The implicit diffusion network is optimized by the disparity loss and the path loss.

[0031] The implicit diffusion network consists of a forward diffusion network and a backward diffusion network. The backward diffusion network is optimized using disparity loss and path loss. The optimized disparity-guided binocular super-resolution model is constructed from the binocular artifact removal network and the optimized backward diffusion network. After training, the target binocular low-resolution image is input into the binocular artifact removal network, and the output of the binocular artifact removal network is input into the optimized backward diffusion network to obtain the target binocular super-resolution reconstructed image.

[0032] In this embodiment of the present invention, because frequency domain aliasing can cause frequency domain features to disappear or become redundant, an implicit diffusion network converts Euclidean space data into a continuous state space to extract effective texture detail features. The implicit diffusion network is guided by the estimated disparity of the binocular image. Specifically, the disparity-guided loss between the disparity distribution of the reconstructed image and the true disparity distribution is calculated, and the gradient parameters of the implicit diffusion network's reverse iterative denoising process are updated. Multiple loss functions are used to calculate multiple loss values, and a multi-task feedback constraint model is employed to improve the model's reconstruction accuracy.

[0033] Preferably, downsampling the binocular super-resolution image to obtain a binocular low-resolution image comprises:

[0034] The binocular super-resolution image is down-sampled by a bicubic interpolation method to obtain a binocular low-resolution image.

[0035] Specifically, we download super-resolution binocular image datasets, including the MiddleBurry, KITTI 2012, and KITTI 2015 datasets, and the Scene Flow dataset. We downsample the super-resolution datasets using bicubic interpolation to obtain low-resolution binocular image datasets, and then extract any binocular low-resolution image from them. We also partition the low-resolution binocular image datasets into blocks, generating multiple regular image blocks.

[0036] It should be understood that since the image input to the super-resolution task may be irregular, the original 512*1024 pixel image is gradually and iteratively divided into 64*64 pixel blocks and input one by one, which can reduce the computational complexity and improve the local restoration ability of the network. The divided image blocks are generally input into the model for separate processing. The common processing method is to input these image blocks into the network in sequence, that is, to process one image block at a time, rather than inputting multiple image blocks at the same time through multiple entrances. This can maintain the model's independent processing of each image block and reduce interference. In addition, the model can also use parallel computing to batch process image blocks. Although this is essentially sequentially processing each image block independently, it uses parallel technology to improve overall computing efficiency and reduce model computational complexity.

[0037] Preferably, performing artifact removal processing on the binocular low-resolution image by the binocular artifact removal network to obtain a binocular artifact-free image includes:

[0038] The binocular artifact removal network includes a restoration subnetwork and a global subnetwork, wherein the restoration subnetwork includes a plurality of first convolutional layers connected in series and a plurality of restoration residual blocks connected in series, and the global subnetwork includes a mapping layer, a plurality of second convolutional layers connected in series and a plurality of fully connected layers connected in series;

[0039] Iteratively performing a convolution operation on the binocular low-resolution image one by one through the plurality of first convolutional layers, and iteratively performing feature extraction on the binocular super-resolution image after the convolution operation one by one through the plurality of restored residual blocks, to obtain a binocular restored image;

[0040] performing a convolution operation on the binocular low-resolution image one by one through the plurality of second convolutional layers in an iterative manner, performing a fully connected operation on the binocular super-resolution image after the convolution operation one by one through the plurality of fully connected layers in an iterative manner to obtain a binocular global feature, and performing vector mapping on the binocular global feature through the mapping layer to obtain a binocular global image;

[0041] The binocular restored image is spliced ​​with the binocular global image to obtain a binocular artifact-removed image.

[0042] Specifically, the restoration network branch includes multiple convolutional layers and residual networks. Each restoration residual block includes two convolutional layers and a Leaky ReLU activation function. The output of each restoration residual block is expressed as:

[0043] X residual =X+LeakyReLU(W2*LeakyReLU(W1*X)),

[0044] Among them, X residual is the binocular restored image, X is the feature map output by the first convolutional layer (i.e., the binocular low-resolution image after the convolution operation), W1 and W2 are the convolution kernels of the convolution layer of the restored residual block, and * represents the convolution operation.

[0045] The global network branch consists of two convolutional layers, two fully connected layers, and a mapping layer. The mapping layer maps the final global features into a vector representation for combining with local features. The output of the global sub-network is expressed as:

[0046] X global =W g ·{FC2(FC1.Conv2(Conv1(x)) / )},

[0047] Among them, X global is the binocular global image, x is the binocular low-resolution image, FC1 and FC2 are fully connected layers, Conv1 and Conv2 are convolutional layers, and W g is the weight of the linear mapping layer.

[0048] Finally, the output X of the network branch will be recovered residual and the output X of the global network branch global Stitching to get binocular artifact-free image is the left eye artifact-free image, This is the artifact-free image for the right eye.

[0049] It should be understood that the number of iterations corresponds to the number of convolutional layers or restored residual blocks or fully connected layers. Based on the iteration method, the binocular low-resolution image is convolved one by one through multiple first convolutional layers, including: performing a convolution operation on the binocular low-resolution image through the first first convolutional layer to obtain a first first feature map, performing a convolution operation on the first first feature map through the second first convolutional layer to obtain a second first feature map, and so on, performing a convolution operation on the previous first feature map through the last first convolutional layer to output the last first feature map. The remaining iteration methods are the same as the iteration methods of the multiple first convolutional layers and will not be repeated here.

[0050] Preferably, the performing forward iterative denoising processing on the binocular artifact-removed image by the implicit diffusion network to obtain a binocular noise image comprises:

[0051] The implicit diffusion network includes a first conditional probability Markov chain having multiple stages. In any stage, Gaussian noise with a first set variance is added to the binocular de-artifacted image. The first conditional probability Markov chain calculates the positive probability distribution of the added Gaussian noise in the binocular de-artifacted image. When the positive probability distribution conforms to the normal Gaussian distribution, any binocular noise image is obtained. Similarly, the binocular de-artifacted image is processed one by one in multiple stages using the first conditional probability Markov chain to obtain a binocular noise image.

[0052] Specifically, the forward iterative denoising process first assumes that the data exists in Euclidean space, where the forward denoising process is designed as a first conditional probability Markov chain with T stage steps, gradually adding Gaussian random noise to the binocular artifact removal image I m In order to learn the reverse diffusion process. T is n time steps, T = t1, t2, ..., t n , every time step t i Add a preset variance β to the binocular artifact-free image t Gaussian noise with variance β t Responsible for controlling the amount of noise added at each time step, which is set to 10 -4 +(0.02-10 -4 )×t / T, and the distribution of the Gaussian noise added each time in the binocular artifact removal image conforms to the normal Gaussian distribution. Each addition obtains the corresponding initial binocular noise image, and finally until the time step t n Then we get the binocular noise image, that is, the left noise image and the right noise image

[0053] The calculation expression of the positive probability distribution is:

[0054]

[0055] Among them, q(x t |x t-1 ) is a positive probability distribution, is a Gaussian distribution, x t is a binocular noise image, I m is the binocular artifact-free image, β t is a preset variance that determines the amount of noise added at each step.

[0056] It should be understood that when the forward probability distribution does not conform to the normal Gaussian distribution, the distribution of the Gaussian noise with the set variance in the binocular de-artifacted image is adjusted until it conforms to the normal Gaussian distribution. The implicit diffusion network includes a forward diffusion network for forward iterative noise addition processing. The design of the forward diffusion network adopts a parameter sharing method, including a left-eye forward diffusion network and a right-eye forward diffusion network. Taking the left-eye forward diffusion network as an example, the left-eye de-artifacted image is taken as input, and the unconditional diffusion probability with T discrete steps is calculated by pre-training. For each step t i Time generates a variance strategy β i and its forward process conversion distribution

[0057] In an embodiment of the present invention, during the training process of the model, forward iterative denoising is used to add noise interference to the high-resolution image to simulate the information that would be lost in the low-resolution image, so that in the reverse iterative denoising process, the noise to be removed can be inferred based on conditional probability.

[0058] Preferably, performing reverse iterative denoising on the binocular noise image to obtain a binocular super-resolution reconstructed image includes:

[0059] The implicit diffusion network includes a second conditional probability Markov chain with multiple stages. In any stage, Gaussian noise of a second set variance amount is removed from the binocular noise image, and the second conditional probability Markov chain is used to calculate the inverse probability distribution of the remaining Gaussian noise in the binocular artifact-removed image. When the inverse probability distribution conforms to the normal Gaussian distribution, any binocular super-resolution reconstructed image is obtained. Similarly, the binocular noise image is processed one by one in multiple stages using the second conditional probability Markov chain to obtain a binocular high-quality reconstructed image, and the binocular high-quality reconstructed image is upsampled to obtain a binocular super-resolution reconstructed image.

[0060] It should be understood that when the inverse probability distribution does not conform to the normal Gaussian distribution, the distribution of the remaining Gaussian noise in the binocular artifact-removed image is adjusted until it conforms to the normal Gaussian distribution.

[0061] Preferably, the step of removing Gaussian noise of a second set variance in the binocular noise image includes:

[0062] The first set variance of the Gaussian noise is calculated using a denoising factor expression to obtain a second set variance. The Gaussian noise of the binocular noise image is removed according to the second set variance. The denoising factor expression is:

[0063] α i =1-β i ,

[0064] Among them, α i is the denoising factor (i.e. the second set variance), β i To set the variance.

[0065] It should be understood that the implicit diffusion network includes a reverse diffusion network for reverse iterative denoising processing, and the design of the reverse diffusion network adopts a parameter sharing method, including a left-eye reverse diffusion network and a right-eye reverse diffusion network. θ (x t ,t) Estimate a high-quality image for iterative disparity estimation To learn the disparity distribution prior of the left and right images.

[0066] Specifically, in the reverse iterative denoising process, taking the left-eye reverse diffusion network as an example, the conditional probability distribution is used After T times of denoising, a high-quality left-eye image is obtained. Conditional probability distribution P θ It can be expressed as:

[0067]

[0068] in, is the inverse probability distribution, is a Gaussian distribution, E is the unit matrix, θ represents the learnable parameter, Represented as the left noise image The mean of is the gradient of the left implicit diffusion network, is the adjustment parameter variance in the reverse iterative denoising process, is the variance strategy β through the forward iteration process t The estimated variance. By combining with gradient information, the conditional probability distribution is fine-tuned based on the basic variance strategy to generate a probability distribution for the next intermediate noise image that is more consistent with the model's denoising strategy and data characteristics.

[0069] That is, using the left implicit diffusion model Parameterized left-eye noise image at t time steps (i.e., T stages) Get high-quality left-eye image in, High-quality images It can be expressed as:

[0070] α i =1-β i ,

[0071]

[0072] Among them, α i is the noise removal factor (i.e., denoising factor), β i is the noise addition factor (i.e. setting the variance), is the cumulative denoising factor, is the left-eye noise image, is the left-eye implicit diffusion model, is the left eye high-quality image, and t is the conditional probability Markov chain of T stage steps.

[0073] Design the left noise image Estimated parallax via left eye Guide the disparity estimation network, and the right eye noise image Estimated parallax via the right eye Guided disparity estimation network, including: left eye high quality image and left eye high quality image Input the disparity estimation network G, calculate the estimated disparity of the high-quality image, and use the disparity guidance loss function to calculate the estimated disparity of the high-quality image to obtain the disparity guidance loss. The implicit diffusion network is optimized by the disparity guidance loss, and the processing steps of the disparity estimation network and the implicit diffusion network are repeated until the set number of iterations is met to reconstruct a binocular high-quality image with higher clarity.

[0074] In an embodiment of the present invention, the estimated disparity is combined with the true disparity to constrain the disparity guidance loss to optimize the denoising process and update the gradient of the implicit diffusion network, so that the disparity distribution of the generated high-quality image is consistent with the original disparity distribution, ensuring the accuracy of the spatial structure and depth information of the denoised image.

[0075] Preferably, the upsampling of the binocular high-quality reconstructed image to obtain a binocular super-resolution reconstructed image includes:

[0076] The binocular high-quality reconstructed image is converted from the frequency domain to the spatial domain to obtain a spatial feature map, the spatial feature map is calculated using a nearest neighbor interpolation algorithm to obtain an interpolation feature map, and a transposed convolution operation is performed on the interpolation feature map to obtain a binocular super-resolution reconstructed image.

[0077] Specifically, a Markov chain with T discrete steps is used to gradually reversely reconstruct the noise features, including: first, reconstructing the high-quality image The frequency domain is converted into a latent space, simulating noise to recover high-frequency details. After channel cropping the feature map every t time steps, an upsampling transposed convolution operation is performed to obtain an upsampled image. Finally, the upsampled image is weighted to reconstruct a high-quality high-resolution image (i.e., a binocular super-resolution reconstruction image).

[0078] The formula for multi-frequency fusion transposed convolution and nearest neighbor interpolation is (that is, the spatial feature map is calculated by the nearest neighbor interpolation algorithm to obtain an interpolated feature map, and the step of performing a transposed convolution operation on the interpolated feature map can be expressed as follows):

[0079] f(x)=Transconv2D.Grid_sample(F(X1)+F(X2)+…+F(X n )) / ,

[0080] Among them, f(x) is the upsampled image, Transconv2D(·) is the multi-frequency fusion transposed convolution function, Grid_sample(·) is the nearest neighbor interpolation algorithm, F(X i ) is the state t in the latent space (i.e. step t i ) image and perform transposed convolution operation.

[0081] The Markov chain reconstruction formula is (that is, the steps of weighted reconstruction of the upsampled image can be expressed as):

[0082]

[0083] Among them, Y is the binocular super-resolution reconstructed image, P ij is the probability of transitioning from state i to state j (i.e., the state at different time steps t), F t (X n ) is step t i The feature map (i.e., upsampled image) is , and N is the number of states.

[0084] In the embodiment of the present invention, the multi-frequency transposed convolution and nearest neighbor interpolation algorithms are used to improve the Markov chain fusion method, effectively reduce feature redundancy, and reconstruct high-quality binocular images.

[0085] Preferably, performing path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain the path loss includes:

[0086] The binocular super-resolution reconstructed image and the binocular high-quality reconstructed image are calculated using a first path loss function to obtain a first path loss, where the first path loss function is:

[0087]

[0088] in, is the first path loss, Reconstruct binocular images with high quality, is the binocular super-resolution reconstructed image, n is the total number of upsampling factors, i is the i-th upsampling factor, and ||·||1 is the L1 norm;

[0089] Downsampling the binocular super-resolution reconstructed image to obtain a binocular low-quality reconstructed image, downsampling the binocular super-resolution image to obtain a binocular low-quality image, and calculating the binocular high-quality reconstructed image, the binocular low-quality reconstructed image, and the binocular low-quality image using a second path loss function to obtain a second path loss, where the second path loss function is:

[0090]

[0091] in, is the second path loss, is a binocular low-quality image, and down(·) is a downsampling process;

[0092] The first path loss and the second path loss are calculated using a total path loss function to obtain a path loss, where the total path loss function is:

[0093]

[0094] Among them, L Total is the path loss, and δ is the loss function parameter.

[0095] Specifically, a binocular constraint twin network based on the L1 norm is designed to constrain the reconstruction process of the left eye image and the right eye image respectively, which includes left eye path constraint and right eye path constraint, and the processing method is the same.

[0096] Taking the left eye image as an example, the path constraint is to generate a high-quality left eye image by the left eye implicit diffusion model. After inputting the disparity estimation network, the left eye estimated disparity is obtained The path is constrained and a loss function is added to correct the unstable factors in the reconstruction process of each pair of binocular images, which is expressed as:

[0097]

[0098] in, Represents ×2 i times higher quality reconstructed images, Expressed as High-resolution images of the same resolution.

[0099] In path loss, is the super-resolution reconstructed image of the left eye that is finally described as restored. Generate low-resolution images by downsampling. In order to reduce the solution space as much as possible and mine the potential prior information of the network, pixel-by-pixel supervised learning is performed in the low-resolution space. The loss function of this path is expressed as:

[0100]

[0101] in, is the true value image A low-resolution image generated by downsampling.

[0102] Therefore, the total path loss can be obtained by the path loss of the left image processing and the path loss of the right image processing. The total loss function is:

[0103]

[0104] Among them, L Total is the path loss, is the first path loss of the left eye, is the first path loss of the right eye, is the second path loss of the left eye, is the second path loss of the right eye.

[0105] In the embodiment of the present invention, the main function of the path loss function is to constrain the binocular path so that the binocular super-resolution reconstructed image can correctly learn the high-frequency detail information of the high-resolution true value image.

[0106] Preferably, performing disparity calculation on the binocular super-resolution reconstructed image by the disparity estimation network to obtain binocular estimated disparity includes:

[0107] The disparity estimation network includes an encoder and a decoder, the encoder includes a plurality of third convolutional layers connected in series, and the decoder includes a convolution filter, a disparity calculation module and a plurality of decoding residual blocks;

[0108] Performing a convolution operation on the binocular super-resolution reconstructed image one by one through the plurality of third convolutional layers in an iterative manner to obtain binocular high-level semantic features;

[0109] The binocular high-level semantic features are extracted by a first decoding residual block to obtain a first binocular residual feature, the binocular high-level semantic features are filtered by the convolution filter to obtain a binocular filtering feature, the first binocular residual feature and the binocular filtering feature are fused to obtain an initial binocular deep feature, the initial binocular deep feature is extracted by a second decoding residual block to obtain a second binocular residual feature, the initial binocular deep feature and the second binocular residual feature are fused to obtain a binocular deep feature, the binocular deep feature is parameter-fused by the disparity calculation module to obtain a depth unary feature, and the depth unary feature is disparity extracted by a cost function to obtain a binocular estimated disparity.

[0110] Specifically, each residual block in the decoder is composed of two convolution filters connected in series. Convolution filters are used to process semantic features, and the decoded residual block composed of two convolution filters connected in series extracts deep features. The disparity calculation module forms deep unary features by sharing parameters between the left and right images. A cost function is designed to extract disparity information from the deep unary features. The formula is:

[0111]

[0112] in, is the disparity offset (i.e. binocular estimated disparity, including the left eye estimated disparity and the left eye estimated disparity ), I is the input image (i.e., binocular super-resolution reconstruction image), p is the image pixel, λ is the control parallax smoothing parameter, is the disparity gradient, ‖·‖1 is the L1 norm, is the L1 norm of the disparity gradient, F uni is a deep unary feature, and They are the depth features of the left and right images obtained by sharing the decoding residual block parameters.

[0113] It should be understood that when training the model, the disparity estimation network is used to calculate the disparity of the binocular low-resolution image to obtain the binocular estimated disparity. The initial extraction of the disparity is used to initialize the initial state and parameter settings of the model, directly providing the model with image space structure priors.

[0114] Preferably, the performing disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image through the multi-task mutual feedback constraint mechanism to obtain the disparity loss includes:

[0115] Dividing the binocular estimated disparity into multiple discrete intervals according to a value range, the multiple discrete intervals corresponding to multiple disparity center values, performing weighted calculation on the multiple disparity center values ​​using a Gaussian kernel to obtain multiple disparity weight distributions, normalizing the multiple disparity weight distributions to obtain multiple probability weight distributions, and summing the multiple probability weight distributions to obtain a disparity distribution;

[0116] The disparity distribution is calculated respectively by using the disparity guidance loss function to obtain the disparity guidance loss, and the disparity guidance loss function is:

[0117]

[0118] in, is the disparity-guided loss, is the disparity distribution, p θ (d|I HR ) is the true disparity distribution, max(d) is the maximum disparity value;

[0119] The binocular super-resolution reconstructed image is calculated using a pixel-level loss function to obtain a pixel-level loss. The pixel-level loss function is:

[0120]

[0121] in, is the pixel-level loss, is the left-eye super-resolution image, is the right eye super-resolution image, For the left eye super-resolution reconstructed image, is the super-resolution reconstructed image of the right eye, ||·||1 is the L1 norm;

[0122] The binocular estimated disparity is calculated using a disparity estimation loss function to obtain a disparity estimation loss. The disparity estimation loss function is:

[0123]

[0124] in, is the disparity estimation loss, is a smooth function, d is the true value disparity, Estimating disparity for binocular vision;

[0125] The disparity loss is composed of the disparity guidance loss, the pixel-level loss and the disparity estimation loss.

[0126] It should be understood that the imported binocular super-resolution images include a left-eye super-resolution image and a right-eye super-resolution image, and the true disparity distribution and the true disparity are calculated for the binocular super-resolution images.

[0127] Specifically, in order to increase the accuracy of disparity estimation and promote the performance of binocular image super-resolution algorithm, a multi-task mutual feedback constraint mechanism is designed and constructed to update the disparity and high-quality image generation, which is used to guide the reverse iterative denoising process in the implicit diffusion network.

[0128] In order to ensure that the high-quality images generated by the implicit diffusion network reverse process and the corresponding disparity obtained in the disparity estimation network are consistent with the original distribution, a disparity guidance loss is introduced to guide the implicit diffusion network to generate high-visual-fidelity images by minimizing the disparity distribution gap. Since the left and right perspectives are processed in the same way, taking the left perspective as an example, after the implicit diffusion network reverse process network Estimated high-quality reconstructed image of the left eye After disparity estimation, the estimated disparity map of the left eye high-quality reconstructed image is obtained (i.e., the estimated disparity of the left eye high-quality reconstructed image) and the original disparity d L Approximate the disparity distribution using a differentiable soft histogram:

[0129]

[0130] Among them, w k (d i ) is to use a Gaussian kernel to weight the center of each interval, σ is the standard Gaussian distribution, d i is a disparity value in the disparity map, b k is the disparity center value, p k (d i ) is the disparity value d i The probability distribution weight in the kth interval, h k is the final probability value of the kth interval represented by the histogram (i.e., disparity distribution), and N is the total number of pixels in the disparity map.

[0131] Left eye disparity distribution and the true disparity distribution Use KL divergence as a constraint and obtain the gradient through parallax guidance loss L Guides the update of the diffusion model. KL divergence The specific formula of the parallax guidance loss L is:

[0132]

[0133] Among them, K represents the Kth interval, is the average value of the Kth interval of the left eye disparity distribution, and max(d) is the maximum disparity value.

[0134] In order to effectively constrain the diffusion model to generate high-quality images, pixel-level loss is introduced The specific formula is:

[0135]

[0136] in It is the binocular super-resolution reconstructed image finally output by the diffusion network. is the true value image.

[0137] In order to effectively constrain the disparity, the disparity estimation loss is introduced. The specific formula is:

[0138]

[0139] Where d represents the true disparity, represents the estimated disparity.

[0140] like Figure 3 As shown, an embodiment of the present invention provides a disparity-guided binocular super-resolution image reconstruction system, comprising:

[0141] An import module is used to import a binocular super-resolution image and perform downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image;

[0142] A construction module is used to construct a disparity-guided binocular super-resolution model, wherein the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructs a multi-task mutual feedback constraint mechanism and a path total loss function;

[0143] a computing module, configured to perform de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, perform forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, perform reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and perform disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity;

[0144] an optimization module, configured to perform path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, perform disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimize the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model;

[0145] The application module is used to reconstruct the target binocular low-resolution image by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

[0146] The process of experimenting with the present invention comprises:

[0147] The stereo super-resolution datasets MiddleBurry, KITTI 2012, KITTI 2015, and Scene Flow were used. 4,000 images were randomly selected for the experiment, of which 3,800 images were used as training datasets, 100 images were used as validation datasets, and 100 images were used as test datasets. The scaling factors were ×2 and ×4, and all true image HRs (i.e., images in the stereo super-resolution dataset) were resized to 512×512. The evaluation indicators used included peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), and the experiment was conducted under the Pytorch framework. In addition, the model was trained by the ADAM optimizer with parameters β1 of 0.9, β2 of 0.999, and ε of 10. -8 .

[0148] The formula for PSNR is:

[0149]

[0150] Among them, MSE is the mean square error, which calculates the difference between the reconstructed image and the true value pixel by pixel, and MAX I is the maximum value of the image pixel, m is the number of pixels in the image height direction, n is the number of pixels in the image width direction, i and j are indexes of the rows and columns of the image, I(i,j) is the pixel value of the original reference high-resolution image at the i-th row and j-th column position, K(i,j) is the pixel value of the reconstructed super-resolution image at the i-th row and j-th column position. The formula for SSIM is:

[0151]

[0152] Among them, x and y are the reference high-resolution image and the reconstructed super-resolution image to be evaluated, respectively, and μ x is the average value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y, c1 and c2 are constants used to maintain stability.

[0153] The present invention is compared with the most advanced methods, including general image super-resolution algorithms, stereo matching image super-resolution algorithms, and disparity-guided binocular super-resolution algorithms and systems, including iPASSR, SSRDE-FNet, and NAFSSR methods.

[0154] The quantitative experimental results are shown in Table 1, which shows the PSNR and SSIM values ​​of different super-resolution algorithms on different datasets. The table lists the experimental results of different state-of-the-art methods and the present invention (CANSSR) on the KITTI2012, KITTI 2015 and Middlebury test datasets at scaling factors of ×2 and ×4. The present invention outperforms existing methods in all indicators.

[0155] Table 1:

[0156]

[0157] As for the actual results, first comparing the overall structure of the image, the Bicubic result is very blurry, with significant loss of texture information, while the overall structure of the present invention and several other methods is better preserved. Then, comparing the details, the Bicubic result is still poor, with details extremely blurry. While the other methods are relatively clear, they do not reconstruct detailed features well. However, due to the combination of binocular constraints, the present invention can well reconstruct the detailed features of feature edges in the image (such as the edge of the iron frame), and has superior reconstruction capabilities compared to other methods.

[0158] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0159] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the module division is only a logical functional division. In actual implementation, other division methods may be used. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0161] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A disparity-guided binocular super-resolution image reconstruction method, characterized in that: The steps include: Importing a binocular super-resolution image, and performing downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image; Constructing a disparity-guided binocular super-resolution model, the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructing a multi-task mutual feedback constraint mechanism and a path total loss function; Performing de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, performing forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, and performing reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and performing disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity; performing path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, performing disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimizing the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model; The target binocular low-resolution image is reconstructed by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

2. The binocular super-resolution image reconstruction method according to claim 1, wherein: The step of performing artifact removal processing on the binocular low-resolution image by the binocular artifact removal network to obtain a binocular artifact-removed image includes: The binocular artifact removal network includes a restoration subnetwork and a global subnetwork, wherein the restoration subnetwork includes a plurality of first convolutional layers connected in series and a plurality of restoration residual blocks connected in series, and the global subnetwork includes a mapping layer, a plurality of second convolutional layers connected in series and a plurality of fully connected layers connected in series; Iteratively performing a convolution operation on the binocular low-resolution image one by one through the plurality of first convolutional layers, and iteratively performing feature extraction on the binocular super-resolution image after the convolution operation one by one through the plurality of restored residual blocks, to obtain a binocular restored image; performing a convolution operation on the binocular low-resolution image one by one through the plurality of second convolutional layers in an iterative manner, performing a fully connected operation on the binocular super-resolution image after the convolution operation one by one through the plurality of fully connected layers in an iterative manner to obtain a binocular global feature, and performing vector mapping on the binocular global feature through the mapping layer to obtain a binocular global image; The binocular restored image is spliced ​​with the binocular global image to obtain a binocular artifact-removed image.

3. The binocular super-resolution image reconstruction method according to claim 1, wherein: The step of performing forward iterative denoising on the binocular artifact-removed image through the implicit diffusion network to obtain a binocular noise image includes: The implicit diffusion network includes a first conditional probability Markov chain having multiple stages. In any stage, Gaussian noise with a first set variance is added to the binocular de-artifacted image. The first conditional probability Markov chain calculates the positive probability distribution of the added Gaussian noise in the binocular de-artifacted image. When the positive probability distribution conforms to the normal Gaussian distribution, any binocular noise image is obtained. Similarly, the binocular de-artifacted image is processed one by one in multiple stages using the first conditional probability Markov chain to obtain a binocular noise image.

4. The binocular super-resolution image reconstruction method according to claim 3, wherein: The performing reverse iterative denoising on the binocular noise image to obtain a binocular super-resolution reconstructed image includes: The implicit diffusion network includes a second conditional probability Markov chain with multiple stages. In any stage, Gaussian noise of a second set variance amount is removed from the binocular noise image, and the second conditional probability Markov chain is used to calculate the inverse probability distribution of the remaining Gaussian noise in the binocular artifact-removed image. When the inverse probability distribution conforms to the normal Gaussian distribution, any binocular super-resolution reconstructed image is obtained. Similarly, the binocular noise image is processed one by one in multiple stages using the second conditional probability Markov chain to obtain a binocular high-quality reconstructed image, and the binocular high-quality reconstructed image is upsampled to obtain a binocular super-resolution reconstructed image.

5. The binocular super-resolution image reconstruction method according to claim 4, characterized in that: The removing of Gaussian noise of a second set variance in the binocular noise image includes: The first set variance of the Gaussian noise is calculated using a denoising factor expression to obtain a second set variance. The Gaussian noise of the binocular noise image is removed according to the second set variance. The denoising factor expression is: α i ＝1-β i ， Among them, α i is the second set variance, β i To set the variance.

6. The binocular super-resolution image reconstruction method according to claim 4, characterized in that: The upsampling process is performed on the binocular high-quality reconstructed image to obtain a binocular super-resolution reconstructed image, comprising: The binocular high-quality reconstructed image is converted from the frequency domain to the spatial domain to obtain a spatial feature map, the spatial feature map is calculated using a nearest neighbor interpolation algorithm to obtain an interpolation feature map, and a transposed convolution operation is performed on the interpolation feature map to obtain a binocular super-resolution reconstructed image.

7. The binocular super-resolution image reconstruction method according to claim 4, characterized in that: The performing path calculation on the binocular super-resolution reconstructed image by using the total path loss function to obtain the path loss includes: The binocular super-resolution reconstructed image and the binocular high-quality reconstructed image are calculated using a first path loss function to obtain a first path loss, where the first path loss function is: in, is the first path loss, Reconstruct binocular images with high quality, is the binocular super-resolution reconstructed image, n is the total number of upsampling factors, i is the i-th upsampling factor, and ||·||1 is the L1 norm; Downsampling the binocular super-resolution reconstructed image to obtain a binocular low-quality reconstructed image, downsampling the binocular super-resolution image to obtain a binocular low-quality image, and calculating the binocular high-quality reconstructed image, the binocular low-quality reconstructed image, and the binocular low-quality image using a second path loss function to obtain a second path loss, where the second path loss function is: in, is the second path loss, is a binocular low-quality image, and down(·) is a downsampling process; The first path loss and the second path loss are calculated using a total path loss function to obtain a path loss, where the total path loss function is: Among them, L Total is the path loss, and δ is the loss function parameter.

8. The binocular super-resolution image reconstruction method according to claim 1, wherein: The performing disparity calculation on the binocular super-resolution reconstructed image by the disparity estimation network to obtain binocular estimated disparity includes: The disparity estimation network includes an encoder and a decoder, the encoder includes a plurality of third convolutional layers connected in series, and the decoder includes a convolution filter, a disparity calculation module and a plurality of decoding residual blocks; Performing a convolution operation on the binocular super-resolution reconstructed image one by one through the plurality of third convolutional layers in an iterative manner to obtain binocular high-level semantic features; The binocular high-level semantic features are extracted by a first decoding residual block to obtain a first binocular residual feature, the binocular high-level semantic features are filtered by the convolution filter to obtain a binocular filtering feature, the first binocular residual feature and the binocular filtering feature are fused to obtain an initial binocular deep feature, the initial binocular deep feature is extracted by a second decoding residual block to obtain a second binocular residual feature, the initial binocular deep feature and the second binocular residual feature are fused to obtain a binocular deep feature, the binocular deep feature is parameter-fused by the disparity calculation module to obtain a depth unary feature, and the depth unary feature is disparity extracted by a cost function to obtain a binocular estimated disparity.

9. The binocular super-resolution image reconstruction method according to claim 1, wherein: The performing disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image through the multi-task mutual feedback constraint mechanism to obtain a disparity loss includes: Dividing the binocular estimated disparity into multiple discrete intervals according to a value range, the multiple discrete intervals corresponding to multiple disparity center values, performing weighted calculation on the multiple disparity center values ​​using a Gaussian kernel to obtain multiple disparity weight distributions, normalizing the multiple disparity weight distributions to obtain multiple probability weight distributions, and summing the multiple probability weight distributions to obtain a disparity distribution; The disparity distribution is calculated respectively by using the disparity guidance loss function to obtain the disparity guidance loss, and the disparity guidance loss function is: in, is the disparity-guided loss, is the disparity distribution, p θ (d|I HR ) is the true disparity distribution, max(d) is the maximum disparity value; The binocular super-resolution reconstructed image is calculated using a pixel-level loss function to obtain a pixel-level loss. The pixel-level loss function is: in, is the pixel-level loss, is the left-eye super-resolution image, is the right eye super-resolution image, For the left eye super-resolution reconstructed image, is the super-resolution reconstructed image of the right eye, ||·||1 is the L1 norm; The binocular estimated disparity is calculated using a disparity estimation loss function to obtain a disparity estimation loss. The disparity estimation loss function is: in, is the disparity estimation loss, is a smooth function, d is the true value disparity, Estimating disparity for binocular vision; The disparity loss is composed of the disparity guidance loss, the pixel-level loss and the disparity estimation loss.

10. A parallax-guided binocular super-resolution image reconstruction system, characterized in that: include: An import module is used to import a binocular super-resolution image and perform downsampling processing on the binocular super-resolution image to obtain a binocular low-resolution image; A construction module is used to construct a disparity-guided binocular super-resolution model, wherein the disparity-guided binocular super-resolution model includes a binocular artifact removal network, an implicit diffusion network, and a disparity estimation network connected in sequence, and constructs a multi-task mutual feedback constraint mechanism and a path total loss function; a computing module, configured to perform de-artifacting processing on the binocular low-resolution image through the binocular artifact removal network to obtain a binocular de-artifacted image, perform forward iterative denoising processing on the binocular de-artifacted image through the implicit diffusion network to obtain a binocular noise image, perform reverse iterative denoising processing on the binocular noise image to obtain a binocular super-resolution reconstructed image, and perform disparity calculation on the binocular super-resolution reconstructed image through the disparity estimation network to obtain a binocular estimated disparity; an optimization module, configured to perform path calculation on the binocular super-resolution reconstructed image using the total path loss function to obtain a path loss, perform disparity calculation on the binocular estimated disparity and the binocular super-resolution reconstructed image using the multi-task mutual feedback constraint mechanism to obtain a disparity loss, and optimize the disparity-guided binocular super-resolution model using the path loss and the disparity loss to obtain an optimized disparity-guided binocular super-resolution model; The application module is used to reconstruct the target binocular low-resolution image by guiding the binocular super-resolution model through the optimized disparity to obtain the target binocular super-resolution reconstructed image.

Images