Stereoscopic image blind super-resolution method based on hybrid degradation and content awareness

Abstract

The invention discloses a stereo image blind super-resolution method based on hybrid degradation and content perception, and belongs to the technical field of image processing. Aiming at the problem that a real scene three-dimensional image data set does not exist at present, the invention provides a three-dimensional image high-order gating degradation model to generate a three-dimensional image synthesis data set, so that complex degradation in a real world scene is simulated. Secondly, a hybrid degradation-content representation learning mechanism for stereo images is provided. According to the mechanism, through an unsupervised contrast learning mode, the network not only can extract degradation information in a three-dimensional image, but also can jointly extract explicit content information in a left viewpoint image and a right viewpoint image, and the learned HDCR is used as prior information to assist the network in better reconstruction. Then, the invention designs a hybrid prior fusion block which can adaptively process various different complex degeneration and efficiently fuse the learned prior information into the network. Finally, the invention provides a cross-viewpoint parallax attention module to accurately capture cross-viewpoint information. The invention provides a research scheme for a three-dimensional image super-resolution method in a real scene, and promotes the development of high-level visual tasks such as image detection and segmentation to a certain extent.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a blind super-resolution method for stereo images based on mixed degradation and content perception. Background Art

[0002] Images and videos have become an important form of visual information due to their vivid and intuitive features. In pursuit of a clearer and more realistic visual experience, digital image acquisition and processing technologies have developed rapidly. With the widespread application of robot navigation, virtual reality, and augmented reality technologies, stereo images with a sense of depth and three-dimensionality have attracted more and more attention from academia and industry. However, in the actual acquisition process, high-resolution stereo images are usually difficult to obtain due to limitations of shooting equipment and environment. Compared with high-cost hardware equipment, the use of software algorithms to achieve stereo image super-resolution (Stereo Image Super-Resolution, Stereo SR) is more flexible and economical. Under the original hardware conditions, this technology uses the spatial information and prior knowledge of the image to restore the high-frequency detail information lost in the image degradation process, thereby reconstructing high-resolution stereo images at a low cost.

[0003] As a professional technology with great practical value, stereoscopic image super-resolution reconstruction technology has been widely used in many fields. In the medical field, image super-resolution technology can enhance the resolution of tumor images, nuclear magnetic resonance images, etc. With the help of reconstructed high-resolution medical stereo images, lesions can be displayed more realistically, helping doctors to locate the location of diseased cells more accurately and improve the accuracy of disease diagnosis; in the field of reconnaissance remote sensing, image super-resolution technology can be applied to reconnaissance satellites and drone satellite systems at different locations to restore detailed information of small target objects taken from high altitudes, so as to clearly identify vehicles, roads, buildings and other objects; in the field of culture and entertainment, mobile phones can use image super-resolution technology to improve image quality, while enhancing user experience, saving hardware costs and increasing corporate profits; in the field of public safety and services, stereoscopic image super-resolution technology can process surveillance videos, which helps to further identify specific targets in surveillance videos, and is of great significance to public safety. In addition, super-resolution is also fully applied in intelligent security, animation industry, astronomy, seismology, biometrics and other aspects. Stereoscopic image super-resolution reconstruction technology can not only improve the visual effect of stereoscopic images, but also can be used as a preprocessing for some high-level visual tasks, such as video image detection, recognition and other understanding tasks, to further improve the performance of high-level visual tasks. Therefore, studying stereoscopic image super-resolution reconstruction technology has very important theoretical significance and application value.

[0004] Researchers' research on image super-resolution reconstruction technology originated in the 1960s. According to different specific usage scenarios, super-resolution reconstruction problems can be divided into three categories, namely single image super-resolution (SISR) applied to single image enlargement, video super-resolution (VSR) applied to continuous frame enlargement, and stereo image super-resolution applied to stereo image enlargement.

[0005] Stereo image super-resolution (Stereo SR) uses the spatial information of the left and right views themselves (single view information) and the complementary information between stereo images (cross view information) to restore the lost details in low-resolution images. One of the key issues in stereo image super-resolution is to effectively utilize cross-view information. Jeon et al. first used deep learning methods to solve the stereo image super-resolution reconstruction problem. By training a two-stage CNN, the reconstruction process of the left view image was realized. In order to alleviate the impact of the parallax between the left and right viewpoints, the method takes the left view image and 64 right view images after horizontal translation as the input of the network. The first stage CNN is responsible for reconstructing the high-resolution brightness image, and the second stage CNN is responsible for restoring the color information in the high-resolution image. However, this method can only handle stereo image super-resolution tasks with horizontal parallax within 64 pixels. In order to adapt to the parallax changes between different stereo images. Wang et al. proposed a parallax attention module (PAM) to handle stereo images with large horizontal parallax changes. By calculating the similarity of the pixels at the epipolar position, PAM can capture the correspondence between the pixels in the left and right views, thereby aligning the features of the two viewpoints and making effective use of the cross-view information. In addition to using a module similar to PAM to alleviate the parallax problem, Yan et al. proposed embedding a stereo matching network in a stereo image super-resolution network, and the two networks share shallow features. Subsequently, the stereo matching network calculates the parallax between the left and right viewpoints based on the shallow features, thereby spatially aligning the two images and alleviating the impact of the parallax in the stereo image. Inspired by the gated recurrent unit mechanism, Lei et al. designed an interactive unit to give full play to the complementary information in stereo images. The unit includes four gates, namely, filter gate, reset gate, screen gate, and update gate. Through these gates, the interactive unit can filter, reset, and update the features of the left and right viewpoints, thereby obtaining enhanced features containing cross-view information.

[0006] Although the above-mentioned CNN-based stereo image super-resolution reconstruction algorithm methods have shown remarkable performance, they all only consider a single degradation case, that is, simply using bicubic downsampling to generate LR (Low-Resolution) image pairs from HR (High-Resolution) image pairs. Due to the various complex degradations in the real world, these stereo super-resolution methods are not suitable for directly processing real-world images, and often have performance degradation and artifacts. However, there is currently no stereo image super-resolution method specifically for real scenes. In addition, if super-resolution reconstruction of stereo images in real scenes is to be performed, LR-HR paired data captured from the real world is required. However, since the capture of stereo images is more complex and challenging than the capture of a single image, there is currently no stereo image dataset for real scenes.

[0007] Based on the above content, the present invention proposes a blind super-resolution method for stereo images based on hybrid degradation and content perception to solve the above problems. Summary of the invention

[0008] 1. Technical Problems to be Solved by the Present Invention

[0009] The purpose of the present invention is to propose a blind super-resolution method for stereo images based on hybrid degradation and content perception to solve the problem raised in the background technology. The present invention provides a research scheme for stereo image super-resolution methods in real scenes, and to a certain extent promotes the development of high-level visual tasks such as image detection and segmentation.

[0010] 2. Technical solution

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] A stereo image blind super-resolution method based on hybrid degradation and content perception is proposed to reconstruct the stereo image in real scene by super-resolution by proposing a stereo image blind super-resolution network (HDCAnet). The stereo image blind super-resolution network includes:

[0013] The stereo image high-order gated degradation (SHG) model is used to generate a synthetic dataset of stereo images that simulates the complex degradations found in real-world scenes;

[0014] The stereoscopic hybrid degradation-content representation (HDCR) learning mechanism extracts degradation information from stereo images through unsupervised contrastive learning, jointly extracts explicit content information from left and right viewpoint images, and uses the learned hybrid degradation-content representation as prior information to assist the stereoscopic image blind super-resolution network in reconstruction.

[0015] Hybrid Prior Fusion Block (HPFB), which is used to adaptively handle various complex degradations and efficiently fuse the learned prior information into the stereo image blind super-resolution network;

[0016] Cross-viewpoint parallax attention module (CVPAM) to accurately capture cross-viewpoint information;

[0017] Based on a stereo image blind super-resolution network, the method comprises the following steps:

[0018] S1, input low-resolution left and right viewpoint images Pass it into the 3×3 convolutional layer to extract shallow features;

[0019] S2, the shallow features extracted in S1 are combined with the prior information through N cascaded hybrid prior transformation modules (HPTM) and cross-viewpoint disparity attention modules to fuse the prior information and extract deep features;

[0020] S3. In the feature reconstruction stage, the extracted deep features are reconstructed into a high-resolution image through a 1×1 convolution layer, a sub-pixel convolution layer, and a 3×3 convolution layer. The high-resolution residual images of the left and right viewpoints are added to the upsampled low-resolution image to obtain the final high-resolution reconstructed image.

[0021] Preferably, the training of the stereo image blind super-resolution network is divided into two stages:

[0022] In the first stage, a hybrid prior encoder (HPE) is trained so that the hybrid prior encoder initially extracts a hybrid degradation-content representation (HDCR);

[0023] In the second stage, the obtained hybrid degradation-content representation (HDCR) is used as the prior information P and sent to the stereo image blind super-resolution network (HDCAnet) for efficient fusion of prior information; in the second stage, the stereo image blind super-resolution network (HDCAnet) and the hybrid prior encoder (HPE) are jointly trained end-to-end.

[0024] Preferably, the stereoscopic image high-order gating (SHG) degradation model is implemented by a random gate controller by adding a gating mechanism on the basis of the high-order degradation model. Under the guidance of the random gate controller, multiple basic degradations are randomly shuffled and combined, thereby expanding the degradation space to cover different subsets of basic degradations;

[0025] The stereo image high-order gated degradation model applies the same degradation operation to a pair of left viewpoint images and right viewpoint images, and performs different degradation operations on different stereo image pairs; the formula of the stereo image high-order gated degradation model is as follows:

[0026] I LR =(G(D1)·G(D2)·G(D3)……G(D n ))(I HR )

[0027] Among them, I LR Represents a low-resolution image; I HR Represents a high-resolution image; Indicates the basic degradation type; G represents the gate controller.

[0028] Preferably, the stereographic hybrid degradation-content representation (HDCR) learning mechanism specifically includes the following contents:

[0029] For the i-th pair of images, first the input stereo image and Perform random cutting to obtain the left and right viewpoint image blocks in the same pair of stereo images and and patches from other stereo images

[0030] Then the obtained as well as After being encoded by the hybrid prior encoder (HPE), query samples, positive samples and negative samples are obtained respectively;

[0031] The loss function L used in the stereographic hybrid degradation-content representation (HDCR) learning mechanism is cl for:

[0032]

[0033] Where E(·) represents the hybrid prior encoder (HPE), N que represents the number of samples in the queue, τ represents the temperature hyperparameter, and B represents the batch size.

[0034] Preferably, the hybrid prior encoder (HPE) is composed of 6 convolutional layers, 1 average pooling layer and 1 multi-layer perceptron; the hybrid prior encoder (HPE) uses the result after the intermediate convolutional layer as prior information to retain spatial context information;

[0035] The hybrid prior transform module (HPTM) consists of a hybrid prior fusion block (HPFB) and a convolutional layer, and each of the hybrid prior fusion blocks (HPFB) consists of a spatial feature transform layer (SFT), a deformable convolution layer (DCN) and a residual connection.

[0036] Preferably, the processing flow of the hybrid prior fusion block (HPFB) specifically includes the following contents:

[0037] A deformable convolution layer (DCN) is used to dynamically adjust the sampling position and receptive field of the convolution kernel by learning modulation and mask. The function of the deformable convolution layer (DCN) processing flow is expressed as:

[0038]

[0039] Among them, Φ DCN Indicates the DCN layer, F in represents the input feature map of the left or right viewpoint, K represents the number of sampling positions, ω n represents the weight, p0 represents the center position, p n represents the predefined offset of the nth position; P represents the prior information, Δp n and Δm n represents the learnable offset and modulation scalar, which are determined by the prior information P and F in It is obtained through a convolutional layer after the splicing operation. The specific calculation formula is as follows:

[0040] (Δp n ,Δm n )=conv(concat(F in ,P))

[0041] Among them, concat(·) represents the cascade operator, conv(·) represents the convolution layer;

[0042] A spatial feature transformation layer (SFT) is used to adaptively fuse the image features with the prior information P. The spatial feature transformation layer (SFT) performs affine transformation through scaling and shifting operations to assist in adjusting the features of the reconstruction network. The function of the spatial feature transformation layer (SFT) is expressed as:

[0043] Φ SFT (F in |P) = γeF in +β=(conv(P))e F in +(conv(P))

[0044] Among them, Φ SFT represents the SFT layer, and e represents element-wise multiplication;

[0045] By using the SFT layer, the stereo image blind super-resolution network (HDCAnet) uses the prior information P to calculate the input feature F in Scaling and shifting operations are performed to adaptively modulate the left and right viewpoint feature maps to achieve the fusion guidance of the prior information P on the reconstruction features. The function of the above process is expressed as follows:

[0046] F out =Φ DCN(F in ∣P)+Φ SFT (F in ∣P)+F in

[0047] Among them, F out Represents the output feature map of the HPFB module for the left or right viewpoint.

[0048] Preferably, the processing flow of the cross-viewpoint parallax attention module (CVPAM) specifically includes the following contents:

[0049] The input feature F L and F R The normalization operation is performed through layer normalization (LN) and then sent to the residual block and linear layer to obtain and Its function is expressed as follows:

[0050]

[0051] Among them, Res represents the shared transition residual block used to alleviate training conflicts. and represents the projection matrix implemented by the 1×1 convolutional layer;

[0052] Will and Pass the whitening layer to generate robust stereo correspondence and obtain normalized features and The function of the above process is as follows:

[0053]

[0054] Will and after transposition Perform batch multiplication to generate a score map S∈R H×W×W ;

[0055] The softmax function is applied to the score map S, and the score map S is transposed along its last dimension, and finally the output is combined with the projection matrix F. L and F R The obtained features are multiplied, and the function of the above process is expressed as follows:

[0056]

[0057] Among them, softmax represents the softmax function, T represents the transposition operation, and W L and W R Represents the projection matrix obtained by the 1×1 convolutional layer;

[0058] Finally, global average pooling is used to transform FL and F R The spatial information of is compressed to the channel dimension, and then a linear layer is used to modulate the cross-view feature F L→R and F R→L The weights are then used to calculate the L→R and F R→L Perform element-by-element multiplication to implement the adaptive modulation process, and compare the modulation result with F L and F R Add element by element to fuse the cross-viewpoint features with the intra-viewpoint features to obtain the final output process and The function of the above process is expressed as:

[0059]

[0060] Among them, pool represents the global average pooling operation.

[0061] 3. Beneficial effects

[0062] The present invention proposes a blind super-resolution method for stereo images based on hybrid degradation and content perception. More specifically, the present invention designs a blind super-resolution network (HDCAnet) for stereo images based on hybrid degradation and content perception. First, in view of the fact that there is no real scene stereo image dataset, the present invention proposes a stereo image high-order gated degradation model (SHG degradation model) to generate a stereo image synthetic dataset, thereby simulating the complex degradation in real-world scenes. Secondly, the present invention proposes a hybrid degradation-content representation (HDCR) learning mechanism for stereo images, which extracts degradation information in stereo images and explicit content information in left-view images and right-view images through unsupervised contrastive learning, and uses it as a prior to assist the network in better reconstruction. Then, the present invention designs a hybrid prior fusion module (HPFB) to adaptively handle various complex degradations and efficiently fuse the learned prior information into the network. Finally, the present invention proposes a cross-view disparity attention module (CVPAM) to more accurately capture cross-view information.

[0063] The experimental results of the present invention show that the proposed method significantly improves the performance of the reconstruction network and obtains the best reconstruction effect on multiple synthetic and real test data sets, which fully proves the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of the SHG degradation model architecture mentioned in Example 1 of the present invention;

[0065] Figure 2This is a network architecture diagram of the stereoscopic image blind super-resolution method based on hybrid degradation and content perception mentioned in Example 1 of the present invention;

[0066] Figure 3 Schematic diagram of the HDCR learning mechanism mentioned in Example 1 of the present invention;

[0067] Figure 4 is a specific structural diagram of the HPE mentioned in Example 1 of the present invention;

[0068] Figure 5 Specific structural diagrams of the HPTM and HPFB mentioned in Example 1 of the present invention, wherein (a) is a schematic diagram of a hybrid prior transformation module (HPTM), and (b) is a schematic diagram of a hybrid prior fusion block (HPFB);

[0069] Figure 6 This is a specific structural diagram of the CVPAM mentioned in Example 1 of the present invention;

[0070] Figure 7 This is a visualization result diagram of the pre-trained Stereo SR method mentioned in Example 2 of the present invention on a synthetic dataset with 4 times super-resolution;

[0071] Figure 8 This is the visualization result of the retrained Stereo SR method mentioned in Example 2 of the present invention on a 4-fold super-resolved synthetic dataset. DETAILED DESCRIPTION

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0073] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0074] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0075] First, the abbreviations and key term definitions mentioned in the present invention are explained:

[0076] Stereo SR: Stereo Image Super-Resolution, stereo image super-resolution

[0077] SISR: Single Image Super Resolution, single image super resolution

[0078] VSR: Video Super-Resolution

[0079] PAM: Parallax Attention Module

[0080] HR: High-Resolution

[0081] LR: Low-Resolution, low resolution

[0082] HDCAnet: hybrid degradation-content aware stereo super-resolution network, a stereo blind super-resolution network based on hybrid degradation and content awareness

[0083] SHG degradation model: Stereo-image High-order Gated Degradation Model, high-order gated degradation model

[0084] HDCR: Hybrid Degradation-Content Representation, Hybrid Degradation-Content Representation

[0085] HPFB: Hybrid Prior Fusion Block, hybrid prior fusion block

[0086] CVPAM: Cross-View Parallax Attention Module

[0087] HPE: hybrid prior encoder, hybrid prior encoder

[0088] HPTM: Hybrid Prior Transform Module

[0089] SFT: spatial feature transform, spatial feature transform

[0090] DCN: deformable convolution

[0091] The following is an explanation of a blind super-resolution method for stereo images based on mixed degradation and content perception proposed by the present invention in conjunction with specific examples and drawings. The specific contents are as follows.

[0092] Embodiment 1:

[0093] The present invention proposes a blind super-resolution method for stereo images based on hybrid degradation and content awareness. First, in view of the fact that there is no real scene stereo image dataset, the present invention proposes a SHG degradation model to generate a stereo image synthetic dataset, thereby simulating the complex degradation in real-world scenes. At the same time, the present invention introduces a blind super-resolution network for stereo images based on hybrid degradation and content awareness (HDCAnet), which can learn the degradation information and jointly learn the explicit content information of the left and right views.

[0094] In order to solve the problem that the simple combination of basic degradation types in classical degradation cannot adapt to the complex degradation of real scenarios and the problem that different subsets of basic degradation may be ignored in high-order gated degradation, the present invention proposes a SHG degradation model, whose specific structure is as follows: Figure 1 As shown. Specifically, the SHG degradation model is implemented by adding a gating mechanism on the basis of the high-order degradation model, which is implemented by a random gate controller. Under the guidance of the random gate controller, multiple basic degradations are randomly shuffled and combined, thereby further expanding the degradation space to cover different subsets of basic degradations. In addition, the model applies the same degradation operation to a pair of left viewpoint images and right viewpoint images, and performs different degradation operations on different stereo image pairs. The model is expressed by the following formula:

[0095] I LR =(G(D1)·G(D2)·G(D3)…G(D n ))(I HR )

[0096] Among them, I LR and I HR Represent low-resolution images and high-resolution images respectively, represents the basic degradation type, and G represents the gate controller. By adjusting the probability of the random gate controller, the SHG degradation model greatly expands the degradation space, simulates the complex degradation in the real scene to a great extent without collecting additional data, and enhances the robustness of HDCAnet.

[0097] The overall framework of HDCAnet is as follows Figure 2Its training is mainly divided into two stages: the first stage is to train the hybrid prior encoder (HPE), as shown in Figure 3 As shown, HPE initially extracts HDCR; in the second stage, the obtained HDCR is used as prior information P (in the present invention, P is used to represent HDCR), and is sent to HDCAnet for efficient fusion of prior information, as shown in FIG. Figure 2 In the second stage, HDCAnet and HPE are jointly trained end-to-end.

[0098] like Figure 2 As shown in the figure, first, HDCAnet takes the input low-resolution left and right viewpoint images First, the shallow features are extracted by passing the 3×3 convolutional layer. Then, the extracted shallow features and P are passed through N cascaded hybrid prior transform modules (HPTM) and CVPAM modules to fuse the prior information and extract deep features. The specific structure of HPTM is as follows: Figure 5 As shown in (a), it mainly consists of two HPFBs, two 3×3 convolutional layers and residual connections. The specific structure of HPFB is as follows Figure 5 (b) is used to efficiently fuse the self-viewpoint information with the prior information. The specific structure of CVPAM is as follows: Figure 6 As shown in Figure 1, it is used to extract cross-viewpoint features. Finally, in the feature reconstruction stage, the features are reconstructed into a high-resolution image through a 1×1 convolution layer, a sub-pixel convolution layer, and a 3×3 convolution layer, and the high-resolution residual images of the left and right viewpoints are added to the upsampled low-resolution image to obtain the final high-resolution reconstructed image. and The network simultaneously performs left and right viewpoint stereo image super-resolution reconstruction and shares the weights of the left and right branches.

[0099] The image degradation in real scenes is diverse and complex, which seriously affects the reconstruction performance of existing networks. To solve this problem, the present invention proposes a HDCR learning mechanism, such as Figure 3 Specifically, for the i-th pair of images, first the input stereo image and Perform random cutting to obtain the left and right viewpoint image blocks in the same pair of stereo images and and patches from other stereo images Then, the obtained as well as After HPE encoding, we get query samples, positive samples, and negative samples. As an example, it is regarded as the query Q, also used as HDCR, and its corresponding right view image block can be regarded as positive samples, while patches from other images is considered as a negative sample. The loss function used in this contrastive learning is as follows:

[0100]

[0101] Where E(·) represents HPE, N que represents the number of samples in the queue, τ is the temperature hyperparameter, and B is the batch size.

[0102] The basic structure of HPE proposed by the present invention is as follows Figure 4 Specifically, the basic structure of HPE consists of 6 convolutional layers, 1 average pooling layer and a multi-layer perceptron. In order to better preserve the spatial context information, HPE chooses to use the result after the intermediate convolutional layer as P, so that the obtained P is consistent with other features. Figure 1 In this way, a four-dimensional tensor that retains the spatial structure is obtained, rather than a two-dimensional vector that lacks the spatial structure, thereby further realizing the efficient use of image content information.

[0103] The HPTM structure consists of HPFB and convolutional layers. The specific structure is as follows Figure 5 (a) As shown. Each HPFB block consists of a spatial feature transform layer (Spatial Feature Transform, SFT), a deformable convolution layer (DeformableConvolution, DCN) and a residual connection. The specific structure is as follows Figure 5 (b) as shown.

[0104] Specifically, first, DCN is used in HPFB to dynamically adjust the sampling position and receptive field of the convolution kernel by learning the modulation offset and mask. The entire process of DCN can be expressed as follows:

[0105]

[0106] Among them, Φ DCN Indicates the DCN layer, F in is the input feature map of the left or right viewpoint, K is the number of sampling locations, ω n is the weight, p0 is the center position, p n is the predefined offset of the nth position. Δp n and Δm n are learnable offset and modulation scalars, both of which are determined by the priors P and F inIt is obtained through a convolutional layer after the splicing operation. The calculation formula is as follows:

[0107] (Δp n ,Δm n )=conv(concat(F in ,P))

[0108] Among them, concat(·) is a cascade operator, and conv(·) represents a convolutional layer. Using DCN, HDCAnet can dynamically adjust the receptive field according to the image content. In addition, the SFT layer is used in HPFB to adaptively fuse image features with P. The SFT layer assists in adjusting the features in the reconstruction network by performing affine transformations using scaling and shift operations. The SFT layer can be expressed by the following formula:

[0109] Φ SFT (F in |P) = γe F in +β=(conv(P))e F in +(conv(P))

[0110] Among them, Φ SFT represents the SFT layer, and e represents element-by-element multiplication. By using the SFT layer, HDCAnet can perform an operation on the input feature F according to the prior P. in Scaling and shifting operations are performed to adaptively modulate the left and right viewpoint feature maps, realizing the fusion guidance of the prior P on the reconstruction features. The whole process can be expressed by the following formula:

[0111] F out =Φ DCN (F in ∣P)+Φ SFT (F in ∣P)+F in

[0112] Among them, F out is the output feature map of the left or right viewpoint of the HPFB module. In general, through the HPFB block, HDCAnet can adaptively handle different degradations and effectively fuse prior information.

[0113] In order to better obtain cross-viewpoint features from corresponding viewpoints while fusing prior information, the present invention proposes CVPAM, the specific structure of which is as follows: Figure 6 shown.

[0114] First, input feature F L and F R It is normalized by layer normalization (LN) and then sent to the residual block and the linear layer (implemented by 1×1 convolution) to obtain and The formula can be expressed as follows:

[0115]

[0116] Among them, Res represents the shared transition residual block used to alleviate training conflicts. and is the projection matrix implemented by the 1×1 convolutional layer. After that, and Generate robust stereo correspondence through the whitening layer and obtain normalized features and The process can be expressed by the following formula:

[0117]

[0118] and after transposition Perform batch multiplication to generate a score map S∈R H×W×W Then apply the softmax function to S and transpose S along their last dimension, and finally pass the output through the projection matrix F L and F R The obtained features are multiplied, and the process can be expressed by the following formula:

[0119]

[0120] Where softmax is the softmax function, T represents the transposition operation, and W L and W R is the projection matrix obtained by the 1×1 convolution layer. Finally, global average pooling is used to convert F L and F R The spatial information of is compressed to the channel dimension, and then a linear layer is used to modulate the cross-view feature F L→R and F R→L The weight is then used with F L→R and F R→L Perform element-by-element multiplication to implement the adaptive modulation process, and add the modulation result to F L and F R Add element by element to fuse the cross-viewpoint features with the intra-viewpoint features to obtain the final output process and The process can be expressed by the following formula:

[0121]

[0122] Where pool represents the global average pooling operation. HDCAnet inserts a CVPAM after each HPTM to interact with the left and right viewpoint features, so as to fully extract and accurately fuse complementary information.

[0123] Embodiment 2:

[0124] Based on Example 1, but different in that, a stereoscopic image blind super-resolution method based on mixed degradation and content perception proposed by the present invention is described below in combination with specific experiments, and the specific content is as follows.

[0125] (1) Dataset

[0126] Training data set: In view of the relative scarcity of data sets in the field of blind super-resolution reconstruction of stereo images, the present invention preprocesses the commonly used Flickr1024 data set and generates a new training data set suitable for blind super-resolution reconstruction of stereo images through the above-mentioned SHG degradation model.

[0127] Test datasets: First, in order to conduct appropriate tests in the field of blind super-resolution reconstruction of stereo images, new test datasets are generated using the SHG degradation model for four public benchmark datasets: KITTI2012, KITTI2015, Middlebury, and Flickr1024. The specific divisions are as follows: the KITTI 2012 dataset contains 20 pairs of stereo images; the KITTI 2015 dataset contains 20 pairs of stereo images; the Middlebury dataset contains 5 pairs of stereo images, which are mainly used for testing indoor scenes; the Flickr1024 dataset contains 112 pairs of stereo images, covering multiple categories of images, such as buildings, street scenes, indoors, and people. Secondly, the Track3 dataset (Track3-Flickr1024) in the NTIRE2023 stereo image super-resolution reconstruction competition is additionally used. This dataset contains 112 pairs of stereo images (×4), which comes from the verification dataset of Flickr1024 and is an artificial synthetic dataset generated using a classic degradation model.

[0128] (2) Experimental setup

[0129] During the training phase, for simplicity, the probability of each gate G is set to 0.5. Then, the present invention crops the obtained left and right images into small blocks of size 30×90. The training data is randomly flipped horizontally and vertically for data enhancement. The training is completed in two stages. First, the contrast loss in formula (3-4) is used in the first stage. The encoder network is trained for 30K iterations, with the initial learning rate set to 2×10 -3 , after 18K iterations, it drops to 2×10 -4 , N in formula (3-4) queand τ are set to 0.07 and 8192 respectively. Then in the second stage, the whole network is jointly trained for 100K iterations. During the training process, the AdamW optimizer is used, and the parameters β1 = 0.9, β2 = 0.9 are set to the default values, the batch size is set to 32, and the initial learning rate is set to 5×10 -4 , using the cosine annealing strategy and setting its minimum learning rate to 1×10 -7 The model proposed in this invention is implemented based on Python language, and the experimental environment is shown in Table 1:

[0130] Table 1: Experimental environment

[0131]

[0132] (3) Experimental results

[0133] The proposed method is compared with previous scene text detection methods on five data sets, and the performance of the proposed method is evaluated from two indicators: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

[0134] Tables 2 and 3 compare the proposed HDCAnet with several state-of-the-art super-resolution methods, including two single image super-resolution methods (EDSR and RCAN), four blind super-resolution methods (DANv2, DCLS, DASR and CDSR) and five stereo image super-resolution methods (PASSRnet, iPASSR, SSRDE-FNet, NAFSSR and swinFIRSSR). The best evaluation indicators are marked in bold and the suboptimal indicators are marked in underline. It can be seen that when faced with complex degraded stereo images, the performance of previous single image super-resolution methods and stereo image super-resolution methods is greatly reduced, and the blind super-resolution method is also unsatisfactory when processing stereo images due to the lack of cross-view information. The method of the present invention is significantly better than previous methods in terms of PSNR and SSIM indicators.

[0135] Table 2: Experimental results on KITTI2012 dataset and KITTI2015 dataset

[0136]

[0137]

[0138] Table 3: Experimental results on the Middlebuy dataset, Flickr1024 dataset, and Track3-Flickr1024 dataset

[0139]

[0140]

[0141] In addition, in order to make a fair comparison, we also selected some existing stereo image super-resolution methods, namely PASSRnet, iPASSR, NAFSSR, swinFIRSSR, and retrained them from scratch using the same training data set as the present invention (referred to as constraints in the table). The experimental results can be seen from Tables 4 and 5, where the best evaluation indicators are marked in bold and the suboptimal indicators are marked in underline. It can be seen that the performance of the retrained stereo image super-resolution method is greatly improved than before (approximately 0.2dB-0.6dB on the Track 3-Flickr 1024 dataset), which proves the key role of the SHG degradation model proposed in the present invention. In addition, compared with these retrained stereo image super-resolution networks, the present invention has achieved the best results in most results, showing the superiority of the network design of the present invention.

[0142] Table 4: Experimental results on KITTI2012 and KITTI2015 datasets under restricted conditions

[0143]

[0144] Table 5: Experimental results on the Middlebuy dataset, Flickr1024 dataset, and Track3-Flickr1024 dataset under restricted conditions

[0145]

[0146] In order to more intuitively demonstrate the effectiveness of the method proposed in the present invention, a visual comparison is made between the HDCAnet proposed in the present invention and several super-resolution methods. Figure 7 The visualization results of HDCAnet and pre-trained super-resolution methods in 4 super-resolution experiments are shown. It can be observed that all other methods cannot remove noise or reconstruct clear lines, and some super-resolution results of these pre-trained methods are even close to the original LR, while HDCAnet can not only effectively remove noise and blur, but also restore clearer patterns and textures.

[0147] Figure 8The visualization results of HDCAnet and the retrained super-resolution method in the 4x super-resolution experiment are shown. It can be seen that when dealing with severe noise and blur, the zebra crossings (in KITTI_2015_0001L) and stripes (in Flickr1024_0106_L) restored by other methods are more blurred and distorted, accompanied by artifacts. In contrast, the images reconstructed by HDCAnet are clearer, and the restored zebra crossings and stripes are straighter, which shows the superiority and robustness of the proposed method.

[0148] Table 6 shows the results of ablation experiments on various modules in HDCAnet. Variant 1 removes L in the loss function during the second training phase without changing the network. cl In addition, the pre-trained HPE was removed in the first training stage, and the entire network was jointly trained 100,000 times to obtain Variant 2. Finally, HPE was converted to the degradation encoder used in DASR, and HPFB was converted to the DA module used in DASR to construct Variant 3. The experimental results are shown in Table 6. The best performance is bold. The results show that both the HDCR learning mechanism and HPE are indispensable to the network, because HDCR learning enhances the ability of HPE to obtain discriminative features, while pre-training HPE in the first training stage can obtain more accurate prior information in advance, which is beneficial to subsequent joint training. In addition, compared with DASR, which extracts global degradation information, HDCAnet has obvious advantages in utilizing content information by retaining spatial context information.

[0149] Table 6: Ablation results of 4x super-resolution under different settings

[0150]

[0151]

[0152] Table 7 shows the ablation experimental results of the extracted explicit content information. The prior P extracted by the method of the present invention contains not only degradation information but also explicit content information. Deformation 4 adopts a different training method. Specifically, two non-overlapping patches are randomly selected in the same LR image to obtain positive samples with the same degradation, and different degradations from other LR images are used as negative samples, so that the extracted P contains degradation information and implicit content information. It can be seen from Table 7 that after removing the explicit content information, the indicators on the four data sets are significantly reduced. The experimental results fully demonstrate the rationality and effectiveness of using explicit content information, as well as its importance for the HDCAnet stereo image blind super-resolution reconstruction method.

[0153] Table 7: Ablation results of explicit content information

[0154]

[0155] In order to prove the effectiveness of the HPFB proposed in the present invention, ablation experiments were performed by removing the DCN layer, the SFT layer and the entire HPFB. The results are shown in Tables 8 and 9, and the best performance is bolded. Taking Real-KITTI 2012 as an example, HDCAnet was used to conduct experiments and the best result 21.89 / 0.6003 was obtained. When HPFB was deleted, the PSNR and SSIM values ​​​​were greatly reduced from 21.89 / 0.6003 to 21.64 / 0.5953. By deleting the DCN layer and the SFT layer respectively, the PSNR of the network decreased by 0.09dB and 0.07dB, which also proves that the DCN layer and the SFT layer are both important for improving the performance of HDCAnet.

[0156] Table 8: HPFB ablation experiment results with 4x over-resolution on KITTI2012 dataset and Middlebuy dataset

[0157]

[0158] Table 9: HPFB ablation experimental results with 4x super-resolution on Flickr1024 dataset and Track3-Flickr102 dataset

[0159]

[0160] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.

Claims

1. A blind super-resolution method for stereo images based on hybrid degradation and content awareness, characterized in that: A stereo image blind super-resolution network is proposed to reconstruct a stereo image in a real scene with super-resolution. The stereo image blind super-resolution network includes: A high-order gated degradation model for stereo images, which is used to generate synthetic datasets of stereo images and simulate the complex degradation in real-world scenes; The mixed degradation-content representation learning mechanism for stereo graphics extracts degradation information from stereo images through unsupervised contrast learning, jointly extracts explicit content information from left-view and right-view images, and uses the learned mixed degradation-content representation as prior information to assist the stereo image blind super-resolution network in reconstruction. The hybrid prior fusion block is used to adaptively handle various complex degradations and efficiently fuse the learned prior information into the stereo image blind super-resolution network; Cross-viewpoint disparity attention module to accurately capture cross-viewpoint information; Based on the above stereo image blind super-resolution network, the method comprises the following steps: S1, input low-resolution left and right viewpoint images Pass it into the 3×3 convolutional layer to extract shallow features; S2, the shallow features extracted in S1 are combined with the prior information through N cascaded hybrid prior transformation modules and cross-viewpoint parallax attention modules to fuse the prior information and extract deep features; S3. In the feature reconstruction stage, the extracted deep features are reconstructed into a high-resolution image through a 1×1 convolution layer, a sub-pixel convolution layer, and a 3×3 convolution layer. The high-resolution residual images of the left and right viewpoints are added to the upsampled low-resolution image to obtain the final high-resolution reconstructed image.

2. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 1, characterized in that: The training of the stereo image blind super-resolution network is divided into two stages: In the first stage, the hybrid prior encoder is trained so that the hybrid prior encoder initially extracts the hybrid degradation-content representation; In the second stage, the obtained mixed degradation-content representation is used as the prior information P and sent to the stereo image blind super-resolution network for efficient fusion of the prior information; In the second stage, the stereo image blind super-resolution network and the hybrid prior encoder are jointly trained end-to-end.

3. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 1, characterized in that: The stereoscopic image high-order gated degradation model is implemented by adding a gating mechanism on the basis of the high-order degradation model, and is implemented by a random gate controller. Under the guidance of the random gate controller, multiple basic degradations are randomly shuffled and combined, thereby expanding the degradation space to cover different subsets of basic degradations. The stereo image high-order gated degradation model applies the same degradation operation to a pair of left viewpoint images and right viewpoint images, and performs different degradation operations on different stereo image pairs; the formula of the stereo image high-order gated degradation model is as follows: I LR =(G(D1)·G(D2)·G(D3)……G(D n ))(I HR ) Among them, I LR Represents a low-resolution image; I HR Represents a high-resolution image; Indicates the basic degradation type; G represents the gate controller.

4. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 1, characterized in that: The three-dimensional graphics mixed degradation-content representation learning mechanism specifically includes the following contents: For the i-th pair of images, first the input stereo image and Perform random cutting to obtain the left and right viewpoint image blocks in the same pair of stereo images and and patches from other stereo images Then the obtained as well as After encoding by the hybrid prior encoder, query samples, positive samples and negative samples are obtained respectively; The loss function L used by the stereographic hybrid degradation-content representation learning mechanism is cl for: Where E(·) represents the hybrid prior encoder, N que represents the number of samples in the queue, τ represents the temperature hyperparameter, and B represents the batch size.

5. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 4, characterized in that: The hybrid a priori encoder consists of 6 convolutional layers, 1 average pooling layer and 1 multi-layer perceptron; the hybrid a priori encoder uses the result after the intermediate convolutional layer as a priori information to retain spatial context information; The hybrid prior transformation module consists of a hybrid prior fusion block and a convolution layer, and each of the hybrid prior fusion blocks consists of a spatial feature transformation layer, a deformable convolution layer and a residual connection.

6. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 1 or 5, characterized in that: The processing flow of the hybrid prior fusion block specifically includes the following contents: A deformable convolution layer is used to dynamically adjust the sampling position and receptive field of the convolution kernel by learning modulation and mask. The function of the deformable convolution layer processing flow is expressed as: Among them, Φ DCN Indicates the DCN layer, F in represents the input feature map of the left or right viewpoint, K represents the number of sampling positions, ω n represents the weight, p0 represents the center position, p n represents the predefined offset of the nth position; P represents the prior information, Δp n and Δm n represents the learnable offset and modulation scalar, which are determined by the prior information P and F in It is obtained through a convolutional layer after the splicing operation. The specific calculation formula is as follows: (Δp n ,Δm n )=conv(concat(F in ,P)) Among them, concat(·) represents the cascade operator, conv(·) represents the convolution layer; A spatial feature transformation layer is used to adaptively fuse the image features with the prior information P. The spatial feature transformation layer performs affine transformation through scaling and shifting operations to assist in adjusting the features of the reconstruction network. The function of the spatial feature transformation layer is expressed as: Φ SFT (F in ∣P)=γe F in +β=(conv(P))e F in +(conv(P)) Among them, Φ SFT represents the SFT layer, and e represents element-wise multiplication; By using the SFT layer, the stereo image blind super-resolution network uses the prior information P to calculate the input feature F in Scaling and shifting operations are performed to adaptively modulate the left and right viewpoint feature maps to achieve the fusion guidance of the prior information P on the reconstruction features. The function of the above process is expressed as follows: F out =Φ DCN (F in ∣P)+Φ SFT (F in ∣P)+F in Among them, F out Represents the output feature map of the HPFB module for the left or right viewpoint.

7. The method for blind super-resolution of stereo images based on hybrid degradation and content perception according to claim 1, characterized in that: The processing flow of the cross-viewpoint disparity attention module specifically includes the following contents: The input feature F L and F R The normalization operation is performed through layer normalization and then sent to the residual block and linear layer to obtain and Its function is expressed as follows: Among them, Res represents the shared transition residual block used to alleviate training conflicts. and represents the projection matrix implemented by the 1×1 convolutional layer; Will and Pass the whitening layer to generate robust stereo correspondence and obtain normalized features and The function of the above process is as follows: Will and after transposition Perform batch multiplication to generate a score map S∈R H×W×W ; The softmax function is applied to the score map S, and the score map S is transposed along its last dimension, and finally the output is combined with the projection matrix F. L and F R The obtained features are multiplied, and the function of the above process is expressed as follows: Among them, softmax represents the softmax function, T represents the transposition operation, and W L and W R Represents the projection matrix obtained by the 1×1 convolutional layer; Finally, global average pooling is used to transform F L and F R The spatial information of is compressed to the channel dimension, and then a linear layer is used to modulate the cross-view feature F L→R and F R→L The weights are then used to calculate the L→R and F R→L Perform element-by-element multiplication to implement the adaptive modulation process, and compare the modulation result with F L and F R Add element by element to fuse the cross-viewpoint features with the intra-viewpoint features to obtain the final output process and The function of the above process is expressed as: Among them, pool represents the global average pooling operation.

Images