A near-infrared image-guided cross-modal stereo image super-resolution reconstruction method

Abstract

The present invention belongs to the field of image editing technology, and specifically is a cross-modal stereo image super-resolution reconstruction method guided by near-infrared images. The present invention includes: using a cross-view matching module to extract the disparity between the left and right images of different modes of a stereo image, and then aligning the near-infrared image of the right view to the left view according to the disparity; the image super-resolution reconstruction network uses two network branches to extract the features of the left and right images of the stereo image respectively; using a cross-modal feature migration module to fuse the features of the left and right images; when reconstructing the image of the visible light modality of the left view of the stereo image, the image spatial information of the visible light modality is used, combined with the rich details contained in the near-infrared image of the corresponding right view to assist the super-resolution process of the left image, thereby generating more high-frequency detail textures in the image and restoring a more realistic high-resolution image. Experimental results show that the present invention can effectively improve the resolution of visible light modality images, enhance the subjective visual effect of the image, and the objective accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image editing, and in particular relates to a cross-modal stereo image super-resolution reconstruction method. Background Art

[0002] In recent years, with the popularity of multi-camera cross-spectral imaging systems on many modern RGBD devices, cross-modal stereo images have received widespread attention. Cross-modal stereo images, including visible light modality images and near-infrared modality images, can be obtained through visible light-near-infrared cameras that have been embedded in imaging devices such as Kinect and iPhone X, and play an important role in many scientific research and practical application fields, such as face recognition, target detection, vehicle counting, scene parsing, etc. These fields usually require high-quality images to improve the performance of specific different tasks and methods.

[0003] As a research branch of image quality enhancement technology, image super-resolution reconstruction is a modern image processing technology with high scientific research value and a wide range of applications. Image resolution can usually measure the quality of an image, that is, the higher the resolution, the clearer the details and the higher the image quality, which can provide richer and more accurate information. However, due to the inevitable limitations or interference factors in the acquisition, storage and transmission process, the quality of the image is affected by varying degrees of degradation. Usually, the most direct way to obtain high-resolution images is to use a high-resolution camera, but due to cost issues, many applications do not have the conditions to use high-resolution cameras in actual situations. Image super-resolution technology uses a signal processing-based method to improve image resolution. It is an effective way to improve image resolution and image performance, and this method is low-cost. Therefore, the research on efficient and high-quality image super-resolution technology is more important. At present, by reconstructing new high-resolution images with richer details from low-quality images, image super-resolution technology has shown great application potential in biomedicine, aviation research, military applications, image monitoring, microscopic imaging, virtual reality and other fields. Therefore, in a multi-camera cross-spectral imaging system, image super-resolution technology can be used as a preprocessing to improve image quality, thereby enhancing the effect of subsequent tasks, and has very important application value and research prospects.

[0004] Near-infrared images usually have better brightness contrast and rich texture details than visible images. This rich information helps to restore high-frequency details in visible light images. Therefore, it is of great significance to use the reference of near-infrared images to guide the super-resolution of visible light images. However, for multi-camera cross-spectral imaging systems, near-infrared images and visible images are collected at different perspectives, and it is difficult to directly use the near-infrared image of the right perspective to guide the generation of texture information of the visible light image of the left perspective. The visible light image super-resolution reconstruction technology guided by near-infrared images can utilize the image information of different perspectives and different modalities in the multi-camera cross-spectral imaging system, thereby assisting the image super-resolution reconstruction, and therefore has great practical application significance and research value. Summary of the invention

[0005] The purpose of the present invention is to provide a cross-modal stereo image super-resolution reconstruction method guided by near-infrared images to improve the resolution of visible light modality images and enhance the subjective visual effect and objective accuracy of the images.

[0006] The near-infrared image-guided cross-modal stereo image super-resolution reconstruction method provided by the present invention mainly processes low-resolution visible light images of a multi-camera cross-spectral imaging system, and constructs an image super-resolution reconstruction convolutional neural network. It simultaneously utilizes cross-modal and cross-view image information to enhance the image resolution in the multi-camera cross-spectral imaging system, and reconstructs high-quality high-resolution visible light images. Figure 1 The specific steps are as follows:

[0007] (1) Cross-view matching:

[0008] The left and right views of different modalities of low-resolution stereo images (I left ,NIR right ) are input into the cross-view matching module (CVM) for cross-view matching. Specifically, the left and right views of different modalities are aligned to the same view, that is, the near-infrared image of the right view is aligned to the left view, and the result of aligning the right view image to the left view is obtained (NIR left ), expressed as:

[0009] NIR left =CVM(I left ,NIR right ); (1)

[0010] (2) Super-resolution feature extraction of left and right images:

[0011] The low-resolution left view and the right view aligned to the left perspective are respectively input into the two super-resolution branches of the image super-resolution reconstruction convolutional neural network model constructed by the present invention, and a convolutional layer and a residual block are used to represent the image super-resolution features of different modal views. It is expressed as:

[0012]

[0013] (3) Left and right view feature fusion:

[0014] The extracted visible light modality image features of the left view and the near infrared modality image features of the right view are Input into the cross-modal feature transfer (CMFT) module for cross-modal feature transfer; fuse images of different modalities to obtain new image features The near-infrared image information is effectively transferred to the visible light image features through feature fusion, so that the features fused with the near-infrared modality image information generate richer texture and other detail information of the visible light image modality, which is expressed as;

[0015]

[0016] (4) Left and right view super-resolution:

[0017] The image features of different modalities are integrated and near infrared image features Input the convolutional layers and residual blocks of the two super-resolution branches to obtain new image features And input it again into the cross-modal feature transfer module; repeat the above cross-modal feature transfer module and the calculation of the convolution layer and residual block to obtain the final image features that integrate different modal information and near infrared modality image features Finally, a convolutional layer and a deconvolutional layer are used to calculate the visible light image of the left view after super resolution. And the super-resolution left-view near-infrared image The features of near-infrared image information are integrated to help visible light images recover richer texture and other detail information, which is expressed as;

[0018]

[0019] In the present invention, the specific process of performing cross-view matching in step (1) is as follows:

[0020] First, the low-resolution cross-modal stereo image is fed into a cross-modal disparity prediction network (DPN), which outputs a cross-modal disparity, expressed as:

[0021] disparity=DPN(NIR right ), (5)

[0022] Secondly, the near infrared image of the right view is aligned to the left view according to the parallax, and the near infrared image NIR of the left view is obtained. left , expressed as:

[0023] NIR left =Warp(NIR right ,disparity), (6)

[0024] Specifically, the image alignment method is to align the near infrared image (NIR right ) is horizontally offset according to the disparity value of the corresponding position in the disparity map, and aligned to the left perspective to obtain the near infrared image (NIR left ), the formula is:

[0025] NIR left (i,j)=NIR right (i,y-disparity(i,j)), (7)

[0026] In the present invention, the specific process of performing cross-modal feature migration described in step (3) is as follows:

[0027] First, a convolutional layer is used to transform the input visible light image features (f VIS ) calculates a spatial weight map (w s ) is used to represent the spatial area that needs to be enhanced in the visible light image, which is expressed as:

[0028] w s =Conv(f VIS ), (8)

[0029] Secondly, learn a channel-dimensional weight vector (w c ) to assign different importance to the image features of different modalities. NIR ) and visible image features (f VIS ) are cascaded and use a global average pooling layer, convolution layer, and Sigmoid activation layer to learn the weights w c Then, w c Split into two vectors of the same dimension (w c1 ,wc2 ), respectively used to rescale the image features f NIR and f VIS The characteristic value range of is expressed as:

[0030]

[0031] Finally, using the spatial weight map (w s ) and the channel weight vector (w c1 ,w c2 ) Weighted combination f NIR and f VIS Obtain the visible light image features (f VIS '), the formula is:

[0032] f VIS '=w s × c1 ×f NIR +w s × c2 ×f VIS , (10)

[0033] In the present invention, step (4) requires iterative calculation of the cross-modal feature migration module and the convolutional layer and residual block, and the specific steps are:

[0034] First, the image super-resolution reconstruction convolutional neural network model extracts the visible light image features obtained in step (3). and near infrared image features Input the cross-modal feature transfer module to obtain image features that integrate information from different modalities It is expressed as:

[0035]

[0036] Then, a convolutional layer and a residual block are used to calculate and process and Get new visible light image features and near infrared image features And input it into the cross-modal feature transfer module to output image features Respectively expressed as:

[0037]

[0038] Repeat this step once to obtain the image features that finally integrate different modal information. and near infrared modality image features Respectively expressed as:

[0039]

[0040] Finally, a convolutional layer and a deconvolutional layer are used to calculate the super-resolution left-view visible light image. And the super-resolution left-view near-infrared image Respectively expressed as:

[0041]

[0042] The present invention uses a cross-view matching module to extract the disparity between the left and right images of different modalities of a stereo image, and then aligns the near-infrared image of the right view to the left view according to the disparity; the image super-resolution reconstruction network uses two network branches to extract the features of the left and right images of the stereo image respectively; a cross-modal feature transfer module is used to fuse the features of the left and right images; when reconstructing the image of the visible light modality of the left view of the stereo image, not only the image spatial information of the visible light modality is used, but also the rich details contained in the near-infrared image corresponding to the right view are combined to assist the super-resolution process of the left image, thereby generating more high-frequency detail textures in the visible light image and restoring a more realistic high-resolution image. The experimental results show that the method of the present invention can effectively utilize the image information of different modalities of the cross-modal stereo image to improve the resolution of the visible light modality image, generate rich image content, and enhance the subjective visual effect and objective accuracy of the image. In addition, the high-resolution image reconstructed by this method can effectively improve the effect of subsequent tasks in the multi-camera cross-spectral imaging system, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the present invention.

[0044] Figure 2 This is a structural diagram of the cross-view matching module used in the present invention.

[0045] Figure 3 This is a structural diagram of the cross-modal feature transfer module adopted by the present invention.

[0046] Figure 4 This is the result of super-resolution of low-resolution visible light images using this method. DETAILED DESCRIPTION

[0047] For a pair of low-resolution stereo images, super-resolution processing is performed. The specific process is as follows Figure 1 The specific steps are:

[0048] First, the visible light image of the left view of the low-resolution cross-modal stereo image (I left ) and the near infrared image (NIR right ) Input as Figure 2 The cross-view matching module (CVM) shown in the figure obtains the result of aligning the right view image to the left view (NIRleft ):

[0049] NIR left =CVM(I left ,NIR right );

[0050] Secondly, the low-resolution left view I left and the right view NIR aligned to the left view left Input two branches of the image super-resolution network, each branch uses a convolutional layer and residual block to represent the image super-resolution features of different modal views:

[0051]

[0052] Then, the extracted visible light modality image features of the left view are Image features of near infrared modality of right view Input to the cross-modal feature transfer (CMFT) module to obtain new image features

[0053]

[0054] Use a convolutional layer and a residual block to calculate and process separately and Get new visible light image features and near infrared image features And input it into the cross-modal feature transfer module to output image features

[0055]

[0056] Repeat this step once to obtain the image features that finally integrate different modal information. and near infrared modality image features

[0057]

[0058] Finally, a convolutional layer and a deconvolutional layer are used to calculate the super-resolution left-view visible light image. And the near-infrared image of the left view after super-resolution

[0059]

[0060] Figure 4is an experimental example of the present method. Among them, (a) is the input low-resolution visible light image, (b) is the high-resolution visible light image reconstructed using the method of the present invention, and (c) is the real high-resolution visible light image. It can be seen that the method of the present invention can effectively use the image information of the near-infrared modality to assist in restoring clear image textures and edges, generate real and richer visible light image information that is closer to the original high-resolution visible light image, and reconstruct a high-resolution visible light image with higher visual quality.

[0061] References:

[0062] [1]DSJeon, S.Baek, I.Choi, and MHKim.2018.Enhancing the SpatialResolution of Stereo Images Using a Parallax Prior.In IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR).1721–1730.

[0063] [2]L.Wang,Y.Wang,Z.Liang,Z.Lin,J.Yang,W.An,and Y.Guo.2019.Learningparallax attention for stereo image super-resolution.In IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).12250–12259.

[0064] [3]C.Duan and N.Xiao.Parallax-Based Spatial and Channel Attention for Stereo Image Super-Resolution.IEEE Access 7(2019),183672–183679.

[0065] [4]C.Guo,C.Li,Jichang,R.Cong,H.Fu,and Ping.Hierarchical FeaturesDriven Residual Learning for Depth Map Super-Resolution.IEEE Transactions onImage Processing A Publication of the IEEE Signal Processing Society(2018).

[0066] [5]X.Song,Y.Dai,and X.Qin.Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis.IEEE Transactions on Circuits and Systemsfor Video Technology 29,8(2019),2323–2336.

[0067] [6]Y.Wen,B.Sheng,P.Li,W.Lin,and D.D.Feng.Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution.IEEETransactions on Image Processing 28,2(2019),994–1006。

Claims

1. A near-infrared image-guided cross-modal stereo image super-resolution reconstruction method, characterized in that: For the low-resolution visible light images in the multi-camera cross-spectral imaging system, an image super-resolution reconstruction convolutional neural network is constructed. At the same time, cross-modal and cross-view image information is used to enhance the image resolution in the multi-camera cross-spectral imaging system and reconstruct high-quality high-resolution visible light images. The specific steps are as follows: (1) Cross-view matching: The left and right views of different modalities of the low-resolution stereo image are left ,NIR right The two images are input into the cross-view matching module (CVM) for cross-view matching; the left and right views of different modalities are aligned to the same view, that is, the near-infrared image of the right view is aligned to the left view, and the result NIR image of the right view is aligned to the left view is obtained. left , expressed as: NIR left =CVM(I left ,NIR right ); (1) (2) Super-resolution feature extraction of left and right images: The low-resolution left view and the right view aligned to the left perspective are respectively input into the two super-resolution branches of the constructed image super-resolution reconstruction convolutional neural network model, where a convolutional layer and a residual block are used to represent the extracted image super-resolution features of different modal views. It is expressed as: (3) Left and right view feature fusion: The extracted visible light modality image features of the left view and the near infrared modality image features of the right view are The image is input into the cross-modal feature transfer (CMFT) module for cross-modal feature transfer; the image information of different modalities is integrated to obtain new image features. The near-infrared image information is effectively transferred to the visible light image features through feature fusion, so that the features fused with the near-infrared modality image information generate richer texture detail information of the visible light image modality, which is expressed as; (4) Left and right view super-resolution: The image features of different modalities are integrated and near infrared image features Input the convolutional layers and residual blocks of the two super-resolution branches to obtain new image features And input it again into the cross-modal feature transfer module; repeat the above cross-modal feature transfer module and the calculation of the convolution layer and residual block to obtain the final image features that integrate different modal information and near infrared modality image features Finally, a convolutional layer and a deconvolutional layer are used to calculate the visible light image of the left view after super-resolution. And the near-infrared image of the left view after super-resolution The features of near-infrared image information are integrated to help visible light images recover richer texture detail information, which is expressed as; 2. The near-infrared image-guided cross-modal stereo image super-resolution reconstruction method according to claim 1, characterized in that: The specific process of performing cross-view matching in step (1) is as follows: First, the low-resolution cross-modal stereo image is fed into a cross-modal disparity prediction network (DPN) to output a cross-modal disparity disparity, expressed as: disparity=DPN(NIR right ), (5) Secondly, the near infrared image of the right view is aligned to the left view according to the parallax, and the near infrared image NIR of the left view is obtained. left , expressed as: NIR left =Warp(NIR right ,disparity), (6) Image alignment method: Align the near infrared image NIR of the right view right Each pixel point (i, j) is horizontally offset according to the disparity value of the corresponding position in the disparity map, and aligned to the left perspective to obtain the near infrared image NIR of the left perspective. left : NIR left (i,j)=NIR right (i,j-disparity(i,j)), (7)。 3. The near-infrared image-guided cross-modal stereo image super-resolution reconstruction method according to claim 2, characterized in that: The specific process of performing cross-modal feature transfer described in step (3) is as follows: First, a convolutional layer is used to transform the input visible light image features f VIS Calculate a spatial weight map w s To represent the spatial area that needs to be enhanced in the visible light image: w s =Conv(f VIS ) , (8) Secondly, learn a channel-dimensional weight vector w c To assign different importance to image features of different modalities; Specifically, the near infrared image feature f NIR and visible light image feature f VIS Cascaded and use a global average pooling layer, convolution layer and Sigmoid activation layer to learn the weight w c ; Then, w c Split into two vectors w of the same dimension c1 ,w c2 , respectively used to rescale the image features f NIR and f VIS The characteristic value range of is expressed as: Finally, using the spatial weight map w s and the channel weight vector w c1 ,w c2 Weighted combination f NIR and f VIS , and obtain the visible light image feature f that integrates the near-infrared modality image information VIS ', the formula is: f VIS '=w s ×w c1 ×f NIR +w s ×w c2 ×f VIS , (10)。 4. The near-infrared image-guided cross-modal stereo image super-resolution reconstruction method according to claim 3, characterized in that: Step (4) requires iterative calculation of the cross-modal feature transfer module and the convolutional layer and residual block, and the specific steps are: First, the image super-resolution convolutional neural network is used to reconstruct the visible light image features extracted by step (3). and near infrared image features Input the cross-modal feature transfer module to obtain image features that integrate information from different modalities It is expressed as: Then, a convolutional layer and a residual block are used to calculate and process and Get new visible light image features and near infrared image features And input it into the cross-modal feature transfer module to output image features It is expressed as: Repeat this step once to obtain the image features that finally integrate different modal information. and near infrared modality image features Respectively expressed as: Finally, a convolutional layer and a deconvolutional layer are used to calculate the super-resolution left-view visible light image. And the near-infrared image of the left view after super-resolution Respectively expressed as:

Images