Fisheye distortion correction image super-resolution method in combination with spatial variation characteristics

Abstract

The invention discloses a fisheye distortion correction image super-resolution method in combination with spatial variation characteristics, the method constructs a fisheye distortion correction image super-resolution reconstruction model in combination with spatial variation characteristics, and the fisheye distortion correction image super-resolution reconstruction model comprises a pixel displacement matrix generation module and a super-resolution reconstruction module, the pixel displacement matrix generation module calculates the position difference of each pixel in the fisheye distortion correction image before and after fisheye distortion by using a fisheye distortion model according to the fisheye distortion parameter estimation result, and generates a pixel displacement matrix as resolution space change information, and the super-resolution reconstruction module comprises a coding sub-network and a decoding sub-network. The module aggregates the pixel displacement matrix and the fisheye distortion correction image, generates an aggregation vector as input, and finally generates a super-resolution reconstruction result of the fisheye distortion correction image. According to the method, fisheye distortion characteristics and resolution space change information can be fully mined, and the resolution of the fisheye distortion correction image is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to image processing technology, in particular to a fisheye distortion correction image super-resolution method combined with spatial variation characteristics. BACKGROUND

[0002] The traditional fisheye image distortion correction model only uses the estimated distortion parameters to perform inverse transformation on the fisheye image, and the generated distortion correction image has the problem of reduced resolution. Unlike the general image resolution reduction problem, the degree of resolution reduction of the fisheye distortion correction image is spatially variable. When forming a fisheye image, the direction and size of each pixel displacement determine the resolution reduction process at that pixel position. Overall, the farther away from the center of the fisheye image, the more serious the resolution reduction. Therefore, the super-resolution reconstruction of the fisheye distortion correction image is significantly different from the general super-resolution reconstruction task. How to effectively improve the resolution of the fisheye distortion correction image has become one of the important problems in the field of image processing and computer vision, but the existing deep learning method cannot fully consider the spatial variation characteristics of the fisheye distortion correction image resolution reduction. Therefore, how to propose a more efficient super-resolution reconstruction method for the spatial variation characteristics of the fisheye distortion correction image resolution reduction is a problem to be solved in the super-resolution reconstruction of the fisheye distortion correction image. SUMMARY

[0003] Therefore, it is necessary to provide a fisheye distortion correction image super-resolution method combined with spatial variation characteristics to improve the training effect by automatically learning different characteristics of a large number of data samples, so as to continuously optimize the model and improve the image resolution.

[0004] The fisheye distortion correction image super-resolution method combined with spatial variation characteristics comprises the following steps:

[0005] Step 1: Construct a fisheye distortion correction image super-resolution reconstruction simulation data set. Each sample in the data set includes a fisheye distortion correction image, corresponding fisheye distortion parameters, and a corresponding fisheye distortion correction super-resolution reconstruction image. The fisheye distortion parameters include fisheye center coordinates and distortion polynomial coefficients [1]. The fisheye image and the distortion correction image have a size of HxWx3, where H and W represent the height and width (pixel number) of the image, respectively, and the channel number is 3.

[0006] Step two: construct a fisheye distortion correction image super-resolution reconstruction model combined with spatial variation characteristics, including a pixel displacement matrix generation module and a super-resolution reconstruction module. The pixel displacement matrix generation module calculates the position difference of each pixel in the fisheye distortion correction image before and after fisheye distortion according to the fisheye distortion parameter estimation result, generates a pixel displacement matrix as resolution spatial variation information, and uses a fisheye distortion model. The super-resolution reconstruction module includes an encoding subnetwork and a decoding subnetwork. The module aggregates the pixel displacement matrix and the fisheye distortion correction image to generate an aggregated vector as input, and finally generates a fisheye distortion correction image super-resolution reconstruction result.

[0007] wherein the size of the pixel displacement matrix R is HxWx2, wherein H and W represent the number of rows and columns (pixels) of the pixel displacement matrix, respectively. Let the pixel [i,j] (i,j are the row and column numbers of the pixel, respectively) in the fisheye distortion correction image correspond to the pixel coordinates [i',j'] in the fisheye image, then the values of the elements R(i,j,1) and R(i,j,2) in the pixel displacement matrix are i-i' and j-j', respectively. The super-resolution reconstruction module includes an encoding subnetwork and a decoding subnetwork. The input of the encoding subnetwork is the aggregated vector of the pixel displacement matrix and the fisheye distortion correction image. The subnetwork is used to extract the resolution spatial variation characteristics and image structure characteristics in the input data. The encoding subnetwork includes 6 convolutional layers, and the output feature size of each layer is represented as heightxwidthxchannel number. The output feature map sizes of the convolutional layers in the encoding subnetwork are HxWx32, H / 2xW / 2x64, H / 4xW / 4x128, H / 8xW / 8x256, H / 16xW / 16x512, and H / 32xW / 32x512, respectively. The decoding subnetwork decodes the features extracted by the encoding subnetwork to generate a fisheye distortion correction image super-resolution reconstruction result. The decoding subnetwork includes 6 convolutional layers, and the output feature map sizes of the convolutional layers are H / 16xW / 16x512, H / 8xW / 8x256, H / 4xW / 4x128, H / 2xW / 2x64, HxWx32, and HxWx3, respectively. A connection is added between the encoding subnetwork and the decoding subnetwork to further integrate the resolution spatial variation characteristics and image structure characteristics information extracted by the encoding subnetwork into the decoding subnetwork. The decoding subnetwork finally outputs a fisheye distortion correction super-resolution reconstruction image with a size of HxWx3.

[0008] Step three: use the fisheye distortion correction image super-resolution reconstruction simulation dataset constructed in step one to train the proposed model. Specifically, the Caffe, Tensorflow, or Pytorch deep learning framework can be used for training.

[0009] Step four: using the trained model, the test fisheye distortion correction image is super-resolution reconstructed, and the super-resolution reconstruction result of the fisheye distortion correction image is generated.

[0010] According to the fisheye distortion parameters, the displacement amount of each pixel in the fisheye distortion correction image is calculated by using the fisheye distortion model, and a pixel displacement matrix ΔP is formed, which is defined as:

[0011] ΔP(i,j)=p distorted (i,j)-p corrected (i,j)

[0012] Wherein, p distorted (i,j) represents the pixel position before fisheye distortion, p corrected (i,j) represents the pixel position after correction, (i,j) is the image pixel coordinate;

[0013] The distortion parameter adaptive optimization module based on deep learning is introduced, and through the diversified fisheye parameters in the training data set, a more accurate displacement prediction function is realized:

[0014]

[0015] Wherein, f θ Indicates the parameterized deep network, outputs the optimized pixel displacement vector, and improves the correction accuracy;

[0016] The pixel displacement matrix and the distortion correction image are spliced or fused according to the channel to generate an aggregation vector X input , which facilitates the coding subnetwork to extract spatial changes and image structure information at the same time:

[0017]

[0018] Wherein, 3 represents the RGB channel number, 2 represents the horizontal and vertical displacement channels of the displacement matrix, H and W represent the height and width of the image respectively, and Concat represents the splicing operation.

[0019] The spatial and channel attention mechanism is embedded in the coding subnetwork of the super-resolution reconstruction module, and residual connection, spatial attention module and channel attention module are used to enhance the coding subnetwork's ability to capture resolution space change features:

[0020] F l =Conv l (F l-1 )+F l-1

[0021] F l ′ =Attention(F l )=CA(SA(F l))

[0022] where CA is channel attention, SA is spatial attention, l is the convolution layer number, Conv l is the l-th layer convolution operation, Attention denotes the attention module, F l denotes the output of the l-th convolution layer, F l-1 denotes the output of the l-1-th convolution layer, F l ′ denotes the output of the spatial attention module and the channel attention module after enhancement.

[0023] The encoding sub-network extracts features hierarchically in scale, and combines a multi-scale feature fusion mechanism to ensure image detail recovery and spatial continuity in multiple scales. The multi-scale feature set of the encoding sub-network is defined as:

[0024]

[0025] Through upsampling and fusion function Φ, the features are fused into the encoding sub-network:

[0026]

[0027] wherein, is the feature corresponding to the decoding layer;

[0028] The residual block and the upsampling module of the decoding sub-network improve the detail reconstruction effect, and realize the fusion of spatial change features:

[0029]

[0030] wherein, Ψ denotes the output mapping layer of the decoding sub-network, and the output size is HxWx3.

[0031] The entire super-resolution reconstruction process is defined as:

[0032]

[0033] wherein, denotes the super-resolution reconstruction deep network with parameters θ, and the input is the aggregated fisheye corrected image and the optimized pixel displacement matrix;

[0034] During training, a comprehensive loss function is defined, which comprehensively considers the overall pixel level accuracy loss, the perceptual quality loss and the edge detail recovery loss.

[0035] The present application provides a fisheye distortion correction image super-resolution method combined with spatial variation characteristics, first generates a pixel displacement matrix using fisheye distortion parameters, and uses the matrix as resolution spatial variation information to guide the subsequent super-resolution reconstruction process. On this basis, the resolution spatial variation characteristics and image structure characteristics in the input data are extracted using the encoding sub-network, and the characteristics are decoded using the decoding sub-network to generate the fisheye distortion correction image super-resolution reconstruction result. The present application can fully consider the resolution reduction characteristics of the fisheye distortion correction image spatial variation, effectively improve the resolution of the fisheye distortion correction image, restore the details in the image, and provide higher quality input data for subsequent image analysis. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the method of the present application;

[0037] Figure 2 is the overall architecture of the model proposed by the present application;

[0038] Figure 3 is the network structure of the super-resolution reconstruction module;

[0039] Figure 4 is the fisheye distortion correction image super-resolution reconstruction result. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below in conjunction with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0041] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0042] As shown in Figure 1 , the fisheye distortion correction image super-resolution method combined with spatial variation characteristics provided by the present embodiment includes the following steps:

[0043] Step one: construct a fisheye distortion correction image super-resolution reconstruction simulation data set. Each sample in the data set includes a fisheye distortion correction image, corresponding fisheye distortion parameters and corresponding fisheye distortion correction super-resolution reconstruction image. Among them, the fisheye distortion parameters include fisheye center coordinates and distortion polynomial coefficients [1]. The fisheye image and the distortion correction image have a size of HxWx3, where H and W represent the height and width (pixel number) of the image, respectively, and the channel number is 3. In specific embodiments, H=W=128 is set.

[0044] Step two: construct a fisheye distortion correction image super-resolution reconstruction model combined with spatial variation characteristics, including a pixel displacement matrix generation module and a super-resolution reconstruction module, as shown in Figure 2 . Among them, the pixel displacement matrix generation module calculates the position difference of each pixel in the fisheye distortion correction image before and after fisheye distortion according to the fisheye distortion parameter estimation result, generates a pixel displacement matrix as the resolution spatial variation information, and uses a fisheye distortion model. The super-resolution reconstruction module includes an encoding subnetwork and a decoding subnetwork, as shown in Figure 3 . This module aggregates the pixel displacement matrix and the fisheye distortion correction image to generate an aggregated vector as input, and finally generates a fisheye distortion correction image super-resolution reconstruction result.

[0045] Among them, the size of the pixel displacement matrix R is HxWx2, where H and W represent the number of rows and columns (pixels) of the pixel displacement matrix, respectively. Let the pixel [i,j] (i,j are the row and column numbers of the pixel, respectively) in the fisheye distortion correction image correspond to the pixel coordinates [i',j'] in the fisheye image, then the values of the elements R(i,j,1) and R(i,j,2) in the pixel displacement matrix are i-i' and j-j', respectively. The super-resolution reconstruction module includes an encoding subnetwork and a decoding subnetwork. Among them, the input of the encoding subnetwork is the aggregated vector of the pixel displacement matrix and the fisheye distortion correction image. This subnetwork is used to extract the resolution spatial variation characteristics and image structure characteristics in the input data. The encoding subnetwork includes 6 convolutional layers, and the output feature size of each layer is represented as heightxwidthxchannel number. The output feature map sizes of the convolutional layers in the encoding subnetwork are HxWx32, H / 2xW / 2x64, H / 4xW / 4x128, H / 8xW / 8x256, H / 16xW / 16x512, H / 32xW / 32x512, respectively. The decoding subnetwork decodes the features extracted by the encoding subnetwork to generate a fisheye distortion correction image super-resolution reconstruction result. The decoding subnetwork includes 6 convolutional layers, and the output feature map sizes of each convolutional layer are H / 16xW / 16x512, H / 8xW / 8x256, H / 4xW / 4x128, H / 2xW / 2x64, HxWx32, HxWx3, respectively. The connection (as shown by the dashed arrow in Figure 3 ) between the encoding subnetwork and the decoding subnetwork further integrates the resolution spatial variation characteristics and image structure characteristics information extracted by the encoding subnetwork into the decoding subnetwork. The decoding subnetwork finally outputs a fisheye distortion correction super-resolution reconstruction image with a size of HxWx3, as shown in Figure 4 .

[0046] Step three: use the fisheye distortion correction image super-resolution reconstruction simulation data set constructed in step one to train the proposed model. Specifically, the Caffe, Tensorflow or Pytorch deep learning framework can be used for training.

[0047] Step four: use the trained model to perform super-resolution reconstruction on the test fisheye distortion correction image, which can quickly generate the super-resolution reconstruction result of the fisheye distortion correction image, as shown in Figure 4 The proposed model can fully consider the resolution reduction characteristics of spatial changes in the fisheye distortion correction image, effectively improve the resolution of the fisheye distortion correction image, and restore the details in the image.

[0048] Specifically, according to the fisheye distortion parameters, the displacement amount of each pixel in the fisheye distortion correction image is calculated using the fisheye distortion model to form a pixel displacement matrix ΔP, which is defined as:

[0049] ΔP(i,j)=p distorted (i,j)-p corrected (i,j)

[0050] Where p distorted (i,j) represents the pixel position before fisheye distortion, p corrected (i,j) represents the pixel position after correction, and (i,j) is the image pixel coordinate.

[0051] The distortion parameter adaptive optimization module based on deep learning is introduced, and through the diversified fisheye parameters in the training data set, a more accurate displacement prediction function is realized:

[0052]

[0053] Where f θ represents a parameterized deep network, and the optimized pixel displacement vector is output to improve the correction accuracy.

[0054] The pixel displacement matrix and the distortion correction image are spliced or fused according to the channel to generate an aggregated vector X input , which facilitates the coding subnetwork to extract spatial changes and image structure information at the same time:

[0055]

[0056] Where 3 represents the RGB channel number, 2 represents the horizontal and vertical displacement channels of the displacement matrix, H and W represent the height and width of the image respectively, and Concat represents the splicing operation.

[0057] The super-resolution reconstruction module encodes the sub-network in the embedding space and channel attention mechanism, adopts residual connection, spatial attention module and channel attention module to enhance the capture ability of the encoding sub-network to the resolution space change characteristics:

[0058] F l = Conv l (F l-1 )+F l-1

[0059] F l ′ = Attention(F l )= CA(SA(F l ))

[0060] Where CA is the channel attention, SA is the spatial attention, l is the convolution layer number, Conv l is the lth convolution operation, Attention represents the attention module, F l represents the output of the lth convolution layer, F l-1 represents the output of the (l-1)th convolution layer, F l ′ represents the output of the spatial attention module and the channel attention module after enhancement.

[0061] The encoding sub-network extracts features hierarchically in scale, combines multi-scale feature fusion mechanism to ensure image detail recovery and spatial continuity in multiple scales, and defines the multi-scale feature set of the encoding sub-network as:

[0062]

[0063] Through upsampling and fusion function Φ, the features are fused into the encoding sub-network:

[0064]

[0065] Where, is the decoding layer corresponding feature;

[0066] The residual block and layer-by-layer feature integration upsampling module in the decoding sub-network improves the detail reconstruction effect, and realizes the fusion of spatial change characteristics:

[0067]

[0068] Where, Ψ represents the output mapping layer of the decoding sub-network, and the output size is H×W×3.

[0069] The whole super-resolution reconstruction process is defined as:

[0070]

[0071] wherein, denotes a super-resolution reconstruction depth network with parameter θ, and the input is the aggregated fisheye corrected image and the optimized pixel displacement matrix;

[0072] During training, a comprehensive loss function is defined, which comprehensively considers the overall pixel level accuracy loss, the perceptual quality loss and the edge detail recovery loss.

[0073] The technical features of the above embodiments can be combined in any manner. To make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure

[0074] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

Claims

1. An image super-resolution method for fisheye distortion correction combined with spatial variation characteristics, characterized in that, The method comprises the following steps: Step 1: Construct a fisheye distortion correction image super-resolution reconstruction simulation data set, each sample in the data set comprising a fisheye distortion correction image, corresponding fisheye distortion parameters and a fisheye distortion correction super-resolution reconstruction image; Step 2: Construct a fisheye distortion correction image super-resolution reconstruction model combining spatial variation characteristics, comprising a pixel displacement matrix generation module and a super-resolution reconstruction module, wherein the pixel displacement matrix generation module estimates the fisheye distortion parameters, uses a fisheye distortion model to calculate the position difference of each pixel in the fisheye distortion correction image before and after fisheye distortion, and generates a pixel displacement matrix as resolution spatial variation information; the super-resolution reconstruction module comprises an encoding subnetwork and a decoding subnetwork, aggregates the pixel displacement matrix and the fisheye distortion correction image to generate an aggregated vector as input, and finally generates a fisheye distortion correction image super-resolution reconstruction result; Step 3: Use the fisheye distortion correction image super-resolution reconstruction simulation data set constructed in step 1 to train the model; Step 4: Use the trained model to perform super-resolution reconstruction on a test fisheye distortion correction image to generate a fisheye distortion correction image super-resolution reconstruction result.

2. The fish-eye distortion correction image super-resolution method incorporating spatial variation characteristics according to claim 1, wherein, According to the fisheye distortion parameters, the displacement of each pixel in the fisheye distortion correction image is calculated using the fisheye distortion model to form a pixel displacement matrix ΔP, which is defined as: ΔP(i,j) = p distorted (i,j) - p corrected (i,j) where p distorted (i,j) represents the pixel position before fisheye distortion, p corrected (i,j) represents the pixel position after correction, (i,j) is the image pixel coordinate; A distortion parameter adaptive optimization module based on deep learning is introduced to achieve a more accurate displacement prediction function through diversified fisheye parameters in the training data set: ΔP = f θ (p corrected ,k,c) where f θ represents a parameterized deep network, outputting the optimized pixel displacement vector, improving the correction accuracy; concatenate or fuse the pixel displacement matrix and the distortion corrected image by channel to generate an aggregated vector X input facilitate the encoding subnetwork to extract spatial variation and image structure information simultaneously: Where 3 represents the RGB channel number, 2 represents the horizontal and vertical displacement channels of the displacement matrix, H and W represent the height and width of the image respectively, and Concat represents the concatenation operation. 3.The method of claim 2, wherein, The super-resolution reconstruction module embeds a spatial and channel attention mechanism in the encoding subnetwork, uses residual connection, spatial attention modules and channel attention modules to enhance the encoding subnetwork's ability to capture resolution spatial variation characteristics: F l = Conv l (F l-1 )+F l-1 F l ′ = Attention(F l ) = CA(SA(F l )) wherein CA is channel attention, SA is spatial attention, l is the convolution layer number, Conv l is the lth convolution operation, Attention denotes the attention module, F l is the output of the lth convolution layer, F l-1 is the output of the (l-1)th convolution layer, F l ′ is the output of the spatial attention module and the channel attention module after enhancement. 4.The method of claim 2, wherein, The encoding subnetwork extracts features hierarchically in scale, combines a multi-scale feature fusion mechanism to ensure image detail recovery and spatial continuity in multiple scales, and defines the multi-scale feature set of the encoding subnetwork as: The features are fused to the encoding subnetwork through upsampling ↑ and a fusion function Φ: wherein, is a decoded layer corresponding feature; The decoding subnetwork designs a residual block and an upsampling module that integrates features layer by layer to improve the detail reconstruction effect, and realizes the fusion of spatial variation characteristics: Where Ψ represents the output mapping layer of the decoding subnetwork, and the output size is HxWx3.

5. The fish-eye distortion correction image super-resolution method incorporating spatial variation characteristics according to claim 3 or 4, characterized in that, The entire super-resolution reconstruction process is defined as: wherein, denotes a super-resolution reconstruction depth network with parameter θ, inputting the aggregated fisheye corrected image and the optimized pixel displacement matrix; During training, a comprehensive loss function is defined to consider the overall pixel level accuracy loss, perceptual quality loss and edge detail recovery loss.

6. The method of claim 1, wherein the fish-eye distortion correction image super-resolution method with spatial variation characteristics is characterized by: The input of the encoding subnetwork is the aggregated vector of the pixel displacement matrix and the fisheye distortion correction image, the encoding subnetwork is used to extract resolution spatial variation characteristics and image structure characteristics in the input data, the decoding subnetwork decodes the features extracted by the encoding subnetwork to generate a fisheye distortion correction image super-resolution reconstruction result.

7. The method of claim 6, wherein the method of super-resolution of fisheye distortion corrected image incorporating spatially varying characteristics is characterized by: The output feature map size of each convolutional layer in the encoding subnetwork is HxWx32, H / 2xW / 2x64, H / 4xW / 4x128, H / 8xW / 8x256, H / 16xW / 16x512, H / 32xW / 32x512, respectively, and the output feature map size of each convolutional layer in the decoding subnetwork is H / 16xW / 16x512, H / 8xW / 8x256, H / 4xW / 4x128, H / 2xW / 2x64, HxWx32, HxWx3, respectively, a connection is added between the encoding subnetwork and the decoding subnetwork, and resolution space change features and image structure feature information extracted by the encoding subnetwork are further integrated into the decoding subnetwork, and the decoding subnetwork finally outputs a fisheye distortion correction super-resolution reconstructed image with a size of HxWx3.

Images