Wide-angle image correction method with multi-scale features

Abstract

The invention discloses a multi-scale feature wide-angle image correction method, which comprises the following steps of: firstly, inputting a photo and a corresponding distortion mask, processing through a neural network of which the structure is an encoder, a TPS prediction module and an optical flow estimation module, obtaining image feature points under multiple scales through a decoder module, and obtaining the image feature points under multiple scales through the TPS prediction module. Acquiring thin plate module control points of the features in the scene, and inputting the thin plate module control points into a decoder to obtain a motion change flow; in addition, the features obtain an optical flow field through an optical flow estimation module, the optical flow field and the motion change flow are combined, a local correction direction is obtained, and finally an input original wide-angle deformation image is corrected. Experiments on a data set in the real world show that the wide-angle image correction of the square can be realized, the training and rendering speeds are increased by multiple times, and the evaluation index is better. The method solves the problems of poor correction performance and slow speed under high resolution, and can be widely applied to related applications of image correction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image restoration, and in particular to a wide-angle image correction method with multi-scale features. Background Art

[0002] Wide-angle lenses have been widely used in many fields because they can capture a wider field of view. For example, in the field of photography, wide-angle lenses are often used to shoot landscapes, buildings, portraits, etc. In the fields of machine vision and virtual reality, wide-angle lenses are also favored because of their wide field of view. However, while wide-angle lenses bring a wide field of view, they are often accompanied by image distortion. This distortion mainly comes from the optical properties of the lens. Especially when the lens viewing angle is large, the edge of the image will appear obviously stretched or bent. This distortion not only affects the aesthetics of the image, but may also have an adverse effect on subsequent image processing and analysis.

[0003] With the continuous development of artificial intelligence and big data technology, image correction technology is also facing new opportunities. Although image correction technology has made significant progress, it still faces some challenges. For example, deep learning technology can be used to train more accurate distortion models, or big data can be used to optimize image correction algorithms. In past studies, others added TPS transformation to the convolutional network to correct images, but the final model had poor correction effect on high-resolution images, because the existing image correction algorithms often have poor correction effects on high-resolution distorted images, because they cannot recognize and use the multi-scale features of the image.

[0004] In view of the above problems, the present invention proposes a wide-angle image correction method with multi-scale features, which realizes fast training and corrects distorted images by utilizing a multi-scale feature sampling method supplemented by the direction guidance and constraints of the optical flow field. Summary of the invention

[0005] The present invention discloses a wide-angle image correction method with multi-scale features, which can quickly perform distortion correction on real-time monitoring images of most non-specific square wide-angle monitoring cameras, making the monitored content more detailed and realistic. To achieve the above objectives, the present invention provides a wide-angle image correction method with multi-scale features, comprising the following steps:

[0006] Prepare a set of images and their corresponding masks for network training;

[0007] Train a neural network with the structure of encoder, TPS prediction module and optical flow estimation module.

[0008] The convolutional network in the encoder is used to extract deep features of images at different scales.

[0009] In the TPS prediction module, the thin plate spline control points are extracted from the features to obtain the control point map;

[0010] extracting a motion estimation stream from the thin plate spline control points in a decoder;

[0011] The feature association layer in the optical flow estimation module obtains the feature information through vector inner product, and then constructs the optical flow field in the update layer GRU cycle.

[0012] The motion estimation flow is combined with the optical flow map, and distortion correction and loss calculation are performed with the real image and real optical flow to obtain the trained model.

[0013] The image to be corrected and the mask are input into the trained model for image correction.

[0014] Specifically, first prepare a set of images and their corresponding masks for network training. First, obtain a public data set, which includes normal images and images with various deformations after post-processing for training. Second, prepare a mask image with the same resolution as the original image, set the non-deformed area to black, and the deformed area to white. After preparation, send it to the network for training.

[0015] The encoder structure is responsible for converting the input data into a feature vector containing all the necessary information, and the decoder converts these feature vectors into the target output sequence. In order to facilitate the use of this network in the context of wide-angle images, a multi-scale feature extraction module is also added to the encoder structure to obtain multi-scale features of the image.

[0016] The encoder structure is mainly composed of 3 layers of sliding window attention layers with different dimensions, and the output feature vector dimensions are 4, 2, and 1, respectively, with decreasing dimensions. The decoder structure corresponds to the encoder structure, which is composed of 3 layers of sliding window attention layers with different dimensions, and the output feature vector dimensions are 1, 2, and 4, respectively, with increasing dimensions. The encoding block and the decoding block are connected by skip connections to obtain more scale information in the image restoration process.

[0017] The sliding window attention layer divides the image into blocks and then performs sampling convolution on each block, so that each block can contain part of the information of the surrounding adjacent blocks. The sliding window layer is introduced to enhance the model's ability to capture global information through the window sliding operation between layers.

[0018] In order to obtain the degree of image distortion, the image features obtained by the encoder will be input into the TPS prediction module for TPS (thin plate spline) prediction, and finally the number of thin plate spline control points of the entire image will be obtained. Different numbers of thin plate spline control points represent different degrees of deformation.

[0019] After obtaining the control point map, the control point map is input into the decoder, and the feature vectors of these thin plate spline control points are converted into a target output sequence to obtain a motion estimation stream.

[0020] The image features are input into the feature association layer, generating W*H*W*H 4D association information for all pixels, and obtaining a 4D information body. The information of the 4D information body is input into the cyclic update GRU layer, and the optical flow field is obtained by iterative update.

[0021] The characteristic direction vector of the optical flow map is distorted with the motion estimation flow, and the loss is calculated with the real image to obtain the trained model. Four losses are used for training, namely image reconstruction loss LRe, perception loss LPe, grid loss LGrid and optical flow distance loss LLight.

[0022] The image to be corrected and the mask are input into the trained model for image correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art, and the present invention can be further understood in conjunction with the contents of the drawings. The components shown in the drawings are not necessarily drawn to scale, but the focus is on illustrating the principles of the embodiments.

[0024] Figure 1 It is a schematic diagram of the steps of a wide-angle image correction method with multi-scale features provided by the present invention.

[0025] Figure 2 This is a diagram of the optical flow field network separation structure proposed by the present invention.

[0026] Figure 3 The figures are effect diagrams before and after wide-angle distortion correction provided by an embodiment of the present invention.

[0027] Figure 4 This is a comparison diagram of the effects of other correction methods provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Embodiments of the wide-angle image correction of multi-scale features of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0029] See also Figure 1 The present invention provides a wide-angle image correction method with multi-scale features, comprising the following steps:

[0030] First, prepare the data set and corresponding masks for network training. Specifically, the data set is divided into lossless real images and corresponding images that have been pre-processed with wide-angle magnification or distortion. The corresponding mask has a black background color with the same resolution as the source image, and a white mask corresponding to the distorted or deformed position of the original image.

[0031] The neural network structure is mainly divided into two functional modules. The first functional structure is the TPS prediction module of S102, and the second functional structure is the optical flow estimation module of S103. The encoder of S101 is set with a specific convolution kernel, which is a common feature extraction module. The following is a detailed introduction.

[0032] S101, the encoder structure is mainly composed of 3 layers of sliding window attention layers of different dimensions, and the latitudes of the output feature vectors are 8, 4, and 2 respectively; and the layers are connected with downsampling pooling layers to reduce the dimension and amount of calculation of the data while retaining important feature information. After being processed by the attention layer, the features are extracted by the multi-scale feature extraction module at multiple resolutions. The structure of the multi-scale feature module is a token mixer for spatial feature interaction, a deep convolutional layer, and a two-level MLP (multi-layer perceptron) network for channel feature interaction.

[0033] Specifically, input an image of size H×W×3, perform patch partition, divide the image into H / 4×W / 4×48 patches, perform Linear Embedding, perform linear transformation on the channel data of each pixel, unify the shape of the feature, and construct feature maps of different sizes through three attention layers with different output dimensions of 8, 4, and 2. Then interact with the MLP network, compress feature maps of different sizes and interact with features, and output multi-scale feature maps.

[0034] S102. In the TPS prediction module, convolutional layers with different kernels are used to predict a gradually increasing number of control points, which are set to 10*10, 12*12, and 14*14 respectively. The number of thin plate spline control points gradually increases. The control points predicted by the previous head are upsampled and integrated into the prediction of the next head. Then, these control points are arranged to generate a grid. Next, we use the TPS transformation to distort this grid so that it aligns with the regular grid defined on the ground truth image. In implementation, considering that the series connection of fully connected layers will introduce a lot of computational and storage costs, we use one or two convolutional layers after each TPS transformation head to predict the control points. The control points are calculated as follows:

[0035] q (t) =h (t) [Ρ(s (t-1) ,q (t-1))]+UP[q (t-1) ] (1)

[0037] where h(t) is the t-th TPS transformation layer, s(t-1) and q(t-1) are the feature map and control point of the t-th (t-1) head, respectively. P(·, ·) represents the warping operation of the feature map given the control point, and UP[·] is the customized upsampling layer of the control point.

[0038] After obtaining the thin plate spline control point map, the motion change flow is obtained through the decoder and upsampling layer.

[0039] S103, the multi-scale image features obtained by the encoder are input to the feature association layer, which generates 4D association information of W*H*W*H for all pixels, and uses the confluence layer to generate lower resolution association information to obtain a 4D information body, where W is the width of the input image and H is the height of the input image.

[0040] The information of the 4D information volume is input into the cyclic update GRU layer, and the optical flow field is obtained by iterative update. The update operator estimates a series of optical flow: {f 0 ,f 1 ,f 2 ,....f k}, in each iteration k, it generates a flow update direction Δf and adds it to the current estimate: f k+1 =f k +Δfk.

[0041] Specifically, given the current optical flow estimate, an algorithm is used to retrieve relevant features from the 4D information volume. The relevant features are then processed by 2 convolutional layers. In addition, 2 convolutional layers are used for the optical flow estimate itself to generate optical flow features. Finally, the relevant optical flow and context features are concatenated as the input feature map.

[0042] In the process of training the network, specifically, the feature direction vector of the optical flow map and the motion estimation flow are distorted and corrected, and the loss is calculated with the real image to obtain the trained model. After the feature map is decoded and output by the decoder, it is superimposed with the motion estimation flow formed by the up-sampled interpolated control point map, and then distorted with the optical flow field. The specific calculation formula for the distortion correction process is as follows:

[0043] T=W(D[F]+UP[q (tps) ],S,L) (2)

[0045] Among them, T is the correction target image, F is the feature map, D[·] is the decoder output operation, U[·] is the upsampling interpolation operation, S is the original image, and L is the optical flow map.

[0046] The present invention uses four losses for training, namely image reconstruction loss LRe, perception loss LPe, grid loss LGrid and optical flow loss LLight. The reconstruction loss and perception loss supervise the distorted image at the pixel level and feature level respectively.

[0047] Image reconstruction loss is used to guide the model to generate an image that is as similar as possible to the original image. MSE (mean square error) or MAE (mean absolute error) is usually used to calculate the feature map difference. The present invention uses MSE for calculation, and the specific calculation is as follows:

[0048]

[0049] Where N is the total number of pixels, y i is the pixel value of the real image, y i ' is the pixel value of the generated image.

[0050] The perceptual loss is usually calculated by passing the input image and the target image through a pre-trained neural network to obtain their feature representations in the network. These feature representations are then used as inputs to the loss function to calculate the Euclidean distance or Manhattan distance between them.

[0051] The mesh loss constrains the edges of two consecutive deformed meshes, where M represents the number of tuples of two consecutive edges in mesh m. When maximizing the above cosine representation, the corresponding two edges become collinear. Therefore, the loss reaches a minimum value, ensuring that the image content remains consistent.

[0052]

[0053] The vector e t1 and vector e t2 are the edges of all continuously deformed meshes in the feature map.

[0054] The optical flow distance loss Light is calculated by calculating the L1 distance between the predicted optical flow and the true optical flow with an exponentially growing weight. Given the motion change flow of the true optical flow as D[F], the loss is defined as

[0055]

[0056] Therefore, the final loss function is as follows:

[0057] L=L Re +L Pe +L Grid +L Light (6)

[0059] The image to be corrected and the mask are input into the trained model to perform image correction. The experimental verification is regarded as the final stage of the process of an embodiment of the present invention, and the feasibility and effect of the method proposed by the present invention are verified through actual experiments.

[0060] In the experimental verification stage, the present invention selected other reconstruction methods for quantitative experiments and qualitative experiments. In the quantitative experiment part, the model of the present invention achieved the best in all indicators. The quantitative experimental indicators are shown in Table 1. The calculated indicators are PSNR, SSIM, and parameter. The higher the PSNR and SSIM index values, the better. The lower the parameter index value, the fewer model parameters.

[0061] Table 1 Comparison results of the indicators of the method of the present invention and other methods on the Places2 dataset.

[0062]

[0063] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A wide-angle image correction method based on a transformer neural network, comprising the following steps: Prepare a set of images and their corresponding masks for network training; Train a neural network with an encoder, TPS prediction module, and optical flow estimation module. Use the convolutional network in the encoder to extract deep features of images at different scales. In the TPS prediction module, the features are extracted into thin plate spline control points to obtain a control point map; in the decoder, the motion estimation stream is extracted from the thin plate spline control points; The feature association layer in the optical flow estimation module obtains the feature information through vector inner product, and then constructs the optical flow field in the update layer GRU cycle. The motion estimation flow and optical flow map are combined, and distortion correction and loss calculation are performed with the real image and real optical flow to obtain the trained model. The image to be corrected and the mask are input into the trained model for image correction.

2. The picture set and its corresponding mask prepared for network training as described in claim 1, first, obtain a public data set, which includes normal pictures and images with various deformations after post-processing for training, and secondly, prepare a mask image with the same resolution and size as the original image, set the non-deformed area to black, and the deformed area to white, and then send it to the network training after preparation.

3. As described in claim 1, the convolutional neural network module in the encoder extracts the multi-scale features of the image. The encoder is responsible for converting the input data into a feature vector containing all necessary information.

4. As required by claim 1, in order to obtain the degree of distortion of the image, the image features obtained by the encoder are input into the TPS prediction module for TPS prediction, and finally the number of thin plate spline control points of the whole image is obtained, and different numbers of thin plate spline control points represent different degrees of deformation. The control point map is input into the decoder, and the feature vectors of these thin plate spline control points are converted into a target output sequence to obtain a motion estimation stream.

5. As described in claim 1, the image features are input into the feature association layer, and 4D association information of W*H*W*H is generated for all pixels to obtain a 4D information body. The information of the 4D information body is input into the cyclic update GRU layer, and the optical flow field is obtained by iterative update.

6. Perform distortion correction on the feature direction vector of the optical flow map and the motion estimation flow, and calculate the loss with the real image to obtain the trained model. Four losses are used for training, namely image reconstruction loss LRe, perception loss LPe, grid loss LGrid and optical flow distance loss LLight.

Images