A wide-angle image correction method of multi-scale features

Abstract

The application discloses a wide-angle image correction method of multi-scale features, which firstly inputs a photo and a corresponding distortion mask, and processes through a neural network with a structure of an encoder, a TPS prediction module and an optical flow estimation module, wherein the image feature points under multi-scale are obtained through the decoder module, the control points of the thin plate module of the features under the scene are obtained through the TPS prediction module, and the motion change flow is obtained by inputting the decoder; in addition, the optical flow field is obtained through the optical flow estimation module, the optical flow field and the motion change flow are combined, the local correction direction is obtained, and finally the original wide-angle deformation image is corrected. Experiments on a real-world dataset show that the application can realize wide-angle image correction about the square, the training and rendering speed are improved by several times, and the evaluation index is better. The application solves the problems of poor correction performance and slow speed under high resolution, and can be widely applied to related applications of image correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision image inpainting, and in particular to a wide-angle image correction method based on multi-scale features. BACKGROUND

[0002] Wide-angle lenses have been widely used in various fields due to their ability to capture a wider field of view. For example, in the field of photography, wide-angle lenses are often used to take pictures of landscapes, buildings, and portraits. In the field of machine vision and virtual reality, wide-angle lenses are also favored due to their wide field of view. However, wide-angle lenses often come with image distortion, which is mainly caused by the optical properties of the lens. When the angle of view is large, the edge of the image will be stretched or curved, which not only affects the aesthetics of the image, but also may have a negative impact on subsequent image processing and analysis.

[0003] With the continuous development of artificial intelligence and big data technology, image correction technology is also facing new opportunities. Although image correction technology has made significant progress, it still faces some challenges. For example, deep learning technology can be used to train more accurate distortion models, or big data can be used to optimize image correction algorithms. In the past, some researchers have added TPS transformation to the convolutional network to correct images, but the final model performs poorly on high-resolution images. The reason is that existing image correction algorithms often perform poorly on high-resolution distorted images because they cannot identify and use multi-scale features of the image.

[0004] To solve the above problems, the present application provides a wide-angle image correction method based on multi-scale features, which uses multi-scale feature sampling method, supplemented by the direction guidance and constraint of optical flow field, to realize fast training and correct distorted images. SUMMARY

[0005] The present application discloses a wide-angle image correction method based on multi-scale features, which can quickly correct the real-time monitoring images of most non-specific square wide-angle monitoring cameras, making the monitoring content more detailed and realistic. To achieve the above goal, the present application provides a wide-angle image correction method based on multi-scale features, comprising the following steps:

[0006] Prepare a set of images for network training and their corresponding masks;

[0007] Train a neural network with a structure of encoder, TPS prediction module and optical flow estimation module.

[0008] Extract deep features of the image at different scales through the convolutional network in the encoder;

[0009] The TPS prediction module extracts the feature from the thin plate spline control points of the sheet, to obtain a control point graph;

[0010] The motion estimation flow is extracted from the thin plate spline control points in the decoder;

[0011] The feature association layer in the optical flow estimation module obtains the information between the features by vector inner product, and then the optical flow field is constructed in the update layer GRU cycle.

[0012] The motion estimation flow and the optical flow map are combined, and the distortion correction and loss calculation are performed with the real image and the real optical flow to obtain the trained model.

[0013] The image to be corrected and the mask are input into the trained model for image correction.

[0014] Specifically, first, prepare the picture set and its corresponding mask for network training. First, obtain the public data set, which includes normal pictures and images after post-processing with various deformations for training. Second, prepare a mask image with the same resolution size as the original image, with non-deformation areas set to black and deformation areas set to white. After preparation, send it to the network for training.

[0015] The encoder structure is responsible for converting the input data into a feature vector containing all necessary information, and the decoder converts these feature vectors into the target output sequence. In order to facilitate the use of this network in the context of wide-angle images, a multi-scale feature extraction module is added to the encoder structure to obtain multi-scale features of the image.

[0016] The encoder structure mainly consists of 3 layers of sliding window attention layers with different dimensions, and the output feature vector dimensions are 4, 2, and 1 respectively, with the dimension decreasing step by step. The decoder structure corresponds to the encoder and is also composed of 3 layers of sliding window attention layers with different dimensions, and the output feature vector dimensions are 1, 2, and 4 respectively, with the dimension increasing step by step. The encoding block and the decoding block are connected by a skip connection to obtain more scale information during image restoration.

[0017] The sliding window attention layer divides the image into blocks, and then performs sampling convolution on each block to make each block contain part of the information of the surrounding blocks. The sliding window layer is introduced to enhance the model's ability to capture global information through window sliding operations between layers.

[0018] In order to obtain the distortion degree of the image, the image features obtained by the encoder are input into the TPS prediction module for TPS (Thin Plate Spline) prediction, and finally the number of thin plate spline control points of the whole image is obtained. Different numbers of thin plate spline control points represent different degrees of deformation.

[0019] After obtaining the control point map, the control point map is input into a decoder to convert the feature vectors of the thin plate spline control points into a target output sequence to obtain a motion estimation flow.

[0020] The image features are input into a feature association layer to generate 4D association information of W*H*W*H for all pixels to obtain a 4D information body. Information of the 4D information body is input into a recurrent update GRU layer to iteratively update to obtain an optical flow field.

[0021] The feature direction vector of the optical flow map is corrected by the motion estimation flow, and loss calculation is performed with the real image to obtain a trained model. Four losses are used for training, which are image reconstruction loss LRe, perception loss LPe, grid loss LGrid and optical flow distance loss LLight.

[0022] The image to be corrected and the mask are input into the trained model for image correction. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description, and the content of the drawings can further understand the present application. The components shown in the drawings are not necessarily drawn to scale, but the emphasis is placed on showing the principles of the embodiments.

[0024] Figure 1 is a step schematic diagram of a multi-scale feature wide-angle image correction method provided by the present application.

[0025] Figure 2 is a structure diagram of an optical flow field network separation provided by the present application.

[0026] Figure 3 is an effect diagram before and after wide-angle distortion correction provided by an embodiment of the present application.

[0027] Figure 4 is an effect comparison diagram of other correction methods provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the multi-scale feature wide-angle image correction of the present application will be described in detail below, and the examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0029] Please refer to Figure 1 The present application provides a multi-scale feature wide-angle image correction method, comprising the following steps:

[0030] First, prepare the data set and the corresponding mask for network training, specifically, the data set is divided into lossless real images and the corresponding images after pre-processing, wide-angle enlargement or distortion, and the corresponding mask is a mask with the same resolution size as the source image, the background color is black, and the position corresponding to the original image distorted or deformed is white.

[0031] The neural network structure is mainly divided into two functional modules, the first functional structure is the TPS prediction module of S102, and the second functional structure is the optical flow estimation module of S103. The encoder of S101 is set with a specific convolution kernel, which is a common feature extraction module. The following will be introduced in detail.

[0032] S101, the encoder structure is mainly composed of three layers of sliding window attention layers with different dimensions, and the output feature vector latitude is 8, 4, and 2 respectively. The layers are connected by down-sampling pooling layers to reduce the dimension and calculation amount of data while retaining important feature information. After the features are processed by the attention layer, they are extracted by the multi-scale feature extraction module to extract features at multiple resolutions. The multi-scale feature module is composed of a token mixer for spatial feature interaction, a deep convolution layer, and an MLP (Multi-Layer Perceptron) network with two levels for channel feature interaction.

[0033] Specifically, input an image of size HxWx3, perform patch partition to divide the image into H / 4xW / 4x48 patches, perform Linear Embedding to linearly transform the channel data of each pixel, unify the shape of the features, and construct feature maps of different sizes through three attention layers with different output dimensions of 8, 4, and 2. Then, interact with the MLP network, compress and interact features of different sizes, and output multi-scale feature maps.

[0034] S102, in the TPS prediction module, different kernel convolution layers are used to predict an increasing number of control points, with the number set to 10*10, 12*12, and 14*14. The number of thin plate spline control points increases gradually. The control points predicted by the previous head are up-sampled and integrated into the prediction of the next head. Then, these control points are arranged to generate a grid. Next, we use TPS transformation to distort this grid to align it with the regular grid defined on the ground real image. In implementation, considering that the concatenation of fully connected layers will introduce a large amount of calculation and storage cost, we use one or two convolution layers after each TPS transformation head to predict control points. The control points are calculated as follows:

[0035] q (t) =h (t) [Ρ(s (t-1) ,q (t-1))]+UP[q (t-1) ] (1)

[0037] where h(t) is the t-th TPS transformation layer, s(t-1) and q(t-1) are the feature map and control points of the t-th (t-1) head, respectively. P(·,·) represents the warping operation of the feature map given the control points, and UP[·] is the custom upsampling layer of the control points.

[0038] After obtaining the thin-plate spline control point map, the motion change stream is obtained through the decoder and the upsampling layer.

[0039] S103, the multi-scale image features obtained at the encoder are input to the feature association layer, which generates W*H*W*H 4D association information for all pixels, and uses a pooling layer to generate lower resolution association information to obtain a 4D information body. Where W is the width of the input image, and H is the height of the input image.

[0040] The information of the 4D information body is input into the recurrently updated GRU layer to iteratively update the optical flow field. The update operator estimates a series of flow rates: {f0, f1, f2,.... f k}, and in each iteration k, it generates a flow update direction Δf and adds it to the current estimate: f k+1 = f k + Δfk.

[0041] Specifically, given the current optical flow estimate, the algorithm retrieves the relevant features from the 4D information body. Then the relevant features are processed by 2 convolutional layers. In addition, 2 convolutional layers are used for optical flow estimation itself to generate optical flow features. Finally, the relevant optical flow and context features are concatenated as input feature maps.

[0042] The process of training the network, specifically, the feature direction vector of the optical flow map is corrected by warping with the motion estimation stream, and the loss calculation is performed with the real image to obtain the trained model. After the feature map is decoded by the decoder, the motion estimation stream formed by the control point map after upsampling interpolation is superimposed, and then the warping correction is performed with the optical flow field. The process of warping correction is specifically calculated as follows:

[0043] T = W(D[F] + UP[q (tps) ], S, L) (2)

[0045] where T is the correction target image, F is the feature map, D[·] is the decoder output operation, U[·] is the upsampling interpolation operation, S is the original image, and L is the optical flow map.

[0046] The present application utilizes four losses for training, which are image reconstruction loss LRe, perceptual loss LPe, grid loss LGrid and optical flow loss LLight respectively. The reconstruction loss and perceptual loss supervise the warped image at the pixel level and feature level respectively.

[0047] The image reconstruction loss is used to guide the model to generate an image as similar to the original image as possible, and the feature map difference is usually calculated using MSE (Mean Square Error) or MAE (Mean Absolute Error). The present application uses MSE to calculate, and the specific calculation is as follows:

[0048]

[0049] Where N is the total number of pixels, y i is the pixel value of the real image, y i is the pixel value of the generated image.

[0050] The calculation method of perceptual loss is usually to pass the input image and the target image through the pre-trained neural network respectively to obtain their feature representations in the network. Then, the feature representations are taken as the input of the loss function, and the Euclidean distance or Manhattan distance between them is calculated.

[0051] The grid loss constrains the edges of two consecutive deformed grids, where M represents the number of tuples of two consecutive edges in the grid m. When maximizing the above cosine representation, the corresponding two edges become collinear. Therefore, the loss reaches the minimum value, ensuring that the image content remains consistent.

[0052]

[0053] Where vector e t1 and vector e t2 are the edges of all consecutive deformed grids in the feature map.

[0054] The optical flow distance loss Light is calculated by calculating the L1 distance between the predicted optical flow with exponentially increasing weights and the real optical flow, given that the real optical flow is the motion change flow D[F], and the loss is defined as

[0055]

[0056] Therefore, the final loss function is as follows:

[0057] L=L Re +L Pe +L Grid +L Light (6)

[0059] The image to be corrected and the mask are input to the trained model to perform image correction. The experimental verification is taken as the last stage of the process of an embodiment of the present application, and the feasibility and effect of the method proposed in the present application are verified through actual experiments.

[0060] In the experimental verification stage, other reconstruction methods are selected for quantitative and qualitative experiments. In the quantitative experiment part, the model of the present application achieves the best in all indicators. The quantitative experiment indicators are shown in Table 1. The calculated indicators are PSNR, SSIM and parameter. The higher the values of PSNR and SSIM are, the better. The lower the value of the parameter is, the fewer the model parameters are.

[0061] Table 1 Comparison of indicators of the method of the present application and other methods on Places2 dataset.

[0062]

[0063] The above disclosure is only a preferred embodiment of the present application, and of course cannot limit the scope of the rights of the present application. Those skilled in the art can understand that the implementation of all or part of the above-mentioned embodiments, and the equivalent changes made according to the claims of the present application, still belong to the scope covered by the present application.

Claims

1. A wide-angle image correction method based on a transformer neural network, characterized in that, Includes the following steps: Prepare the image set and its corresponding mask for network training; A neural network with an encoder, a TPS prediction module, and an optical flow estimation module is trained. The encoder consists of three sliding window attention layers of different dimensions. The sliding window attention layers divide the image into blocks and then perform sampling convolution on each block so that each block can contain some information from its surrounding blocks. Downsampling pooling layers are connected between the layers to retain important feature information. After the features are processed by the attention layers, they are then processed by a multi-scale feature extraction module to extract features at multiple resolutions. The multi-scale feature module consists of a token mixer for spatial feature interaction, a deep convolutional layer, and an MLP network with two levels for channel feature interaction. Deep features at different scales of the image are extracted through the convolutional network in the encoder, including constructing feature maps of different sizes through three attention layers with different output dimensions, and then interacting with the MLP network to compress and interact with the feature maps of different sizes to output multi-scale feature maps. To obtain the degree of distortion in the image, the image features obtained by the encoder are input into the TPS prediction module for thin-plate spline prediction. Finally, the number of thin-plate spline control points in the entire image is obtained. Different numbers of thin-plate spline control points represent different degrees of deformation. In the TPS prediction module, convolutional layers with different kernels predict an increasing number of control points from the feature map. The control points predicted by the previous head are upsampled and integrated into the prediction of the next head. These control points are arranged to generate a grid. The TPS transform is used to warp the grid so that it is aligned with the regular grid defined on the ground real image. After obtaining the control point map, the control point map is input into the decoder, and the feature vectors of the control points of the thin plate spline are converted into the target output sequence to obtain the motion estimation stream; In the optical flow estimation module, the multi-scale feature map obtained by the encoder is input to the feature association layer. The feature association layer generates 4D association information of W*H*W*H for all pixels, and a pooling layer is used to generate association information of lower resolution to obtain a 4D information volume, where W is the width of the input map and H is the height of the input map. The information of the 4D information volume is input to the GRU layer to update iteratively, and then the optical flow field is obtained through iterative updates. The characteristic direction vector of the optical flow field is distorted and the estimated flow is corrected; loss is calculated to obtain a trained model; the image to be corrected and the mask are input into the trained model for image correction.

2. The wide-angle image correction method as described in claim 1, characterized in that, The preparation of the image set and its corresponding mask for network training includes the following steps: First, prepare the image set and its corresponding mask for network training by obtaining a public dataset, which includes normal images and images with various deformations after post-processing; Second, prepare a mask image with the same resolution as the original image, setting non-deformed areas to black and deformed areas to white, and then feed it into the network for training.

3. The wide-angle image correction method as described in claim 1, characterized in that, The loss calculation includes the following steps: performing distortion correction on the feature direction vector of the optical flow field and the motion estimation flow, and performing loss calculation on the real image to obtain the trained model; using four losses for training, namely image reconstruction loss LRe, perception loss LPe, grid loss LGrid and optical flow distance loss LLight.

Images