Hybrid homography transform rolling shutter single image rectification system and method, computer storage medium

Abstract

The application discloses a hybrid homography transform rolling shutter single image correction system and method, and a computer storage medium. The method comprises the following steps: dividing an image to be processed into K blocks, and assigning a weight to each motion base of a homography matrix of each block; processing the K blocks respectively to obtain an initial homography transform flow field corresponding to each of the K blocks; defining a Gaussian weight along a center line of the K blocks by using a Gaussian smoothing feature to obtain K Gaussian weight maps; performing a dot product on the initial homography transform flow field corresponding to each of the K blocks and the K Gaussian weight maps to obtain a final homography transform flow field of the image to be processed; and mapping the image to be processed based on the final homography transform flow field of the image to be processed to obtain a corrected image. Compared with a traditional single homography transform, the hybrid homography transform has a higher spatial degree of freedom and can be used for rolling shutter correction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision and computer graphics, and particularly relates to a hybrid homography transformation rolling shutter single image correction system and method and a computer storage medium. BACKGROUND

[0002] CMOS sensors are widely used in consumer devices due to their compactness and low power consumption, and are applied on many portable devices, such as cameras on mobile phones, robots or drones. However, CMOS sensors are affected by the rolling shutter (RS) effect, in which different rows of pixels are exposed at different times during the shooting and imaging process. This row-by-row exposure often brings some artifacts, including but not limited to the skew of image content, the mixing of straight lines, and the destruction of images, which will cause the tilt of image content, the mixing of straight lines and the swing in time, resulting in the distortion of images and videos. Worse still, the greater the camera shake, the stronger the dynamic motion of the scene. The stronger the dynamic motion of the scene, the stronger the RS effect. The distortion of images and videos caused by this will not only be visually unpleasant, but also harmful to downstream tasks such as three-dimensional reconstruction, pose estimation and depth estimation, and will have some impact. Compared with RS, global shutter (GS) is more friendly, but global shutter needs additional design to realize internal synchronization, which is more complex. SUMMARY

[0003] The application provides a hybrid homography transformation rolling shutter single image correction system and method, and a computer storage medium, which realizes hybrid homography matrix estimation and uses it to eliminate the rolling shutter effect, effectively correcting the rolling shutter single image.

[0004] The application is implemented by the following technical solutions:

[0005] In one aspect, the present application provides a hybrid homography transform rolling shutter single image rectification method, comprising the following steps: dividing the image to be processed into K blocks, and assigning a weight to each motion basis of the homography matrix of each block; wherein each motion basis of the homography matrix of each block is a set of motion bases, and each weight of each block is a set of weights; processing the K blocks respectively to obtain the initial homography transform flow field corresponding to each of the K blocks; wherein the processing steps for each block are: weighting a set of weights set in each block with a set of motion bases of the block; defining a Gaussian weight along the center line of the K blocks using a Gaussian smoothing feature to obtain K Gaussian weight maps; performing a dot product on the initial homography transform flow field corresponding to each of the K blocks and the K Gaussian weight maps to obtain the final homography transform flow field of the image to be processed; mapping the image to be processed based on the final homography transform flow field of the image to be processed to obtain a rectified image.

[0006] In some embodiments, the obtaining of the set of motion bases of each block of the image to be processed comprises the following steps: extracting n pairs of adjacent frames from the video data set; estimating n homography matrices of the adjacent frames; converting all the homography matrices into motion flow fields; unfolding and splicing all the motion flow fields to obtain a matrix F; singular value decomposing the matrix F:

[0007] F = U∑V T

[0008] wherein U and V are both unitary matrices, respectively containing left singular vectors and right singular vectors; ∑ represents a matrix containing all singular values, and the singular values are distributed on the diagonal line; the left singular vectors corresponding to the first N singular values are extracted; the N left singular vectors are adjusted to tensors matching the image size to obtain a set of motion bases.

[0009] In some embodiments, the obtaining of the weights of the set of motion bases set in each block is achieved through neural network learning.

[0010] In some embodiments, the loss function of the neural network is as follows:

[0011]

[0012] wherein H and W are the sizes of the image; m is the flow field corresponding to the position predicted by the network after weighting; M is the flow field of the same position labeled when generating the rolling shutter image; m ij refers to the coordinate transformation corresponding to the i-th row and j-th column of the motion flow field output by the network, and M ij refers to the coordinate transformation corresponding to the i-th row and j-th column of the motion flow field of the training label.

[0013] In some embodiments, dividing the image to be processed into K blocks comprises: dividing the image to be processed into K blocks along the longitudinal direction.

[0014] In some embodiments, dividing the image to be processed into K blocks further comprises: dividing the image to be processed into K blocks uniformly.

[0015] In another aspect, the application provides a hybrid homography transform rolling shutter single image rectification system, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the hybrid homography transform rolling shutter single image rectification method of any one of the above embodiments.

[0016] The application also provides a computer storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the hybrid homography transform rolling shutter single image rectification method of any one of the above embodiments.

[0017] Compared with the prior art, the application has the following advantages and beneficial effects:

[0018] The hybrid homography transform rolling shutter single image rectification method provided by the application adopts hybrid homography transform to eliminate the rolling shutter effect. Compared with traditional single homography transform, hybrid homography transform has higher spatial freedom and can be used for rolling shutter rectification.

[0019] The application estimates hybrid homography transform in a deep learning manner. Compared with directly learning homography matrix elements, the application learns a set of motion basis weights, and the set of basis weights weights multiple motion flow fields to obtain hybrid homography transform. Compared with using predefined motion flow fields, the application proposes a motion flow field learning method to construct more realistic motion flow fields for motion basis, which helps motion basis weight learning. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 The hybrid homography transform rolling shutter single image rectification method provided by the embodiment of the application is shown in the hybrid homography transform schematic diagram.

[0022] Figure 2 The flowchart of the hybrid homography transform rolling shutter single image rectification method provided by the embodiment of the application is shown in the flowchart.

[0023] Figure 3 A homographic transformation-based rolling shutter single image rectification method provided by the embodiment of the present application is shown in the schematic diagram of a homographic transformation-based rolling shutter single image rectification method learning process.

[0024] Figure 4 The effect comparison diagram of the homographic transformation-based rolling shutter single image rectification method provided by the embodiment of the present application and other methods is shown in the effect comparison diagram of a homographic transformation-based rolling shutter single image rectification method and other methods. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0026] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0027] In addition, in the description of the present application, the terms "horizontal", "vertical" and the like do not mean that the components must be absolutely horizontal or vertical, but can be slightly inclined. For example, "horizontal" only means that it is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0028] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense, for example, they can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium; can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0029] The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but can optionally include steps or modules that are not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0030] Related description:

[0031] Homography estimation is an important problem in computer vision, which is widely used in downstream tasks such as camera pose estimation. Traditional homography estimation highly depends on the detection of matched feature point pairs between images. In recent years, the application of deep learning in this task effectively avoids the dependence on feature point detection and improves the estimation accuracy.

[0032] The homogeneous coordinates of a point pair (a point pair is a position in the A-view image, and there is a corresponding position in the B-view image, and the two positions are called a point pair) on two images of different perspectives can be related by a transformation, which is called a homography transformation, and the corresponding 3x3 transformation matrix is the homography matrix. For the rolling shutter phenomenon, different pixel rows have different exposure delays, and a homography transformation relationship can be established between such an image and a global exposure image. For a homography matrix, there are 8 degrees of freedom; and for each row, the homography transformation relationship is different. However, the camera motion corresponding to adjacent rows is approximately the same, so the relationship of each row of the whole image can be simplified, and the image is divided into several blocks, each block corresponding to the same homography transformation relationship. In order to ensure the smoothness of the transformation, a Gaussian weight is set along the center line of each block - the transformation corresponding to any position on the final rolling shutter image is the weighted sum of the homography matrices of each block corresponding to the Gaussian weight of the row, that is, the mixed homography transformation relationship.

[0033] In one aspect, the embodiment of the present application provides a mixed homography transformation rolling shutter single image correction method, comprising the following steps:

[0034] S10, divide the image to be processed into K blocks, and assign a weight to all motion bases of the homography matrix of each divided block. In S10, taking one block as a reference, all motion bases of the homography matrix of one block are defined as a set of motion bases, and the weight assigned to one block according to one set of motion bases is defined as a set of weights. Specifically, the image to be processed can be input into a VGG-style deep network, and K blocks and weights corresponding to the image can be obtained by output.

[0035] Compared with directly learning the elements of the homography matrix, we learn the weights of a set of pre-learned motion bases; the motion flow field obtained by weighted summation of the motion bases is used to express the homography transformation. For the k blocks divided from the image, each block has such a set of motion bases (containing N), after learning the weights of the motion bases by the neural network, the motion flow field corresponding to each block is obtained by weighting the motion bases in each block; then the motion flow field is weighted according to the Gaussian weights corresponding to different blocks to obtain the mixed motion flow field at each position, and the mixed motion flow field corresponds to the mixed homography transformation; finally, the image rectification is performed by using the mixed motion flow field.

[0036] In some embodiments, the step S10 of dividing the image to be processed into K blocks specifically comprises:

[0037] S101, dividing the image to be processed into K blocks along the longitudinal direction. In S101, with reference to the exposure characteristics of the rolling shutter, the image obtained by the rolling shutter has exposure delay between rows, and the direction corresponding to the row is the horizontal direction of the image, and the direction perpendicular to the horizontal direction in the image is the longitudinal direction.

[0038] In some other embodiments, the step S10 of dividing the image to be processed into K blocks further specifically comprises uniformly dividing the image to be processed into K blocks. After each block is uniformly divided, it is convenient for subsequent uniform processing, and the processing efficiency is higher.

[0039] In some embodiments, the method for obtaining a set of motion bases for each block in the step S10 comprises the following steps:

[0040] S1021, extracting n pairs of adjacent frames from the video data set. In S1021, the video data set is preferably a conventional data set, and n pairs of adjacent frames are extracted from the conventional data set. Compared with unconventional data sets, the use of conventional data sets has smaller deviation in the final result. On the other hand, the feature point matching of adjacent frames has better robustness.

[0041] S1022, estimating n homography matrices corresponding to n adjacent frames. In S1022, the traditional method of SIFT / SURT+RANSAC can be used to estimate the homography matrix.

[0042] S1023, converting all the n homography matrices obtained in S1022 into motion flow fields.

[0043] S1024, unfolding and splicing all the motion flow fields obtained in S1023 to obtain a matrix F.

[0044] S1025, performing singular value (SVD) decomposition on the matrix F obtained in S1024, and the specific process is as follows:

[0045] F = U∑V T

[0046] Wherein, U and V are both unitary matrices, respectively containing left singular vectors and right singular vectors; Σ represents a matrix containing all singular values, and the singular values are distributed on its diagonal.

[0047] S1026, extract the left singular vector corresponding to the first N singular values.

[0048] S1027, adjust the N left singular vectors obtained in S1026 to the matching image size tensor according to the image size, and obtain a set of motion bases.

[0049] S20, process K blocks respectively to obtain the initial homographic transformation flow field corresponding to each of the K blocks. In S20, the processing steps for each block are as follows: weight a set of weights set in each block with a set of motion bases of the block to obtain the initial homographic transformation flow field corresponding to each of the K blocks.

[0050] In some embodiments, in S20, the weights of the set of motion bases set in each block are learned through a neural network, and then weighted processing is performed. Please refer to Figure 2 , the model structure diagram can be used to complete the neural network learning. Specifically, the generated data set can be trained, and the generated data set includes Regular, Low Texture, Low Light, Small Foreground, and Large Foreground. Different types of images are used for training, which can make our method achieve good results in challenging scenes. Figure 2

[0051] In some embodiments, the loss function of the neural network for weighting the set of weights set in each block with the set of motion bases of the block can use:

[0052]

[0053] Wherein, H and W are the size of the image; m is the flow field corresponding to the position predicted by the network after weighting; M is the flow field of the same position labeled when generating the rolling shutter image; m ij refers to the coordinate transformation corresponding to the i-th row and j-th column of the motion flow field output by the network, and M ij refers to the coordinate transformation corresponding to the i-th row and j-th column of the motion flow field of the training label.

[0054] ​The purpose of the loss function is to make the homography transformation flow predicted by the network as close as possible to the actual transformation flow from the rolling shutter image to the global image. It can be seen as the average end point error (EPE) of the whole image, the closer m and M are, the more accurately the network can predict the pixel mapping relationship between the rolling shutter image and the global shutter image.

[0055] S30, define a Gaussian weight along the center line of the K blocks using the Gaussian smoothing feature, and obtain K Gaussian weight maps. It should be noted that the order between S30 and S20 is not sequential.

[0056] S40, point product the initial homography transformation flow field corresponding to each of the K blocks obtained in S20 and the K Gaussian weight maps obtained in S30 to obtain the final homography transformation flow field of the image to be processed.

[0057] S50, based on the final homography transformation flow field of the agent image obtained in S40, map the image to be processed to obtain a corrected image.

[0058] Principle: First, generate a rolling shutter dataset and store the generated image to the flow field of the original image. From the adjacent frames of the video, learn the homographic transformation motion basis in the camera motion process for prediction. Then, input the generated rolling shutter single image into the designed neural network, and output the weight corresponding to the motion basis of the homographic transformation. Weight the weight with the motion basis and smooth it through the Gaussian weight to obtain the image flow field used for rolling shutter image correction, and map the rolling shutter image to the corrected image. In addition, an error function based on the end point error (EPE) is designed to make the image flow estimated by the network as close as possible to the generated label value.

[0059] Effect display, please refer to Figure 4 , Figure 4 The effect of rolling shutter image correction estimated by the mixed homographic transformation flow based on deep learning is shown. The top image is the generated image in the dataset, and the bottom image is the rolling shutter image generated in the real shooting. The first row and the third row are the images before correction, and the second row and the fourth row are the images after correction. From the effect diagram, it can be easily seen that although our method is trained on generated data, it is still effective on real rolling shutter images. At the same time, this method is not only effective on feature-rich building images, but also on weak texture images (first row, first column, first row, second column) and weak light images (first row, seventh column) still achieve significant effect - which shows that this method can effectively get rid of the dependence on feature point extraction.

[0060] In another aspect, the application provides a hybrid homography transform rolling shutter single image rectification system, comprising a memory and a processor, the processor storing a computer program, and the processor executing the computer program to implement the hybrid homography transform rolling shutter single image rectification method of any one of the above embodiments.

[0061] The application also provides a computer storage medium storing a computer program, and the computer program is loaded by the processor to execute the hybrid homography transform rolling shutter single image rectification method of any one of the above embodiments.

[0062] The above is only the preferred embodiment of the application, not any form of the application to limit, any simple modification, equivalent changes according to the technical essence of the application to the above embodiment, all fall within the scope of the application.

Claims

1. A method for correcting single-image rolling shutter speed using hybrid homography transformation, characterized in that, Includes the following steps: The image to be processed is divided into K blocks, and a weight is assigned to each motion basis of the homography matrix of each block; wherein, the motion basis of the homography matrix of each block is a set of motion basis, and the weights of each block are a set of weights; the acquisition of the weights of the set of motion basis in each block is achieved through neural network learning; The loss function of the neural network is as follows: Where H and W are the image dimensions; m is the flow field at the corresponding position in the image after the network prediction weights; and M is the flow field at the same position where the label is applied when generating the rolling shutter image. This refers to the coordinate transformation of the motion flow field in the i-th row and j-th column of the network prediction output. This refers to the coordinate transformation of the motion flow field of the training label at the i-th row and j-th column; The K blocks are processed separately to obtain the initial homography transformation flow field corresponding to each of the K blocks; wherein the processing steps for each block are as follows: the set of weights set in each block are weighted with the set of motion basis of that block; Using the Gaussian smoothing feature, Gaussian weights are defined along the center lines of K blocks to obtain K Gaussian weighted maps; The initial homography transformation flow field corresponding to each of the K blocks is multiplied by the K Gaussian weight maps to obtain the final homography transformation flow field of the image to be processed. Based on the final homography transformation flow field of the image to be processed, the image to be processed is mapped to obtain the corrected image.

2. The method for correcting a single image using a hybrid homography transformation rolling shutter according to claim 1, characterized in that, Obtaining a set of motion bases for each block of the image to be processed includes the following steps: Extract n pairs of adjacent frames from the video dataset; Estimate the homography matrices of n adjacent frames; All homography matrices are converted into a moving flow field; Unfold and stitch together all the motion flow fields to obtain matrix F; Perform singular value decomposition on matrix F: ； Where: U and V are both unitary matrices, containing left singular vectors and right singular vectors respectively; Σ represents a matrix containing all singular values, with the singular values ​​distributed on its diagonal; Extract the left singular vectors corresponding to the first N singular values; Adjust the N left singular vectors according to the image size to form tensors that match the image size, and obtain a set of motion basis vectors.

3. The method for correcting a single image using a rolling shutter speed according to claim 1, characterized in that, The step of dividing the image to be processed into K blocks includes: The image to be processed is divided into K blocks along the vertical direction.

4. The hybrid homography transformation rolling shutter single-image correction method according to claim 1 or 3, characterized in that, The step of dividing the image to be processed into K blocks also includes: The image to be processed is divided into K blocks evenly.

5. A hybrid homography transformation rolling shutter single-image correction system, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the hybrid homography transformation rolling shutter single-image correction method according to any one of claims 1 to 4.

6. A computer storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the hybrid homography transformation rolling shutter single image correction method according to any one of claims 1 to 4.

Images