Video stabilization method based on inertial measurement unit data and deep learning

Abstract

The invention discloses a video stabilization method based on inertial measurement unit data and deep learning, and relates to the technical field of image processing, and the training process of a deep neural network model is as follows: obtaining an original image sequence to form a t moment data set, and obtaining an IMU data sequence corresponding to the t moment data set at the same time, calculating a virtual motion vector of each pixel according to the IMU data sequence; compensating each frame of image in the t-moment data set according to the quasi-motion vector of each pixel to obtain a primary compensation image, performing feature matching based on N-1 frames of primary compensation images and N-1 frames of compensated stable images, and calculating to obtain a new global perspective transformation matrix; performing secondary compensation on the current frame image according to a matching result to obtain a stable image after compensation at the moment t; adding the global perspective transformation matrix to the perspective transformation matrix of each Slice, and outputting a t-moment motion vector field; the video stabilization method improves stable transmission of images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video stabilization method based on inertial measurement unit data and deep learning. Background Art

[0002] In recent years, vehicle-mounted cameras and portable video recorders have been increasingly used in public services, administrative law enforcement, public security, traffic police, and other fields. In these scenarios, since the cameras are not fixed, they will encounter the problem of random camera jitter. This will then cause random jitter in the captured images. Watching such videos will cause obvious dizziness, and the user experience is very poor.

[0003] There are two main types of motion in video images. One is the motion caused by the camera's carrier turning; the other is the random jitter of the carrier (such as the vibration caused by the vehicle engine, the slight tremor of the arm, etc.). The former is a normal motion and does not need to be offset. The latter is the motion that really needs to be compensated. As for how to distinguish the two types of motion in the video, there have been many studies. The present invention does not involve this aspect of the problem, but only solves the problem of how to better compensate for the second type of random jitter.

[0004] There are currently two main ways to compensate for random jitter: 1) Install an inertial measurement unit (IMU) on the camera to collect the camera's angular velocity and acceleration data in real time, and then estimate the offset of the stationary target in the image between the previous and next frames based on the camera's internal parameter matrix and other parameters. This offset is generally the number of pixels displaced in the horizontal (x direction) and vertical (y direction) directions, as well as the rotation angle in the vertical screen direction (z direction). Then, based on these estimated parameters, the compensated image is transformed. 2) Compensate the image directly based on the image content. Find the feature points in the picture, and then match these feature points between the previous and next frames to obtain a certain number of feature point pairs. Then, based on the feature point pairs, estimate the transformation parameters between the two frames (generally using affine transformation or perspective transformation). Then, based on these estimated parameters, the compensated image is transformed.

[0005] Both methods can achieve image stabilization to a certain extent, but they also have their own disadvantages. For the first method, due to factors such as the error of the IMU itself and the error of the production and installation process, it is difficult to accurately estimate the accurate angular velocity and acceleration data, so it is not possible to achieve very accurate image compensation, which affects the final image stabilization effect. For the second method, in theory, if the feature point pair is matched very accurately, the compensation parameters can be very accurately estimated to achieve the optimal compensation effect, but there are various limitations in practical applications, and it is impossible to achieve very accurate feature point matching. For example: when the picture itself is relatively flat (such as facing a white wall), it is difficult to find a sufficient number of feature points; when the picture shakes a large amplitude, the distance between the correct feature point pairs is too large, and there will be great problems in matching feature points using image processing methods. The search radius must be set very large, which will lead to a square increase in the amount of calculation. It is difficult to provide sufficient computing power on some cheap edge devices. In addition, the expansion of the search radius will significantly increase the probability of mismatching, thereby affecting the estimation of the correct transformation parameters. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a video stabilization method based on inertial measurement unit data and deep learning, which can better remove random jitter of rotation and translation and improve the stable transmission of images.

[0007] The present invention proposes a video stabilization method based on inertial measurement unit data and deep learning, a video stabilization method based on inertial measurement unit data and deep learning, characterized in that an image to be compensated is input into a trained deep neural network model to output a compensated stabilized image;

[0008] The training process of the deep neural network model is as follows:

[0009] Step 1: obtain the current frame image at time t and the N-1 frames of historical original images before time t to form a data set at time t, and at the same time obtain the IMU data sequence corresponding to the data set at time t, integrate the IMU data sequence to obtain the camera posture data, calculate the perspective transformation matrix of each Slice according to the camera posture data and the camera intrinsic parameter matrix, and calculate the virtual motion vector of each pixel according to the perspective transformation matrix. The Slice is obtained by cutting the image in the data set at time t, and N is the number of images in the data set at time t;

[0010] Step 2: Compensate the corresponding frame image in the data set at time t according to the virtual motion vector of each pixel to obtain a compensated image. The compensated image corresponding to the N-1 historical original image is matched with the compensated stabilized image output by the deep neural network model of the N-1 historical original image to calculate a new global perspective transformation matrix.

[0011] Step 3: Perform secondary compensation on the primary compensation image corresponding to the current frame image according to the feature matching result to obtain a compensated stable image at time t;

[0012] Step 4: Append the new global perspective transformation matrix to the perspective transformation matrix of each Slice, calculate the motion vector of each pixel, and use it as the output motion vector field at time t. Based on the deep neural network model, learn the motion vector fields at different times to achieve image de-shaking.

[0013] Furthermore, the deep neural network model includes a preprocessing module, a channel dimension module and a classification module connected in sequence;

[0014] The input of the preprocessing module is the image in the data set at time t and the corresponding IMU data sequence. One output of the classification module is the compensated stabilized image at time t, and the other output is the motion vector field at time t.

[0015] Furthermore, the preprocessing module includes a concatenation layer concat and a first size adjustment layer connected in sequence, the input of the concatenation layer concat is the image in the data set at time t and the corresponding IMU data sequence, and the output of the first size adjustment layer is connected to the input of the channel dimension module.

[0016] Further, the channel dimension module includes an FCS layer, a second size adjustment layer, a maximum pooling layer Maxpool2d, a residual layer resblock, a third size adjustment layer, an upsample layer UPSample1, a fourth size adjustment layer, an upsample layer UPSample2, a fifth size adjustment layer and an upsample layer UPSample3;

[0017] The input of the FCS layer is connected to the output of the first resizing layer, the output of the FCS layer is connected to the input of the second resizing layer and the output of the upsample 2 layer, and the output of the second resizing layer is connected to the input of the maximum pooling layer Maxpool2d and the output of the upsample 1 layer;

[0018] The maximum pooling layer Maxpool2d, the residual layer resblock, the third size adjustment layer, the upsample layer UPSample1, the fourth size adjustment layer, the upsample layer UPSample2, the fifth size adjustment layer and the upsample layer UPSample3 are connected in sequence, and the output of the upsample layer UPSample3 is connected to the input of the classification module.

[0019] Furthermore, the classification module includes a stable image output channel and a motion vector field output channel, and both the stable image output channel and the motion vector field output channel include a sixth size adjustment layer and a two-dimensional convolution layer Conv2d connected in sequence, the input of the sixth size adjustment layer is connected to the output of the magnification layer UPSample3, the two-dimensional convolution layer Conv2d of the stable image output channel outputs a compensated stable image, and the two-dimensional convolution layer Conv2d of the motion vector field output channel outputs a motion vector field.

[0020] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the video stabilization method as described above when executing the computer program.

[0021] A computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the video stabilization method as described above.

[0022] The advantages of the video stabilization method based on inertial measurement unit data and deep learning provided by the present invention are: the video stabilization method based on inertial measurement unit data and deep learning provided in the structure of the present invention, the deep neural network model set in this embodiment can better remove the random jitter of rotation and translation, and can work well in scenes with relatively flat pictures and few feature points. In addition, in natural scenes, because the network has learned the compensation process of translational motion, the effect is better than the solution of using gyroscope data alone for anti-shake. At the same time, because the training data in the model training includes training data of various scenes with certain installation errors such as lenses and IMU chips, the trained network has a certain robustness and is not very sensitive to assembly errors in the production process, which is conducive to improving production efficiency and reducing production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention;

[0024] Figure 2 It is a schematic diagram of the structure of the deep neural network model;

[0025] Figure 3 Schematic diagram of the structure of FCS in the deep neural network model. DETAILED DESCRIPTION

[0026] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0027] In recent years, the use of deep learning for anti-shake processing has gained increasing attention in the academic community. Since deep learning has been successfully applied in many fields, the idea of ​​deep learning to solve problems has become a paradigm, that is, to give a network structure, prepare training data, and train to converge. However, when it comes to specific problems, how to design the structure of the network, especially the input and output data formats; how to guide the network to learn valuable information; how to obtain more effective training data; these are issues that researchers of specific technical solutions have to spend a lot of manpower and material resources to study and solve.

[0028] like Figures 1 to 3 As shown, the present invention proposes a video stabilization method based on inertial measurement unit data and deep learning, which inputs the image to be compensated into a trained deep neural network model to output a compensated stabilized image.

[0029] This embodiment provides a deep neural network model that can take multiple data as input, including: a current frame image and several historical frame images (input image), virtual motion field data (virtual motionfields) formed by IMU data; and can output multiple data, including: a compensated stabilized image (stabilized image), and motion field data (motion field) used to assist network learning.

[0030] In this embodiment, the training process of the deep neural network model is as follows:

[0031] Step 1: obtain the current frame image at time t and the N-1 frames of historical original images before time t to form a data set at time t, and at the same time obtain the IMU data sequence corresponding to the data set at time t, integrate the IMU data sequence to obtain the camera posture data, calculate the perspective transformation matrix of each Slice according to the camera posture data and the camera intrinsic parameter matrix, and calculate the virtual motion vector of each pixel according to the perspective transformation matrix. The Slice is obtained by cutting the image in the data set at time t, and N is the number of images in the data set at time t;

[0032] In this embodiment, the current frame and several previous frames are buffered and used as input data (input images) of the deep neural network model. Unlike the existing practice of using one frame of image as model input, N frames are used as input (experiments show that N is 8 to 16, which is relatively optimal after weighing various factors). First, the N frames of input images are divided into Slice images, the perspective transformation matrix of each Slice is calculated, and then the motion vector of each pixel is calculated. The motion vectors of all points in a picture form a motion vector field map. Each frame of the image corresponds to 2 motion vector field maps for x and y. Because this motion vector field is calculated based on IMU data rather than actual feature point matching, it is called a virtual motion field (virtualmotion fields), and the virtual motion field maps of the N frames of input images constitute N virtual motion fields.

[0033] Among them, the IMU data sequence needs to be timestamped, and the acquired current frame image and historical original image (input image) are also timestamped, and interactive compensation between the IMU data sequence and the input image is achieved according to the timestamp; in addition, since the image is acquired line by line, the camera postures involved in the same frame of image will be slightly different due to different acquisition time points. In order to improve the stable transmission of the video, this embodiment divides each image into B slices according to the image height, and B is generally 10 to 20, thereby improving the compensation effect of the image.

[0034] Step 2: Compensate each frame of the image in the data set at time t according to the quasi-motion vector of each pixel to obtain a compensated image. The compensated image corresponding to the N-1 historical original image is matched with the compensated stabilized image output by the deep neural network model of the N-1 historical original image to calculate a new global perspective transformation matrix.

[0035] Step 3: Perform secondary compensation on the primary compensated image corresponding to the current frame image in the data set at time t according to the feature matching result, so as to obtain a stable image after compensation at time t;

[0036] The matching result is specifically as follows: according to the virtual motion vector of each pixel in each Slice of the new global perspective transformation matrix, the primary compensation image corresponding to the current frame image is secondary compensated based on the corresponding virtual motion vector, so as to obtain a stable image after compensation at time t.

[0037] That is, the compensated stabilized image at time t is first compensated based on the IMU compensation in step 2 to obtain a compensated image, and then the compensated image after IMU compensation is secondarily compensated based on feature point matching, so as to obtain a stabilized image based on global and local compensation.

[0038] Step 4: Append the global perspective transformation matrix to the perspective transformation matrix of each Slice, calculate the motion vector of each pixel, and use it as the output motion vector field at time t. Based on the deep neural network model, learn the motion vector fields at different times to achieve image de-shaking.

[0039] The output of the deep neural network model is a cropped and enlarged image (stabilizedimage) after de-jittering, that is, a stabilized image, and the other is a motion field map (motion field) of each pixel after de-jittering, that is, motion field data, and the motion field map includes two images, x and y. The purpose of designing an output branch of the motion field in the deep neural network model of this embodiment is to guide the deep neural network model to learn the motion information of the pixels in the image, so that the deep neural network model can use this motion information to de-jitter the image, where the motion vector field corresponds to the offset of the feature point.

[0040] In this embodiment, in order to implement the above-mentioned video stabilization method, this embodiment sets a deep neural network model including a preprocessing module, a channel dimension module and a classification module which are connected in sequence; the input of the preprocessing module is the image in the data set at time t and the corresponding IMU data sequence, one output of the classification module is the stabilized image after compensation at time t, and the other output is the motion vector field at time t.

[0041] like Figure 2 As shown, the preprocessing module includes a concatenation layer concat and a first size adjustment layer connected in sequence. The input of the concatenation layer concat is the image in the data set at time t and the corresponding IMU data sequence. The output of the first size adjustment layer is connected to the input of the channel dimension module.

[0042] like Figure 3 The channel dimension module includes an FCS layer, a second size adjustment layer, a maximum pooling layer Maxpool2d, a residual layer resblock, a third size adjustment layer, an upscaling layer UPSample1, a fourth size adjustment layer, an upscaling layer UPSample2, a fifth size adjustment layer, and an upscaling layer UPSample3;

[0043] The input of the FCS layer is connected to the output of the first size adjustment layer, the output of the FCS layer is respectively connected to the input of the second size adjustment layer and the output of the upsample 2, the output of the second size adjustment layer is respectively connected to the input of the maximum pooling layer Maxpool2d and the output of the upsample 1; the maximum pooling layer Maxpool2d, the residual layer resblock, the third size adjustment layer, the upsample 1, the fourth size adjustment layer, the upsample 2, the fifth size adjustment layer and the upsample 3 are connected in sequence, and the output of the upsample 3 is connected to the input of the classification module.

[0044] The classification module includes a stable image output channel and a motion vector field output channel. Both the stable image output channel and the motion vector field output channel include a sixth size adjustment layer and a two-dimensional convolution layer Conv2d connected in sequence. The input of the sixth size adjustment layer is connected to the output of the magnification layer UPSample3. The two-dimensional convolution layer Conv2d of the stable image output channel outputs a compensated stable image, and the two-dimensional convolution layer Conv2d of the motion vector field output channel outputs a motion vector field.

[0045] The deep neural network model set by this embodiment can effectively remove the random jitter of rotation and translation, and can work well in scenes with relatively flat images and few feature points. In addition, in natural scenes, because the network has learned the compensation process of translational motion, the effect is better than the solution of using gyroscope data alone for anti-shake. At the same time, because the training data in the model training includes training data of various scenes with certain installation errors such as lenses and IMU chips, the trained network has a certain robustness and is not very sensitive to assembly errors in the production process, which is conducive to improving production efficiency and reducing production costs.

[0046] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A video stabilization method based on inertial measurement unit data and deep learning, characterized in that: Input the image to be compensated into the trained deep neural network model to output a compensated stable image; The training process of the deep neural network model is as follows: Step 1: obtain the current frame image at time t and the N-1 frames of historical original images before time t to form a data set at time t, and at the same time obtain the IMU data sequence corresponding to the data set at time t, integrate the IMU data sequence to obtain the camera posture data, calculate the perspective transformation matrix of each Slice according to the camera posture data and the camera intrinsic parameter matrix, and calculate the virtual motion vector of each pixel according to the perspective transformation matrix. The Slice is obtained by cutting the image in the data set at time t, and N is the number of images in the data set at time t; Step 2: Compensate the corresponding frame image in the data set at time t according to the virtual motion vector of each pixel to obtain a compensated image. The compensated image corresponding to the N-1 historical original image is matched with the compensated stabilized image output by the deep neural network model of the N-1 historical original image to calculate a new global perspective transformation matrix. Step 3: Perform secondary compensation on the primary compensation image corresponding to the current frame image according to the feature matching result to obtain a compensated stable image at time t; Step 4: Append the new global perspective transformation matrix to the perspective transformation matrix of each slice in the current frame image, calculate the motion vector of each pixel, and use it as the output motion vector field at time t. Based on the deep neural network model, learn the motion vector fields at different times to achieve image de-shaking.

2. The video stabilization method based on inertial measurement unit data and deep learning according to claim 1, characterized in that: The deep neural network model includes a preprocessing module, a channel dimension module and a classification module connected in sequence; The input of the preprocessing module is the image in the data set at time t and the corresponding IMU data sequence. One output of the classification module is the compensated stabilized image at time t, and the other output is the motion vector field at time t.

3. The video stabilization method based on inertial measurement unit data and deep learning according to claim 2, characterized in that: The preprocessing module includes a concatenation layer concat and a first size adjustment layer connected in sequence. The input of the concatenation layer concat is the image in the data set at time t and the corresponding IMU data sequence. The output of the first size adjustment layer is connected to the input of the channel dimension module.

4. The video stabilization method based on inertial measurement unit data and deep learning according to claim 3, characterized in that: The channel dimension module includes an FCS layer, a second size adjustment layer, a maximum pooling layer Maxpool2d, a residual layer resblock, a third size adjustment layer, an upsample layer UPSample1, a fourth size adjustment layer, an upsample layer UPSample2, a fifth size adjustment layer and an upsample layer UPSample3; The input of the FCS layer is connected to the output of the first resizing layer, the output of the FCS layer is connected to the input of the second resizing layer and the output of the upsample 2 layer, and the output of the second resizing layer is connected to the input of the maximum pooling layer Maxpool2d and the output of the upsample 1 layer; The maximum pooling layer Maxpool2d, the residual layer resblock, the third size adjustment layer, the upsample layer UPSample1, the fourth size adjustment layer, the upsample layer UPSample2, the fifth size adjustment layer and the upsample layer UPSample3 are connected in sequence, and the output of the upsample layer UPSample3 is connected to the input of the classification module.

5. The video stabilization method based on inertial measurement unit data and deep learning according to claim 4, characterized in that: The classification module includes a stable image output channel and a motion vector field output channel. Both the stable image output channel and the motion vector field output channel include a sixth size adjustment layer and a two-dimensional convolution layer Conv2d connected in sequence. The input of the sixth size adjustment layer is connected to the output of the magnification layer UPSample3. The two-dimensional convolution layer Conv2d of the stable image output channel outputs a compensated stable image, and the two-dimensional convolution layer Conv2d of the motion vector field output channel outputs a motion vector field.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the video stabilization method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the video stabilization method as described in any one of 1-5.

Images