ANTI-SHAKE METHOD FOR PANORAMIC VIDEO AND PORTABLE TERMINAL DEVICE

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of panoramic video, and particularly to a panoramic video anti-shake method and a portable terminal.BACKGROUND

[0002] At present, when a panoramic video is shot, it is usually to hold a panoramic shooting device in hand to shoot. When a mobile shooting is taken, a phenomenon of jitter occurs in the panoramic shooting video due to unstable hands. When a panoramic view angle is acquired, the focus of the original lens is often lost due to the movement or shaking of the camera, which affects the viewing experience of the panoramic video. One of the current solutions is to use a pan-tilt to stabilize the panoramic shooting device to stabilize the pictures taken. However, the disadvantage is that the pan-tilt is more expensive and the volume is generally larger, and the pan-tilt does not completely solve the problem of picture jitter when shooting the video with a handheld panoramic shooting device.

[0003] When a panoramic video viewer wants to watch from a viewing angle of an original motion direction of the video, and the impact of artificial shaking on the video can be avoided, it is necessary to enable the video to keep the viewing angle changing in the original shooting direction and keep the video stable. Accordingly, it is necessary to study a panoramic video anti-shake method that can retain the change state of the original shooting direction of the shooting device.

[0004] The document WO 2018 / 184423 A1 discloses a method and system for panoramic video stabilization, and a portable terminal. The method includes: acquiring in real time the timestamp of a current state, an accelerometer count value, and an angular speed value of a portable terminal; establishing the rotation vector of the current state by utilizing an extended Kalman filter combined with the accelerometer count value and the angular speed value; calculating a current rotation matrix via the Rodrigues' rotation formula on the basis of the rotation vector of the current state; and rotating a panoramic image on the basis of the current rotation matrix and producing a stabilized video frame.

[0005] The document XP032769149 "Real-time 3D rotation smoothing for video stabilization", 2014 ASILOMAR CONFERENCE ON SIGNALS; SYSTEMS AND COMPUTERS; IEEE; 2 November 2014, discloses two real-time motion smoothing algorithms for video stabilization using a pure 3D rotation motion model with known camera projection parameters. Both proposed algorithms aim at smoothing 3D rotation matrix sequences in acausal way. The first algorithm smooths the 3D rotation sequences in a way similar to 1st-order IIR filtering. The second algorithm uses sequential probabilistic estimation under a constant angular velocity model. These two algorithms are generalized from classical 2D motion smoothing algorithms, the manifold structure of the rotation matrices is exploited so that the proposed algorithms directly smooth the 3D rotation sequences on the manifold.

[0006] The document XP055361709 "Stabilizing Omnidirectional Videos Using 3D Structure and Spherical Image Warping", 13 June 2011, DOI: 10.1.1.389.5759, addresses the problem of stabilizing spherical videos. Whereas various techniques have been proposed to stabilize conventional videos, this work is the first approach proposed for omnidirectional videos. It introduces a method for extracting the camera path and 3D-information of the environment. A desired smooth stabilized path is obtained by modifying the original camera path. A method for synthesizing a stabilized video with respect to the desired path is provided. Each output frame is generated by warping a single frame from the input video.SUMMARYTechnical Problem

[0007] The purpose of the present application is to provide a panoramic video anti-shake method, a computer-readable storage medium and a portable terminal, and is intended to solve the problem of the loss of the original lens focus when the viewing angle of the panoramic video is acquired. The method can, through decomposing the motion of the camera, retain the original shooting angle of the camera, synthesize the motion of the virtual lens and generate a stable video.

[0008] Technical Solution The invention is set out in the independent claims. The dependent claims define advantageous embodiments.

[0009] In a first aspect, the present disclosure provides a panoramic video anti-shake method, which includes: acquiring a world coordinate of a reference point in a world coordinate system in real time, and simultaneously acquiring a coordinate corresponding to the reference point in a portable terminal in a camera coordinate system and an angular velocity value of a gyroscope in the portable terminal in a current state; smoothing a motion of the camera by using an extended Kalman filter; decomposing the smoothed motion, generating a motion of a virtual lens in a mode in which motions in original shooting changing directions of a shooting device are retained, and calculating a rotation quantity of the virtual lens; performing a reprojection on an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video; the smoothing the motion of the camera by using the extended Kalman filter specifically comprises: using a state model q ˜ k = Φ w ˜ k − 1 • q ˜ k − 1 w ˜ k = w ˜ k − 1 ; using an observation model q k = q ˜ k w k = w ˜ k . wherein, k is time, w k is an obtained angular velocity, and q k is an obtained observation vector of the rotation quantity, w̃ k and q̃ k are state values of the angular velocity and the rotation quantity, w̃ k-1 and q̃ k-1 are state values of the angular velocity and the rotation quantity at time k-1, and q k is a quaternion representation of R w2c -1< , w k is the angular velocity value of the gyroscope, w k = lg q k + 1 ⋅ q k − 1 , Φ(w̃ k-1 ) is a state transition matrix at the time k-1, Φ(w̃ k-1 )=exp([w̃ k-1 ] × ), q̃ k is a quaternion representation of the estimated smoothed motion of the lens, q̃ k is a state value evaluated by a value of q̃ k-1 at previous time; wherein the performing the reprojection on the original video to generate the stable video specifically comprises: calculating a corresponding relationship between a pixel in an original video frame and a pixel in an output video frame, and then performing interpolation sampling on the original video frame according to the corresponding relationship, generating the output video frame and finally generating the stable video.

[0010] In a second aspect, the present disclosure provides a computer-readable storage medium, on which a computer program is stored, the computer program, when executed by a processor, implements the steps of the above-mentioned panoramic video anti-shake method.

[0011] In a third aspect, the present disclosure provides a portable terminal, which includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, wherein the processor, when executing the computer programs, implement the steps of the above-mentioned panoramic video anti-shake method. Advantages

[0012] In the present disclosure, by decomposing the motion of the camera and retaining the synthesized motion of the virtual lens at the original shooting angle of the camera, the stable video can be generated. Therefore, this method can keep smooth motion of the rendering lens, generate the stable video, and retain the original shooting angle of the camera, which has strong robustness to large noise scenes and most sports scenes.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1 is a flow chart showing a panoramic video anti-shake method according to an embodiment I of the present disclosure. FIG. 2 is a schematic structure diagram of a portable terminal according to an embodiment III of the present disclosure. DETAILED DESCRIPTION

[0014] In order to make the objectives, technical solution, and advantages of the present disclosure clearer, the present disclosure will be described in detail with reference to the accompanying drawings and embodiments. It should be appreciated that the specific embodiments described here are only used for explaining the present disclosure, rather than limiting the present disclosure.

[0015] In order to illustrate the technical solution of the present disclosure, specific embodiments are used for description below.Embodiment I

[0016] Referring to FIG. 1, the panoramic video anti-shake method provided by the Embodiment 1 of the present disclosure includes the following steps.

[0017] S101: a world coordinate of any one reference point in a world coordinate system is acquired in real time, and a camera coordinate corresponding to the reference point in a portable terminal and an angular velocity value of a gyroscope in the portable terminal in a current state are simultaneously acquired.

[0018] In the embodiment I of the present disclosure, S101 can specifically be as follows.

[0019] The world coordinate of the reference point is P w , and the camera coordinate is P c , the following relationship is specifically included: P w = R w 2 c P c

[0020] In the formula (1), R w 2 c = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 is a rotation matrix by which the camera coordinate is transformed to the world coordinate; the elements r 11 to r 33 are elements of the rotation matrix R w2c ,

[0021] unit matrix. r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T · r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 · r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T = I, I is the unit matrix.

[0022] The step of acquiring the angular velocity value of the gyroscope in the portable terminal in real time specifically includes: an angular velocity sensor is adopted to read a three-axis angular velocity value as w k .

[0023] S102: an extended Kalman filter is utilized to smooth a motion of the camera.

[0024] The extended Kalman filter algorithm linearizes the nonlinear system, and then performs Kalman filter. The Kalman filter is a high-efficiency recursive filter which can estimate a state of a dynamic system from a series of measurements that do not completely contain noises.

[0025] In the embodiment I of the present disclosure, S102 can specifically be as follows.

[0026] The extended Kalman filter algorithm is utilized to establish a state model and an observation model of a motion state of the camera; specifically: the state model is: q ˜ k = Φ w ˜ k − 1 • q ˜ k − 1 w ˜ k = w ˜ k − 1 ; the observation model is: q k = q ˜ k w k = w ˜ k .

[0027] In the formulas (2) and (3), k is time, w k is an obtained angular velocity, and q k is an obtained observation vector of the rotation quantity; w̃ k and q̃ k are state values of the angular velocity and rotation quantity , w̃ k-1 are q̃ k-1 state values of the angular velocity and rotation quantity at time k-1; q k is a quaternion representation of R w2c -1< ; w k is the angular velocity value of the gyroscope; w k = lg q k + 1 ⋅ q k − 1 , Φ(w̃ k-1 ) is a state transition matrix at the time k-1; Φ(w̃ k-1 )=exp([w̃ k-1 ] × ), q̃ k is the quaternion representation of the estimated smoothed motion of the camera, q̃ k is the state value estimated by the value of q̃ k-1 at the previous time.

[0028] The specific process of updating the prediction includes: at time k, q̃ k-1 estimated at the previous time and the observation value q k the current time are utilized to update the estimation of the state variable q̃ k to obtain an estimated value at the current time. The predicted value q̃ k is the rotation quantity of the virtual lens at the k-th time.

[0029] S103: the smoothed motion is decomposed, a motion of the virtual lens is synthesized in a free lens mode, and a rotation quantity of the virtual lens is calculated.

[0030] In the embodiment I of the present disclosure, S103 can specifically be as follows.

[0031] The coordinate of the reference point in the virtual lens is P c , and the following relationship is specifically included: P c ¯ = R c ¯ 2 w P w ;

[0032] In formula (4), R c2w is a 3*3 matrix, which is the rotation quantity of the virtual lens.

[0033] The free lens mode is a mode in which motions in the original shooting changing directions of the shooting device are retained. For the synthesized motion of the virtual lens in the free lens mode, the rotation quantity is set as R c2w = R̃, R̃ is the original motion trajectory of the shooting device. the step of decomposing the smoothed motion, synthesizing the motion of the virtual lens in the free lens mode and calculating the rotation quantity of the virtual lens specifically includes: For the synthesized motion of the virtual lens, the shaking is prevented in the free lens mode, then R c2w = R̃ k , where R̃ k is the rotation quantity of q̃ k .

[0034] It should be noted that the Rodrigues formula can be utilized to obtain the rotation matrix R from the quaternion after the unit vector is rotated by an angle θ. Specifically, the quaternion is set as q=(θ,x,y,z) T< , then calculation formula of the rotation matrix R is: R = cos θ + x 2 1 − cos θ − z sin θ + xy 1 − cos θ y sin θ + xz 1 − cos θ z sin θ + xy 1 − cos θ cos θ + y 2 1 − cos θ − x sin θ + yz 1 − cos θ − y sin θ + xz 1 − cos θ x sin θ + yz 1 − cos θ cos θ + z 2 1 − cos θ .

[0035] S104: a reprojection is performed on the original video according to the rotation quantity of the virtual lens and the rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video.

[0036] In the embodiment I of the present disclosure, S104 can specifically be as follows.

[0037] A corresponding relationship between a pixel in the original video frame and a pixel in an output video frame is calculated, and then interpolation resampling is performed on the original video frame according to the corresponding relationship to generate the output video frame, and finally a stable video is generated.

[0038] The pixel in the original video frame is set as P s , and the pixel in the corresponding output video frame is P d , then the corresponding relationship is: P s = D c K c R w 2 c − 1 R c ¯ 2 w − 1 K c ¯ − 1 P d , where P s = [x s , y s ] T< , P d = [x d , y d ] T< , x s and y s are the coordinate values of the abscissa and ordinate of the pixel P s in the original video frame; x d and y d are the coordinate values of the abscissa and ordinate of the pixel P d in the output video frame respectively; K c and D c are respectively an internal parameter and a distortion model of the camera, K c is a projection internal parameter of the virtual lens.

[0039] Then the step of performing the interpolation resampling on the original video frame I s according to the corresponding relationship and generating the output video frame I d specifically includes: I d P d = ∑ w i I s P s i ∑ w i

[0040] In formula (5), w i is the interpolation weight, and P s i ∈ U P s δ is a neighborhood coordinate of P s .Embodiment II

[0041] In the embodiment II of the present disclosure, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the panoramic video anti-shake method provided in the embodiment I of the present disclosure.

[0042] The computer-readable storage medium can be a non-transitory computer-readable storage medium.Embodiment III

[0043] FIG. 2 shows a specific structure block diagram of a portable terminal provided in the Embodiment III of the present disclosure. The portable terminal 100 includes: one or more processors 101, a memory 102, and one or more computer programs; the processor 101 is connected to the memory 102 by a bus; the one or more computer programs are stored in the memory 102, and are configured to be executed by the one or more processors 101; and the processor 101, when executing the computer program, implements the steps of the panoramic video anti-shake method provided in the embodiment I of the present disclosure.

[0044] In the present invention, by decomposing the motion of the camera, retaining the original shooting viewing angle of the camera and synthesizing the motion of the virtual lens, and the stable video can be generated. Therefore, this method can maintain smooth motion of the rendering lens, generate the stable video, and retain the original shooting angle of the camera. Accordingly, when the user is watching the panoramic video, this method can avoid artificial video jitters and retain changes in the viewing angle in the original direction of the panoramic video. Therefore, this method retains the original shooting angle of the camera, which can keep the smooth motion of the rendering lens and generate the stable video, and has strong robustness to large noise scenes and most sports scenes.

[0045] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the storage medium can include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

Claims

1. A panoramic video anti-shake method, comprising: acquiring (S101) a world coordinate of a reference point in a world coordinate system in real time, and simultaneously acquiring a camera coordinate corresponding to the reference point in a portable terminal in a camera coordinate system and an angular velocity value of a gyroscope in the portable terminal in a current state; and smoothing (S102) a motion of the camera by using an extended Kalman filter; characterized by decomposing (S103) the smoothed motion, generating a motion of a virtual lens in a mode in which motions in original shooting changing directions of a shooting device are retained, and calculating a rotation quantity of the virtual lens; and performing (S104) a reprojection on an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video; wherein the step of smoothing the motion of the camera by using the extended Kalman filter specifically comprises: using a state model q ˜ k = Φ w ˜ k − 1 • q ˜ k − 1 w ˜ k = w ˜ k − 1 ; and using an observation model q k = q ˜ k w k = w ˜ k . wherein k is time, wk is an obtained angular velocity, qk is an obtained observation vector of the rotation quantity, w̃k and q̃k are state values of the angular velocity and the rotation quantity, w̃k-1 and q̃k-1 are state values of the angular velocity and the rotation quantity at time k-1, qk is a quaternion representation of an inverse of the rotation matrix Rw2c-1, wk is the angular velocity value of the gyroscope, w k = lg q k + 1 ⋅ q k − 1 , Φ(w̃k-1) is a state transition matrix at the time k-1, Φ(w̃k-1)=exp([(w̃k-1)×), q̃k is a quaternion representation of the estimated smoothed motion of the lens, q̃k is a state value evaluated by a value of q̃k-1 at previous time; wherein the step of performing the reprojection on the original video to generate the stable video specifically comprises: calculating a corresponding relationship between a pixel in an original video frame and a pixel in an output video frame, performing interpolation sampling on the original video frame according to the corresponding relationship, generating the output video frame, and generating the stable video.

2. The method according to claim 1, wherein the world coordinate of the reference point is Pw, the camera coordinate is Pc, the angular velocity value wk is a three-axis angular velocity value, and the following relationship is specifically verified: P w = R w 2 c P c wherein, R w 2 c = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 is the rotation matrix by which the camera coordinate is transformed to the world coordinates, elements r11 to r33 are elements of the rotation matrix Rw2c, r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T • r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 • r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T = I, I is an unit matrix; the step of acquiring the angular velocity value of the gyroscope in the portable terminal in real time specifically comprises: using an angular velocity sensor to read the three-axis angular velocity value wk.

3. The method according to claim 1, wherein a coordinate of the reference point in a virtual lens coordinate system is Pc, the following relationship is specifically verified: P c ¯ = R c ¯ 2 w P w wherein Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

4. The method according to claim 1, wherein the step of decomposing the smoothed motion, generating the motion of the virtual lens in the mode in which motions in original shooting changing directions of a shooting device are retained and calculating the rotation quantity of the virtual lens specifically comprises: for the generated motion of the virtual lens, preventing shaking in the mode in which motions in original shooting changing directions of a shooting device are retained, so that Rc2w = R̃k, wherein R̃k is the rotation quantity of q̃k, Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

5. The method according to claim 1, wherein, the pixel in the original video frame is set as Ps, and the pixel in the corresponding output video frame is Pd, accordingly the corresponding relationship is: P s = D c K c R w 2 c − 1 R c ¯ 2 w − 1 K c ¯ − 1 P d , wherein, Ps = [xs, ys]T, Pd = [xd, yd]T, xs and ys are respectively coordinate values of an abscissa and an ordinate of the pixel Ps in the original video frame; xd and yd are respectively coordinate values of an abscissa and an ordinate of the pixel Py in the output video frame; Kc and Dc are respectively an internal parameter and a distortion model of the camera, Kc is a projection internal parameter of the virtual lens; Rw2c is the rotation matrix by which the camera coordinate is transformed to the world coordinates, Rc2w is the rotation quantity of the virtual lens; the step of performing the interpolation sampling on the original video frame Is according to the corresponding relationship, and generating the output video frame Id specifically comprises: I d P d = ∑ w i I s P s i ∑ w i wherein wi is an interpolation weight, P s i ∈ U P s δ is a neighborhood coordinate of Ps.

6. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the panoramic video anti-shake method according to any one of claims 1 to 5.

7. A portable terminal (100), comprising: one or more processors (101); a memory (102); and one or more computer programs, wherein the one or more computer programs are stored in the memory (102) and configured to be executed by the one or more processors (101), wherein the processor (101), when executing the computer programs, implement following steps of a panoramic video anti-shake method: acquiring a world coordinate of a reference point in a world coordinate system in real time, and simultaneously acquiring a camera coordinate corresponding to the reference point in a portable terminal in a camera coordinate system and an angular velocity value of a gyroscope in the portable terminal in a current state; and smoothing a motion of the camera by using an extended Kalman filter; characterized by decomposing the smoothed motion, generating a motion of a virtual lens in a mode in which motions in original shooting changing directions of a shooting device are retained, and calculating a rotation quantity of the virtual lens; and performing a reprojection on an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video; wherein the step of smoothing the motion of the camera by using the extended Kalman filter specifically comprises: using a state model q ˜ k = Φ w ˜ k − 1 • q ˜ k − 1 w ˜ k = w ˜ k − 1 ; and using an observation model q k = q ˜ k w k = w ˜ k . wherein k is time, wk is an obtained angular velocity, qk is an obtained observation vector of the rotation quantity, w̃k and q̃k are state values of the angular velocity and the rotation quantity, w̃k-1 and q̃k-1 are state values of the angular velocity and the rotation quantity at time k-1, qk is a quaternion representation of an inverse of the rotation matrix Rw2c-1, wk is the angular velocity value of the gyroscope, w k = lg q k + 1 ⋅ q k − 1 , Φ(w̃k-1) is a state transition matrix at the time k-1, Φ(w̃k-1)=exp([w̃k-1]×), q̃k is a quaternion representation of the estimated smoothed motion of the lens, q̃k is a state value evaluated by a value of q̃k-1 at previous time; wherein the step of performing the reprojection on the original video to generate the stable video specifically comprises: calculating a corresponding relationship between a pixel in an original video frame and a pixel in an output video frame, performing interpolation sampling on the original video frame according to the corresponding relationship, generating the output video frame, and generating the stable video.

8. The portable terminal according to claim 7, wherein the world coordinate of the reference point is Pw, the camera coordinate is Pc, the angular velocity value wk is a three-axis angular velocity value, and the following relationship is specifically verified: P w = R w 2 c P c wherein, R w 2 c = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 is the rotation matrix by which the camera coordinate is transformed to the world coordinates, elements r11 to r33 are elements of the rotation matrix Rw2c, r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T • r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 • r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 T = I , I is an unit matrix; the step of acquiring the angular velocity value of the gyroscope in the portable terminal in real time specifically comprises: using an angular velocity sensor to read the three-axis angular velocity value wk.

9. The portable terminal according to claim 7, wherein a coordinate of the reference point in a virtual lens coordinate system is Pc, the following relationship is specifically verified: P c ¯ = R c ¯ 2 w P w wherein Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

10. The portable terminal of claim 7, wherein the step of decomposing the smoothed motion, generating the motion of the virtual lens in the mode in which motions in original shooting changing directions of a shooting device are retained and calculating the rotation quantity of the virtual lens specifically comprises: for the generated motion of the virtual lens, preventing shaking in the mode in which motions in original shooting changing directions of a shooting device are retained, so that Rc2w = R̃k, wherein R̃k is the rotation quantity of q̃k, Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

11. The portable terminal according to claim 7,wherein, the pixel in the original video frame is set as Ps, and the pixel in the corresponding output video frame is Pd, accordingly the corresponding relationship is: P s = D c K c R w 2 c − 1 R c ¯ 2 w − 1 K c ¯ − 1 P d , wherein, Ps = [xs, ys]T, Pd = [xd, yd]T, xs and ys are respectively coordinate values of an abscissa and an ordinate of the pixel Ps in the original video frame; xd and yd are respectively coordinate values of an abscissa and an ordinate of the pixel Pd in the output video frame; Kc and Dc are respectively an internal parameter and a distortion model of the camera, Kc is a projection internal parameter of the virtual lens; the step of performing the interpolation sampling on the original video frame Is according to the corresponding relationship, and generating the output video frame Id specifically comprises: I d P d = ∑ w i I s P s i ∑ w i wherein wi is an interpolation weight, P s i ∈ U P s δ is a neighborhood coordinate of Ps.