Motion smoothing processing method, device and equipment of video image and storage medium

Abstract

The application is suitable for the technical field of video image processing, and provides a motion smoothing processing method, device and equipment of a video image and a storage medium, the method comprising: smoothing a current original quaternion corresponding to a current video frame of a video stream; smoothing a smooth quaternion queue according to the current original quaternion, a current smooth quaternion obtained by smoothing, and a field angle parameter of a rendered video frame corresponding to the current video frame; obtaining the field angle parameter of the rendered video frame corresponding to a next video frame according to the smooth quaternion queue and the original quaternion queue after processing, and through a field angle parameter calculation algorithm; when the length of the smooth quaternion queue exceeds the number of buffered frames, outputting a smooth video frame image corresponding to the smooth quaternion corresponding to the head of the queue, and setting the field angle parameter of the rendered video frame corresponding to the current video frame as the field angle parameter of the rendered video frame corresponding to the next video frame, thereby improving the real-time performance and anti-shake effect of video image jitter elimination.

Description

Technical Field

[0001] This invention belongs to the field of video image processing technology, and particularly relates to a method, apparatus, device and storage medium for motion smoothing processing of video images. Background Technology

[0002] With the development of technology and the progress of society, mobile devices have become an indispensable part of people's lives. People can use mobile devices to shoot videos, record every moment of their lives, and share them with friends. Therefore, video shooting has gradually become a way for many users to share and communicate on social media. Although the camera configurations of mobile devices are becoming more and more advanced, resulting in clear and high-quality video images when the mobile device is stationary, when users hold the mobile device while walking, running, or in other complex motion scenarios (such as when it is placed in a car while driving), the mobile device becomes unstable. In this case, the recorded video data often exhibits shaking, which can easily cause blurry video images and result in unsatisfactory video quality.

[0003] In US10027893B2, a method for real-time video stabilization of mobile devices based on vehicle motion sensing is disclosed. While this solution can adjust the current angular velocity in a timely manner according to future motion conditions to avoid sudden out-of-bounds changes in angular velocity, it is difficult to balance motion tracking and image jitter elimination under complex and intense motion conditions. In CN202011541615.7, a method, apparatus, device, and storage medium for motion smoothing of video images are disclosed. While this solution solves the problem of real-time balancing of motion tracking and real-time elimination of image jitter under complex and intense motion conditions, it is prone to poor smoothness when the difference between the rendered video field of view (FOV) and the original video FOV is small and the motion jitter amplitude is large, due to the insufficient space for smoothing adjustment. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, and storage medium for motion smoothing of video images, aiming to solve the problem that the existing technology has poor real-time performance and poor anti-shake effect when the difference between the rendered video field of view and the original video field of view is small in complex motion scenes.

[0005] On one hand, the present invention provides a motion smoothing processing method for video images, the method comprising the following steps:

[0006] Obtain the current camera pose corresponding to the current video frame of the video stream, and represent the current camera pose using a quaternion to obtain the current original quaternion corresponding to the current video frame;

[0007] Obtain the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frames, and smooth the current original quaternion according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame.

[0008] Based on the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue is smoothed to obtain the processed smoothed quaternion queue.

[0009] Based on the processed smooth quaternion queue and the original quaternion queue of the cached frames, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained through a preset field of view parameter calculation algorithm.

[0010] When the queue length of the processed smooth quaternion queue exceeds the number of cached frames, the smooth video frame image corresponding to the smooth quaternion at the head of the queue is output, and the field of view parameter of the current video frame corresponding to the rendered video frame is set as the field of view parameter of the next video frame corresponding to the rendered video frame, thereby realizing smoothing and field of view rendering processing of the next video frame corresponding to the rendered video frame.

[0011] Preferably, the step of obtaining the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frames, and smoothing the current original quaternion based on the smoothed quaternion of the previous video frame includes:

[0012] The current original quaternion is subjected to spherical linear interpolation using the smoothed quaternion of the previous video frame.

[0013] More preferably, the step of performing spherical linear interpolation on the current original quaternion using the smoothed quaternion of the previous video frame includes:

[0014] Using the smoothed quaternion of the previous video frame, spherical linear interpolation is performed on the current original quaternion using a preset smoothing formula v(k) = slerp(v(k-1), p(k), weight), where p(k) is the original quaternion of the k-th frame, v(k) is the smoothed quaternion of the k-th frame, v(k-1) is the smoothed quaternion of the (k-1)-th frame, weight is the interpolation weight of p(k) during spherical linear interpolation, and k is an integer greater than 1.

[0015] Preferably, the step of smoothing the smoothed quaternion queue based on the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, to obtain the processed smoothed quaternion queue includes:

[0016] Based on the original quaternion queue and the corresponding smoothed quaternion queue, the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly to obtain the adjusted smoothed quaternion queue.

[0017] Based on the current original quaternion and the current smoothed quaternion, update the original quaternion queue and the adjusted smoothed quaternion queue respectively to obtain the updated smoothed quaternion queue;

[0018] The updated smooth quaternion queue is iteratively smoothed based on the field of view parameters of the rendered video frame corresponding to the current video frame, resulting in a processed smooth quaternion queue.

[0019] More preferably, the step of adjusting the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue includes:

[0020] Based on the original quaternion corresponding to each video frame in the original quaternion queue, the following formula is used: v(m) = slerp(v(m), p(m)). smooth weight follow The smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly, where v(m) is the smoothed quaternion of the m-th frame in the smoothed quaternion queue, and p(m) is the smoothed quaternion. smooth The weight represents the original quaternion in the m-th frame after unconstrained iterative quaternion smoothing of the original quaternion queue. follow To follow the weights, m is an integer greater than 1.

[0021] More preferably, the step of iteratively smoothing the updated smooth quaternion queue based on the field of view parameters of the rendered video frame corresponding to the current video frame includes:

[0022] The updated smooth quaternion queue is subjected to iterative quaternion smoothing processing, and it is determined whether the smooth quaternion of each video frame in the updated smooth quaternion queue will cause out-of-bounds errors under the field of view parameters of the rendered video frame corresponding to the current video frame after each iteration of smoothing.

[0023] Preferably, the step of obtaining the field of view parameters of the rendered video frame corresponding to the next video frame through a preset field of view parameter calculation algorithm includes:

[0024] According to the preset quaternion selection rules, the smoothed quaternion and the original quaternion of the corresponding frame are selected from the processed smoothed quaternion queue and the original quaternion queue, respectively.

[0025] Calculate the relative quaternion between the selected smoothed quaternion and the original quaternion;

[0026] Based on the relative quaternion and the preset parameter calculation formula, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained.

[0027] On the other hand, the present invention provides a motion smoothing processing apparatus for video images, the apparatus comprising:

[0028] The first parameter acquisition unit is used to obtain the current camera pose corresponding to the current video frame of the video stream, and to represent the current camera pose using a quaternion to obtain the current original quaternion corresponding to the current video frame.

[0029] The second parameter acquisition unit is used to obtain the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frame, and to smooth the current original quaternion according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame.

[0030] The queue smoothing processing unit is used to smooth the smoothed quaternion queue according to the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, so as to obtain the processed smoothed quaternion queue.

[0031] The third parameter acquisition unit is used to obtain the field of view parameters of the rendered video frame corresponding to the next video frame based on the processed smooth quaternion queue and the original quaternion queue of the cached frames, using a preset field of view parameter calculation algorithm; and

[0032] The smoothing image output unit is used to output the smoothed video frame image corresponding to the smoothing quaternion at the head of the processed smoothing quaternion queue when the queue length of the processed smoothing quaternion queue exceeds the number of frames of the buffered frames, and to set the field of view parameter of the rendered video frame corresponding to the current video frame as the field of view parameter of the rendered video frame corresponding to the next video frame, thereby realizing smoothing and field of view rendering processing of the rendered video frame corresponding to the next video frame.

[0033] On the other hand, the present invention also provides a video image processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the steps described in the above-described video image motion smoothing processing method.

[0034] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described motion smoothing processing method for video images.

[0035] This invention smooths the original quaternion corresponding to the current video frame of the acquired video stream to obtain the corresponding smoothed quaternion. Based on the original quaternion, the smoothed quaternion, and the field of view parameter of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue of the buffered frames is smoothed. Based on the smoothed quaternion queue and the original quaternion queue of the buffered frames, the field of view parameter of the rendered video frame corresponding to the next video frame is obtained through a preset field of view parameter calculation algorithm. When the queue length of the smoothed quaternion queue exceeds the number of buffered frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output, and the field of view parameter of the rendered video frame corresponding to the current video frame is set as the field of view parameter of the rendered video frame corresponding to the next video frame. This improves the real-time performance of video image jitter elimination in various complex motion scenes when the difference between the rendered video field of view and the original video field of view is small, thereby improving the anti-shake effect. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the implementation of the motion smoothing processing method for video images provided in Embodiment 1 of the present invention.

[0037] Figure 2 This is a flowchart illustrating the implementation of the motion smoothing processing method for video images provided in Embodiment 2 of the present invention;

[0038] Figure 3 This is a schematic diagram of the motion smoothing processing device for video images provided in Embodiment 3 of the present invention;

[0039] Figure 4 This is a preferred structural schematic diagram of the motion smoothing processing device for video images provided in Embodiment 3 of the present invention; and

[0040] Figure 5 This is a schematic diagram of the structure of the video image processing device provided in Embodiment 4 of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] The specific implementation of the present invention will be described in detail below with reference to specific embodiments:

[0043] Example 1:

[0044] Figure 1 The implementation flow of the motion smoothing processing method for video images provided in Embodiment 1 of the present invention is illustrated. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:

[0045] In step S101, the current camera pose corresponding to the current video frame of the video stream is obtained, and the current camera pose is represented by a quaternion to obtain the current original quaternion corresponding to the current video frame.

[0046] This invention is applicable to video image processing devices such as video display and acquisition. In this invention, after video recording begins, the current camera pose corresponding to the current video frame is obtained. The camera pose is three-dimensional spatial information, which is represented by quaternions to obtain the current original quaternion corresponding to the current video frame.

[0047] In step S102, the smoothed quaternion of the previous video frame corresponding to the current video frame is obtained from the smoothed quaternion queue of the cached frames. The current original quaternion is smoothed according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame.

[0048] In this embodiment of the invention, a buffer frame is set up to buffer the original quaternions and smoothed quaternions related to the video frame in the form of a queue. After obtaining the current original quaternion corresponding to the current video frame, the smoothed quaternion of the previous video frame corresponding to the current video frame is obtained from the smoothed quaternion queue of the buffer frame. The current original quaternion is smoothed according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame. The previous video frame is the frame before the current video frame in two adjacent frames. When the current video frame is the first video image of the video stream, the smoothed current quaternion is the current original quaternion, that is, v(1) = p(1), p(1) is the original quaternion of the first video frame, and v(1) is the smoothed quaternion of the first video frame.

[0049] When smoothing the current original quaternion based on the smoothed quaternion of the previous video frame, preferably, spherical linear interpolation is performed on the current original quaternion using the smoothed quaternion of the previous video frame, thereby improving the smoothness of the transition between the two video images and making the presented video image more stable.

[0050] In step S103, the smoothed quaternion queue is smoothed according to the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, so as to obtain the processed smoothed quaternion queue.

[0051] In this embodiment of the invention, the specific implementation process of smoothing the smooth quaternion queue based on the current original quaternion, the current smooth quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame is detailed in Embodiment 2.

[0052] In step S104, based on the processed smooth quaternion queue and the original quaternion queue of the cached frame, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained by a preset field of view parameter calculation algorithm.

[0053] In this embodiment of the invention, when obtaining the field of view parameters of the rendered video frame corresponding to the next video frame through a preset field of view parameter calculation algorithm, preferably, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained through the following steps:

[0054] (1) According to the preset quaternion selection rules, select the smooth quaternion and original quaternion of the corresponding frame from the processed smooth quaternion queue and the original quaternion queue respectively.

[0055] In this embodiment of the invention, according to a preset quaternion selection rule, smooth quaternions and original quaternions of the corresponding frames are selected from the processed smooth quaternion queue and the original quaternion queue, respectively. The preset quaternion selection rule may be to select the quaternion corresponding to each frame in the queue, or to select the quaternion corresponding to a portion of frames from the queue at intervals, or other selection rules. This invention does not impose any specific limitations.

[0056] (2) Calculate the relative quaternion between the selected smooth quaternion and the original quaternion.

[0057] In this embodiment of the invention, the relative quaternion Δq between the selected smooth quaternion and its corresponding original quaternion is calculated.

[0058] (3) Based on the relative quaternion and the preset parameter calculation formula, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained.

[0059] In this embodiment of the invention, the relative quaternion Δq is used as the relative rotation amount. This relative rotation amount is used to perform polygon clipping on each selected corresponding frame to obtain clipped polygons. The distance between the projection of the clipped polygons onto the original image and the boundary of the original image is calculated. The projection state is determined based on the average value of each metric. Then, based on the projection state and a preset parameter calculation formula, the field of view parameters of the next video frame corresponding to the current video frame are obtained. The average value of each metric is generally between 0.0 and 1.0. The larger the average value (i.e., the closer to 1.0), the closer the projection of the clipped polygons onto the original image and the boundary of the original image. The closer the values ​​are, the smaller the average of the metrics (i.e., closer to 0.0), indicating that the projection of the clipped polygon onto the original image is farther from the original image boundary. Here, based on the average of the metrics, the projection state is divided into three states: the first state, the second state, and the third state. The first state is very close to the boundary (i.e., the average of the metrics is close to 1.0) or out of bounds (i.e., the average of the metrics is greater than 1.0). The second state is within the boundary and the distance from the boundary is between relatively far and very close (e.g., the average of the metrics is 0.5). The third state is within the boundary and relatively far from the boundary (i.e., the average of the metrics is close to 0.0). The preset parameter calculation formula is... FOV cur The field of view (FOV) is the rendering field of view parameter for the video frame corresponding to the current video frame. next The field of view (FOV) is the parameter used to render the next video frame. max To maximize the rendering FOV, FOV min For minimum rendering FOV, FOV decrease_step Decrease the step size of FOV, FOV increase_step Increase the step size for FOV.

[0060] By using the above steps (1) to (3), the field of view parameters of the rendered video frame corresponding to the next video frame are obtained, thereby improving the smoothness and field of view rendering effect of the rendered video frame corresponding to the next video frame.

[0061] In step S105, when the queue length of the processed smooth quaternion queue exceeds the number of cached frames, the smooth video frame image corresponding to the smooth quaternion at the head of the queue is output, and the field of view parameter of the current video frame corresponding to the rendered video frame is set as the field of view parameter of the next video frame corresponding to the rendered video frame, thereby realizing smoothing and field of view rendering processing of the next video frame corresponding to the rendered video frame.

[0062] In this embodiment of the invention, the queue length of the smoothed quaternion queue is detected in real time. When the queue length of the processed smoothed quaternion queue exceeds the preset number of buffer frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output. The smoothed video frame image is then rendered using the field of view parameter of the rendered video frame corresponding to the current video frame. The field of view parameter of the rendered video frame corresponding to the current video frame is then set as the field of view parameter of the rendered video frame corresponding to the next video frame. This achieves smoothing and field of view rendering processing for the rendered video frame corresponding to the next video frame corresponding to the current video frame. The above steps S101 to S105 are repeated until the entire real-time recorded video stream is finished and processed.

[0063] Preferably, the number of cached frames can be set to 5, thereby meeting the needs of hardware devices with a small number of cached frames.

[0064] After outputting the smoothed video frame image corresponding to the smoothed quaternion at the head of the processed smoothed quaternion queue, preferably, the head of the original quaternion queue is dequeued, and the head of the processed smoothed quaternion queue is dequeued, thus updating the original quaternion queue and the smoothed quaternion queue, thereby improving the real-time performance of video image stabilization by setting the queues in the buffered frames.

[0065] In this embodiment of the invention, the original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding smoothed quaternion. Based on the original quaternion, the smoothed quaternion, and the field of view parameter of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue of the cached frames is smoothed. Based on the smoothed quaternion queue and the original quaternion queue of the cached frames, the field of view parameter of the rendered video frame corresponding to the next video frame is obtained through a preset field of view parameter calculation algorithm. When the queue length of the smoothed quaternion queue exceeds the number of cached frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output, and the field of view parameter of the rendered video frame corresponding to the current video frame is set as the field of view parameter of the rendered video frame corresponding to the next video frame. This improves the real-time performance of video image jitter elimination in various complex motion scenes when the difference between the rendered video field of view and the original video field of view is small, thereby improving the anti-shake effect.

[0066] Example 2:

[0067] Figure 2 The implementation flow of the motion smoothing processing method for video images provided in Embodiment 2 of the present invention is illustrated. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:

[0068] In step S201, the current camera pose corresponding to the current video frame of the video stream is obtained, and the current camera pose is represented by a quaternion to obtain the current original quaternion corresponding to the current video frame.

[0069] This invention is applicable to video image processing devices such as video display and acquisition. In this invention, after video recording begins, the current camera pose corresponding to the current video frame is obtained. The camera pose is three-dimensional spatial information, which is represented by quaternions to obtain the current original quaternion corresponding to the current video frame.

[0070] In step S202, the smoothed quaternion of the previous video frame corresponding to the current video frame is obtained from the smoothed quaternion queue of the buffered frames. The smoothed quaternion of the previous video frame is used to perform spherical linear interpolation on the current original quaternion to obtain the current smoothed quaternion corresponding to the current video frame.

[0071] In this embodiment of the invention, a buffer frame is set up to cache the original quaternions and smoothed quaternions related to the video frame in the form of a queue. After obtaining the current original quaternion corresponding to the current video frame, the smoothed quaternion of the previous video frame corresponding to the current video frame is obtained from the smoothed quaternion queue of the buffer frame. Based on the smoothed quaternion of the previous video frame, the current original quaternion is processed by spherical linear interpolation (Slerp) to obtain the current smoothed quaternion corresponding to the current video frame. The previous video frame is the frame before the current video frame in two adjacent frames. When the current video frame is the first video image of the video stream, the current smoothed quaternion after smoothing is the current original quaternion, that is, v(1) = p(1), where p(1) is the original quaternion of the first video frame and v(1) is the smoothed quaternion of the first video frame.

[0072] When performing spherical linear interpolation on the current original quaternion using the smoothed quaternion of the previous video frame, preferably, the current original quaternion is subjected to spherical linear interpolation using the smoothed quaternion of the previous video frame through a preset smoothing formula v(k) = slerp(v(k-1), p(k), weight), where p(k) is the original quaternion of the k-th frame, v(k) is the smoothed quaternion of the k-th frame, v(k-1) is the smoothed quaternion of the (k-1)-th frame, weight is the interpolation weight of p(k) during spherical linear interpolation, and k is an integer greater than 1. This improves the smoothness of the transition between the two video images and makes the presented video image more stable.

[0073] When performing spherical linear interpolation on the current original quaternion using the preset smoothing formula v(k) = slerp(v(k-1), p(k), weight), more preferably, the smoothed quaternion v(k-1) of the previous frame is first used as the initial value to calculate the relative quaternion Δq between it and the original quaternion p(k). Then, the metric value rate of the distance between the cropped polygon and the original image boundary of the current video frame when the relative rotation amount is Δq is calculated. Here, rate is 0 when the cropped polygon is inside the inner border and 1 when it is outside the outer border. When it is between the inner and outer borders, the closer it is to the outer border, the larger the metric value. Then, the metric value rate and weight are calculated according to the formula. The weights are calculated, where min_weight represents the minimum weight, typically between 0.001 and 0.4, and inner_rate represents the metric corresponding to the inner border, ranging from 0 to 1. A larger inner_rate value results in higher smoothness but slower tracking; excessively large values ​​can cause significant jitter when the weights go out of bounds, while smaller values ​​result in relatively lower smoothness but tighter tracking. α controls the sharpness of weight changes, typically between 0.5 and 4. Finally, the original quaternion is smoothed using the smoothing formula v(k) = slerp(v(k-1), p(k), weight) to obtain the current smoothed quaternion corresponding to the current video frame, thereby further improving the smoothness of the transition between two consecutive video frames and making the presented video image more stable.

[0074] In step S203, based on the original quaternion queue and the corresponding smoothed quaternion queue of the buffered frame, the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly to obtain the adjusted smoothed quaternion queue.

[0075] In this embodiment of the invention, based on the original quaternion queue and the corresponding smooth quaternion queue of the buffered frame, the smooth quaternion corresponding to each video frame in the smooth quaternion queue is adjusted to make the smooth quaternion move closer to its corresponding original quaternion, thus obtaining the adjusted smooth quaternion queue.

[0076] When adjusting the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue, preferably, the adjustment is performed based on the original quaternion corresponding to each video frame in the original quaternion queue, using a preset following formula v(m) = slerp(v(m), p(m)). smooth weight follow The smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly, where v(m) is the smoothed quaternion of the m-th frame in the smoothed quaternion queue, and p(m) is the smoothed quaternion. smooth The weight represents the original quaternion in the m-th frame after unconstrained iterative quaternion smoothing of the original quaternion queue.follow To follow the weights, m is an integer greater than 1, thereby improving the smoothness of subsequent video images.

[0077] The preset following formula is used: v(m) = slerp(v(m), p(m)). smooth weight follow When adjusting the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue, more preferably, the relative quaternion Δq between the smoothed quaternion of each video frame in the smoothed quaternion queue and the corresponding original quaternion in the original quaternion queue is calculated first. Then, the metric value of the distance between the cropped polygon and the original image boundary of each video frame in the buffer frame when the relative rotation amount is the corresponding Δq is calculated. The average metric value rate_avr is calculated based on the metric values ​​of all frames in the buffer frame. Then, the following is performed based on rate_avr and the following weight formula. Calculate the weights that follow the original quaternion. follow , where min_weight follow This represents the minimum following weight, typically between 0.002 and 0.4. β controls the sharpness of the weight change, generally between 0.5 and 4. Finally, the following formula is used: v(m) = slerp(v(m), p(m)). smooth weight follow Adjust the buffer frame smoothing quaternion queue to further improve the smoothing effect of subsequent video images.

[0078] In step S204, the original quaternion queue and the adjusted smooth quaternion queue are updated according to the current original quaternion and the current smooth quaternion, respectively, to obtain the updated smooth quaternion queue.

[0079] In this embodiment of the invention, the current original quaternion of the current video frame is added to the original quaternion queue, and the current smoothed quaternion is added to the adjusted smoothed quaternion queue. The update of these two queues is completed to obtain the updated smoothed quaternion queue.

[0080] In step S205, the updated smooth quaternion queue is subjected to iterative quaternion smoothing processing based on the field of view parameters of the rendered video frame corresponding to the current video frame, to obtain the processed smooth quaternion queue.

[0081] In this embodiment of the invention, the updated smooth quaternion queue is subjected to iterative quaternion smoothing processing based on the field of view parameters of the rendered video frame corresponding to the current video frame, thereby obtaining the processed smooth quaternion queue and realizing the noise reduction of the video image corresponding to each smooth quaternion in the smooth quaternion queue and the smoothing of its camera pose.

[0082] When performing iterative quaternion smoothing on the updated smooth quaternion queue based on the field of view parameters of the rendered video frame corresponding to the current video frame, preferably, the updated smooth quaternion queue is subjected to iterative quaternion smoothing, and a judgment is made on whether the smooth quaternion of each video frame in the updated smooth quaternion queue after each iteration will cause it to go out of bounds under the field of view parameters of the rendered video frame corresponding to the current video frame. That is, it is judged whether the projection of the cropped area of ​​each smoothed video frame into the original picture of the video frame exceeds the effective range of the picture. When it will cause it to go out of bounds, the quaternion closest to the smooth quaternion and guaranteed not to go out of bounds is found by binary search algorithm, and the quaternion is used as the smoothed result of the frame corresponding to the smooth quaternion, thereby improving the smoothing and noise reduction effect on the video image corresponding to the smooth quaternion.

[0083] In step S206, based on the processed smooth quaternion queue and the original quaternion queue of the cached frame, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained by a preset field of view parameter calculation algorithm.

[0084] In this embodiment of the invention, the specific implementation of step S206 can be referred to the description of step S104 in Embodiment 1, and will not be repeated here.

[0085] In step S207, when the queue length of the processed smooth quaternion queue exceeds the number of cached frames, the smooth video frame image corresponding to the smooth quaternion at the head of the processed smooth quaternion queue is output, and the field of view parameter of the rendered video frame corresponding to the current video frame is set as the field of view parameter of the rendered video frame corresponding to the next video frame, so as to perform smoothing and field of view rendering processing on the next video frame corresponding to the current video frame.

[0086] In this embodiment of the invention, the queue length of the smoothed quaternion queue is detected in real time. When the queue length of the processed smoothed quaternion queue exceeds the preset number of buffer frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output. The smoothed video frame image is then rendered using the field of view parameter of the rendered video frame corresponding to the current video frame. The field of view parameter of the rendered video frame corresponding to the current video frame is then set as the field of view parameter of the rendered video frame corresponding to the next video frame. This achieves smoothing and field of view rendering processing for the rendered video frame corresponding to the next video frame corresponding to the current video frame. The above steps S201 to S206 are repeated until the entire real-time recorded video stream is finished and processed.

[0087] Preferably, the number of cached frames is set to 5, thereby meeting the needs of hardware devices with a small number of cached frames.

[0088] After outputting the smoothed video frame image corresponding to the smoothed quaternion at the head of the processed smoothed quaternion queue, preferably, the head of the original quaternion queue is dequeued, and the head of the processed smoothed quaternion queue is dequeued, thus updating the original quaternion queue and the smoothed quaternion queue, thereby improving the real-time performance of video image stabilization by setting the queues in the buffered frames.

[0089] In this embodiment of the invention, the original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding smoothed quaternion. Based on the original quaternion corresponding to each video frame in the original quaternion queue of the cached frames, the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly. Based on the current original quaternion and the current smoothed quaternion, the original quaternion queue and the adjusted smoothed quaternion queue are updated respectively. Furthermore, based on the field of view parameter of the rendered video frame corresponding to the current video frame, the updated smoothed quaternion queue undergoes iterative quaternion smoothing processing. Finally, based on the processed smoothed quaternion... The system uses a queue and a raw quaternion queue to obtain the field of view parameters of the rendered video frame corresponding to the next video frame through a preset field of view parameter calculation algorithm. When the queue length of the processed smooth quaternion queue exceeds the number of cached frames, it outputs the smooth video frame image corresponding to the smooth quaternion at the head of the processed smooth quaternion queue, and sets the field of view parameters of the rendered video frame corresponding to the current video frame as the field of view parameters of the rendered video frame corresponding to the next video frame. This improves the real-time performance of video image jitter elimination in various complex motion scenes when the difference between the rendered video field of view and the raw video field of view is small, thereby improving the anti-shake effect.

[0090] Example 3:

[0091] Figure 3 The structure of the motion smoothing processing apparatus for video images provided in Embodiment 3 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, including:

[0092] The first parameter acquisition unit 31 is used to obtain the current camera pose corresponding to the current video frame of the video stream, and to represent the current camera pose using a quaternion to obtain the current original quaternion corresponding to the current video frame.

[0093] The second parameter acquisition unit 32 is used to obtain the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the buffered frame, and to smooth the current original quaternion according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame.

[0094] The queue smoothing processing unit 33 is used to smooth the smoothing quaternion queue according to the current original quaternion, the current smoothing quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, so as to obtain the processed smoothing quaternion queue.

[0095] The third parameter acquisition unit 34 is used to obtain the field of view parameters of the rendered video frame corresponding to the next video frame by means of a preset field of view parameter calculation algorithm, based on the processed smooth quaternion queue and the original quaternion queue of the cached frame.

[0096] The smoothing image output unit 35 is used to output the smoothed video frame image corresponding to the smoothing quaternion at the head of the processed smoothing quaternion queue when the queue length of the processed smoothing quaternion queue exceeds the number of buffer frames, and set the field of view parameter of the current video frame corresponding to the rendered video frame to the field of view parameter of the next video frame corresponding to the rendered video frame, thereby realizing smoothing and field of view rendering processing of the next video frame corresponding to the rendered video frame.

[0097] Among them, such as Figure 4 As shown, preferably, the second parameter obtaining unit 32 includes:

[0098] The parameter acquisition subunit 321 is used to perform spherical linear interpolation on the current original quaternion using the smoothed quaternion of the previous video frame.

[0099] The queue smoothing processing unit 33 includes:

[0100] The queue following adjustment unit 331 is used to follow and adjust the smooth quaternion corresponding to each video frame in the smooth quaternion queue according to the original quaternion queue and the corresponding smooth quaternion queue, so as to obtain the adjusted smooth quaternion queue.

[0101] The first queue update unit 332 is used to update the original quaternion queue and the adjusted smooth quaternion queue respectively based on the current original quaternion and the current smooth quaternion, to obtain the updated smooth quaternion; and

[0102] The iterative smoothing processing unit 333 is used to perform iterative quaternion smoothing processing on the updated smooth quaternion queue according to the field of view parameters of the rendered video frame corresponding to the current video frame, so as to obtain the processed smooth quaternion queue.

[0103] The queue following adjustment unit 331 includes:

[0104] The follow adjustment subunit 3311 is used to adjust the original quaternion corresponding to each video frame in the original quaternion queue according to the preset follow formula v(m)=slerp(v(m),p(m)). smooth weight followThe smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly, where v(m) is the smoothed quaternion of the m-th frame in the smoothed quaternion queue, and p(m) is the smoothed quaternion. smooth The weight represents the original quaternion in the m-th frame after unconstrained iterative quaternion smoothing of the original quaternion queue. follow To follow the weights, m is an integer greater than 1.

[0105] The iterative smoothing processing unit 333 includes:

[0106] The smoothing subunit 3331 is used to perform iterative quaternion smoothing on the updated smooth quaternion queue, and to determine whether the smooth quaternion of each video frame in the updated smooth quaternion queue will cause out-of-bounds errors under the field of view parameters of the corresponding rendered video frame after each iteration of smoothing.

[0107] The third parameter acquisition unit 34 includes:

[0108] Quaternion selection unit 341 is used to select the smoothed quaternion and the original quaternion of the corresponding frame from the processed smoothed quaternion queue and the original quaternion queue respectively according to the preset quaternion selection rules.

[0109] Quaternion calculation unit 342 is used to calculate the relative quaternion between the selected smoothed quaternion and the original quaternion; and

[0110] The rendering parameter acquisition unit 343 is used to obtain the field of view parameters of the rendered video frame corresponding to the next video frame based on the relative quaternion and the preset parameter calculation formula.

[0111] In this embodiment of the invention, each unit of the motion smoothing processing device for video images can be implemented by corresponding hardware or software units. Each unit can be an independent hardware or software unit, or it can be integrated into a single hardware or software unit, which is not intended to limit the invention. Specifically, the implementation methods of each unit can be referred to the description of the foregoing method embodiments, and will not be repeated here.

[0112] Example 4:

[0113] Figure 5 The structure of the video image processing device provided in Embodiment 4 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0114] The video image processing device 5 of this embodiment includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps described in the above-described video image motion smoothing processing method embodiment, for example... Figure 1The steps S101 to S105 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each unit in the above-described device embodiments, for example... Figure 3 The functions of units 31 to 35 are shown.

[0115] In this embodiment of the invention, the original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding smoothed quaternion. Based on the original quaternion, the smoothed quaternion, and the field of view parameter of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue of the cached frames is smoothed. Based on the smoothed quaternion queue and the original quaternion queue of the cached frames, the field of view parameter of the rendered video frame corresponding to the next video frame is obtained through a preset field of view parameter calculation algorithm. When the queue length of the smoothed quaternion queue exceeds the number of cached frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output, and the field of view parameter of the rendered video frame corresponding to the current video frame is set as the field of view parameter of the rendered video frame corresponding to the next video frame. This improves the real-time performance of video image jitter elimination in various complex motion scenes when the difference between the rendered video field of view and the original video field of view is small, thereby improving the anti-shake effect.

[0116] The video image processing device of this invention can be a smartphone or tablet computer with a camera. The steps implemented by the processor 50 in the video image processing device 5 when executing the computer program 52 to achieve the motion smoothing processing method for video images can be referred to the description of the foregoing method embodiments, and will not be repeated here.

[0117] Example 5:

[0118] In this embodiment of the invention, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps in the above-described embodiments of the motion smoothing processing method for video images. For example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the computer program is executed by the processor, it implements the functions of each unit in the above-described device embodiments, for example... Figure 3 The functions of units 31 to 35 are shown.

[0119] In this embodiment of the invention, the original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding smoothed quaternion. Based on the original quaternion, the smoothed quaternion, and the field of view parameter of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue of the cached frames is smoothed. Based on the smoothed quaternion queue and the original quaternion queue of the cached frames, the field of view parameter of the rendered video frame corresponding to the next video frame is obtained through a preset field of view parameter calculation algorithm. When the queue length of the smoothed quaternion queue exceeds the number of cached frames, the smoothed video frame image corresponding to the smoothed quaternion at the head of the queue is output, and the field of view parameter of the rendered video frame corresponding to the current video frame is set as the field of view parameter of the rendered video frame corresponding to the next video frame. This improves the real-time performance of video image jitter elimination in various complex motion scenes when the difference between the rendered video field of view and the original video field of view is small, thereby improving the anti-shake effect.

[0120] The computer-readable storage medium in embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as ROM / RAM, disk, optical disk, flash memory, etc.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for motion smoothing processing of video images, characterized in that, The method includes the following steps: Obtain the current camera pose corresponding to the current video frame of the video stream, and represent the current camera pose using a quaternion to obtain the current original quaternion corresponding to the current video frame; Obtain the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frames, and smooth the current original quaternion according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame. Based on the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, the smoothed quaternion queue is smoothed to obtain the processed smoothed quaternion queue. Based on the processed smooth quaternion queue and the original quaternion queue of the cached frames, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained through a preset field of view parameter calculation algorithm. When the queue length of the processed smooth quaternion queue exceeds the number of cached frames, the smooth video frame image corresponding to the smooth quaternion at the head of the queue is output, and the field of view parameter of the current video frame corresponding to the rendered video frame is set as the field of view parameter of the next video frame corresponding to the rendered video frame, thereby realizing smoothing and field of view rendering processing of the next video frame corresponding to the rendered video frame.

2. The method as described in claim 1, characterized in that, The step of obtaining the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frames, and smoothing the current original quaternion based on the smoothed quaternion of the previous video frame includes: The current original quaternion is subjected to spherical linear interpolation using the smoothed quaternion of the previous video frame.

3. The method as described in claim 2, characterized in that, The step of performing spherical linear interpolation on the current original quaternion using the smoothed quaternion of the previous video frame includes: Using the smoothed quaternion of the previous video frame, spherical linear interpolation is performed on the current original quaternion using a preset smoothing formula v(k) = slerp(v(k-1), p(k), weight), where p(k) is the original quaternion of the k-th frame, v(k) is the smoothed quaternion of the k-th frame, v(k-1) is the smoothed quaternion of the (k-1)-th frame, weight is the interpolation weight of p(k) during spherical linear interpolation, and k is an integer greater than 1.

4. The method as described in claim 1, characterized in that, The step of smoothing the smoothed quaternion queue based on the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, to obtain the processed smoothed quaternion queue, includes: Based on the original quaternion queue and the corresponding smoothed quaternion queue, the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly to obtain the adjusted smoothed quaternion queue. Based on the current original quaternion and the current smoothed quaternion, update the original quaternion queue and the adjusted smoothed quaternion queue respectively to obtain the updated smoothed quaternion queue; The updated smooth quaternion queue is subjected to iterative quaternion smoothing processing based on the field of view parameters of the rendered video frame corresponding to the current video frame, resulting in a processed smooth quaternion queue.

5. The method as described in claim 4, characterized in that, The step of adjusting the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue includes: Based on the original quaternion corresponding to each video frame in the original quaternion queue, the following formula is used: v(m) = slerp(v(m), p(m)). smooth weight follow The smoothed quaternion corresponding to each video frame in the smoothed quaternion queue is adjusted accordingly, where v(m) is the smoothed quaternion of the m-th frame in the smoothed quaternion queue, and p(m) is the smoothed quaternion. smooth The weight represents the original quaternion in the m-th frame after unconstrained iterative quaternion smoothing of the original quaternion queue. follow To follow the weights, m is an integer greater than 1.

6. The method as described in claim 4, characterized in that, The step of iteratively smoothing the updated smooth quaternion queue based on the field of view parameters of the rendered video frame corresponding to the current video frame includes: The updated smooth quaternion queue is subjected to iterative quaternion smoothing processing, and it is determined whether the smooth quaternion of each video frame in the updated smooth quaternion queue will cause out-of-bounds errors under the field of view parameters of the rendered video frame corresponding to the current video frame after each iteration of smoothing.

7. The method as described in claim 1, characterized in that, The step of obtaining the field of view parameters of the rendered video frame corresponding to the next video frame through a preset field of view parameter calculation algorithm includes: According to the preset quaternion selection rules, the smoothed quaternion and the original quaternion of the corresponding frame are selected from the processed smoothed quaternion queue and the original quaternion queue, respectively. Calculate the relative quaternion between the selected smoothed quaternion and the original quaternion; Based on the relative quaternion and the preset parameter calculation formula, the field of view parameters of the rendered video frame corresponding to the next video frame are obtained.

8. A motion smoothing processing device for video images, characterized in that, The device includes: The first parameter acquisition unit is used to obtain the current camera pose corresponding to the current video frame of the video stream, and to represent the current camera pose using a quaternion to obtain the current original quaternion corresponding to the current video frame. The second parameter acquisition unit is used to obtain the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the cached frame, and to smooth the current original quaternion according to the smoothed quaternion of the previous video frame to obtain the current smoothed quaternion corresponding to the current video frame. The queue smoothing processing unit is used to smooth the smoothed quaternion queue according to the current original quaternion, the current smoothed quaternion, and the field of view parameters of the rendered video frame corresponding to the current video frame, so as to obtain the processed smoothed quaternion queue. The third parameter acquisition unit is used to obtain the field of view parameters of the rendered video frame corresponding to the next video frame based on the processed smooth quaternion queue and the original quaternion queue of the cached frames, using a preset field of view parameter calculation algorithm; and The smoothing image output unit is used to output the smoothed video frame image corresponding to the smoothing quaternion at the head of the processed smoothing quaternion queue when the queue length of the processed smoothing quaternion queue exceeds the number of frames of the buffered frames, and to set the field of view parameter of the rendered video frame corresponding to the current video frame as the field of view parameter of the rendered video frame corresponding to the next video frame, thereby realizing smoothing and field of view rendering processing of the rendered video frame corresponding to the next video frame.

9. A video image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images