Video anti-shake processing method and device, computer device and readable storage medium

Abstract

This application relates to a video stabilization method, apparatus, computer device, and readable storage medium. The method includes: acquiring three-dimensional rotation information of an initial video; acquiring global rotational motion information of the initial video based on the three-dimensional rotation information; and performing motion smoothing processing on the global rotational motion information to generate a stabilized video of the initial video. Using the above method, three-dimensional rotation information of the initial video can be acquired, and then processed to achieve three-dimensional stabilization of the initial video, thereby improving the accuracy of the stabilization process and the stabilization effect of the video, and effectively improving the stability and quality of the image in the initial video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video stabilization processing method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] With the development of imaging technology, images or videos are captured using image acquisition devices such as cameras on electronic devices.

[0003] Taking video recording as an example, if the shooting equipment shakes during the recording process, the captured image will be shaky, resulting in poor video quality. Therefore, related technologies often employ image stabilization to ensure the quality of the captured video.

[0004] However, the related technology suffers from poor video stabilization performance. Summary of the Invention

[0005] Therefore, it is necessary to provide a video stabilization method, apparatus, computer equipment, and readable storage medium to address the aforementioned technical problems, which can improve the stabilization effect of the initial video.

[0006] In a first aspect, embodiments of this application provide a video stabilization method, the method comprising:

[0007] Obtain the 3D rotation information of the initial video;

[0008] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0009] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0010] In one embodiment, obtaining the three-dimensional rotation information of the initial video includes:

[0011] Obtain the set of matching point pairs between each two adjacent video frames in the initial video;

[0012] For any two adjacent video frames, determine the target rotation matrix between the two adjacent video frames based on the matching point pair set of the two adjacent video frames;

[0013] The three-dimensional rotation information of the initial video is determined based on the target rotation matrix between each two adjacent video frames.

[0014] In one embodiment, obtaining the set of matching point pairs between each pair of adjacent video frames in the initial video includes:

[0015] For any two adjacent video frames, extract the feature points from the previous video frame to obtain multiple original feature points;

[0016] Optical flow tracing is performed on the next video frame in two adjacent video frames based on each original feature point to obtain multiple tracking feature points in the next video frame.

[0017] Based on each tracking feature point, reflective flow tracing is performed on the previous video frame to obtain multiple anti-tracking feature points in the previous video frame;

[0018] Based on each anti-tracking feature point, determine the set of matching point pairs between two adjacent video frames.

[0019] In one embodiment, determining a set of matching point pairs between two adjacent video frames based on each anti-tracking feature point includes:

[0020] Obtain the distance between each anti-tracking feature point and its corresponding original feature point;

[0021] Anti-tracking feature points with an interval distance less than a preset distance threshold are identified as valid anti-tracking feature points for optical flow tracing;

[0022] Based on all valid anti-tracking feature points of optical flow tracing, determine the set of matching point pairs between two adjacent video frames.

[0023] In one embodiment, determining the target rotation matrix between two adjacent video frames based on the matching point pair set of two adjacent video frames includes:

[0024] Based on the matching point pair set of two adjacent video frames, obtain the optical flow tracing results of two adjacent video frames;

[0025] If the optical flow tracing result of two adjacent video frames is invalid, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames;

[0026] If the optical flow tracing results of two adjacent video frames are valid, then the target rotation matrix between the two adjacent video frames is determined based on the shooting states corresponding to the two adjacent video frames.

[0027] In one embodiment, determining the target rotation matrix between two adjacent video frames based on the shooting states corresponding to two adjacent video frames includes:

[0028] If the shooting state corresponding to two adjacent video frames is a non-static shooting state, then obtain the essential matrix of the matching point pair set of the two adjacent video frames, and obtain the target rotation matrix between the two adjacent video frames based on the essential matrix of the matching point pair set.

[0029] If the shooting state corresponding to two adjacent video frames is a static shooting state, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames.

[0030] In one embodiment, obtaining the target rotation matrix between two adjacent video frames based on the essential matrix of the matching point pair set includes:

[0031] Singular value decomposition is performed on the essential matrix of the matching point pair set to obtain the initial rotation matrix, 3D translation information and depth information set between two adjacent video frames;

[0032] Based on the initial rotation matrix, 3D translation information, and depth information set, obtain the target rotation matrix between two adjacent video frames.

[0033] In one embodiment, the target rotation matrix between two adjacent video frames is obtained based on the initial rotation matrix, 3D translation information, and depth information set, including:

[0034] Based on the depth information set, multiple target matching point pairs are selected from all candidate matching point pairs in the matching point pair set;

[0035] Based on the initial rotation matrix, 3D translation information, and each target matching point pair, the target rotation matrix between two adjacent video frames is determined.

[0036] In one embodiment, determining the target rotation matrix between two adjacent video frames based on the initial rotation matrix, 3D translation information, and each target matching point pair includes:

[0037] By mapping each target matching point pair to the coordinate system of the shooting device using the intrinsic parameters of the shooting device, the mapped coordinates are obtained;

[0038] A nonlinear least squares model is constructed using the initial rotation matrix, 3D translation information, and mapped coordinates.

[0039] Solve the nonlinear least squares model to obtain the target rotation matrix between two adjacent video frames.

[0040] In one embodiment, the method further includes:

[0041] Candidate matching point pairs that meet the preset static point conditions in the matching point pair set are determined as static point pairs;

[0042] If the total number of all stationary point pairs in the matching point pair set is greater than a preset threshold, then the shooting state corresponding to two adjacent video frames is determined to be a stationary shooting state; otherwise, the shooting state corresponding to two adjacent video frames is determined to be a non-stationary shooting state.

[0043] In one embodiment, global rotation motion information of the initial video is obtained based on the three-dimensional rotation information, including:

[0044] Based on the three-dimensional rotation information of the initial video, obtain the global rotation motion matrix of each video frame in the initial video;

[0045] The global rotation motion information of the initial video is determined based on the global rotation motion matrix of each video frame.

[0046] In one embodiment, motion smoothing processing is performed on the global rotational motion information to generate a stabilized video of the initial video, including:

[0047] Motion smoothing processing is performed on the global rotational motion information to obtain motion smoothing data for the initial video.

[0048] Based on the motion smoothing weights of each video frame in the initial video, the global rotational motion information of the initial video, and the motion smoothing data, the target motion smoothing data of the initial video is determined.

[0049] Based on the target motion smoothing data and the initial video, generate a stabilized video.

[0050] In one embodiment, generating a stabilized video based on target motion smoothing data and an initial video includes:

[0051] Based on the target motion smoothing data and global rotation motion information, determine the rotation compensation amount for the initial video;

[0052] The pixels in each video frame are mapped based on the rotation compensation amount of the initial video to generate a stabilized video.

[0053] Secondly, embodiments of this application provide a video stabilization processing device, the device comprising:

[0054] The rotation information acquisition module is used to acquire the three-dimensional rotation information of the initial video.

[0055] The global information acquisition module is used to acquire the global rotation motion information of the initial video based on the three-dimensional rotation information.

[0056] The smoothing module is used to smooth the global rotational motion information and generate a stabilized video of the initial video.

[0057] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0058] Obtain the 3D rotation information of the initial video;

[0059] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0060] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0061] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0062] Obtain the 3D rotation information of the initial video;

[0063] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0064] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0065] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0066] Obtain the 3D rotation information of the initial video;

[0067] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0068] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0069] The video stabilization processing method, apparatus, computer device, and readable storage medium provided in this application include: acquiring three-dimensional rotation information of an initial video; acquiring global rotation motion information of the initial video based on the three-dimensional rotation information; and performing motion smoothing processing on the global rotation motion information to generate a stabilized video of the initial video. Using the above method, three-dimensional rotation information of the initial video can be acquired, and then the three-dimensional rotation information of the initial video can be processed to achieve three-dimensional stabilization processing of the initial video, thereby improving the accuracy of the stabilization processing and the stabilization effect of the video. Attached Figure Description

[0070] Figure 1 This is a structural diagram of a video stabilization processing system in one embodiment;

[0071] Figure 2 This is a flowchart illustrating a video stabilization method in one embodiment;

[0072] Figure 3 This is a flowchart illustrating a video stabilization method in another embodiment;

[0073] Figure 4 This is a flowchart illustrating a video stabilization method in another embodiment;

[0074] Figure 5 This is a flowchart illustrating a video stabilization method in another embodiment;

[0075] Figure 6 This is a flowchart illustrating a video stabilization method in another embodiment;

[0076] Figure 7 This is a flowchart illustrating a video stabilization method in another embodiment;

[0077] Figure 8 This is a flowchart illustrating a video stabilization method in another embodiment;

[0078] Figure 9 This is a flowchart illustrating a video stabilization method in another embodiment;

[0079] Figure 10 This is a flowchart illustrating a video stabilization method in another embodiment;

[0080] Figure 11 This is a flowchart illustrating a video stabilization method in another embodiment;

[0081] Figure 12 This is a flowchart illustrating a video stabilization method in another embodiment;

[0082] Figure 13 This is a flowchart illustrating a video stabilization method in another embodiment;

[0083] Figure 14 This is a structural block diagram of a video stabilization processing device in one embodiment;

[0084] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0086] In the field of new media, capturing images or videos using the built-in video functions of smart devices has gradually become a new way for people to record their lives, disseminate information, and enjoy entertainment. Taking video shooting as an example, if the shooting device shakes during the shooting process, the captured image will be shaky, resulting in poor video quality. Therefore, related technologies often use image stabilization to ensure the quality of the captured video. However, these technologies suffer from poor image stabilization effects. Therefore, this application provides a video stabilization method that can improve the stabilization effect of the initial video.

[0087] The video stabilization method provided in this application embodiment can be applied to... Figure 1 The video stabilization system shown includes a shooting device and a computer device, which are connected via a communication link. This communication method can be Bluetooth, Wi-Fi, mobile network connection, etc. Optionally, the shooting device can be a mobile phone or camera capable of shooting any type of video (e.g., panoramic video and regular video, etc.); the computer device can be a standalone server or a server cluster consisting of multiple servers, and can also be, but is not limited to, various personal computers, laptops, mobile phones, and tablets. This embodiment does not limit the specific form of the shooting device and the computer device; wherein, Figure 1 The illustration uses a computer as the server and a mobile phone as the camera.

[0088] In this system, the shooting device can send the captured initial video to a connected computer device. Correspondingly, after receiving the initial video, the computer device performs image stabilization processing to generate a stabilized video. The following embodiments will specifically describe the process of the video stabilization method, using a computer device as the executing entity.

[0089] like Figure 2 The diagram shown is a flowchart illustrating a video stabilization method provided in an embodiment of this application. This method may include the following steps:

[0090] S100: Obtain the 3D rotation information of the initial video.

[0091] Specifically, the computer device can obtain the three-dimensional rotation matrix between each two adjacent video frames in the initial video, and then determine the three-dimensional rotation information of the initial video based on the three-dimensional rotation matrix between each two adjacent video frames. The three-dimensional rotation matrix between two adjacent video frames can be represented by a 3x3 square matrix.

[0092] In this embodiment, the initial video can be a video shot by the camera in its default shooting mode, or it can be a video shot by the camera in a specific camera movement mode. Correspondingly, the type of the initial video can be a fixed time-lapse video, a moving time-lapse video, a panoramic time-lapse video, etc., and this embodiment does not limit this. Optionally, the camera movement mode can be a push-in shot, a panning shot, a surround shot, etc.

[0093] It should be noted that the initial video mentioned above can be a video directly captured by the shooting device. In this case, the total duration of the initial video is equal to the shooting duration of the shooting device. Alternatively, the initial video can be a video obtained by extracting frames from a video directly captured by the shooting device and then fusing all the extracted video frames. In this case, the total duration of the initial video is less than the shooting duration of the shooting device.

[0094] S200. Based on the three-dimensional rotation information, obtain the global rotation motion information of the initial video.

[0095] The computer device can perform arithmetic operations on the three-dimensional rotation information of the initial video to obtain the global rotation motion information of the initial video. Optionally, the arithmetic operation can be at least one of addition, subtraction, multiplication, division, logarithmic operation, and exponential operation.

[0096] In this embodiment, the global rotation motion information of the initial video may include the global rotation motion matrix of each video frame in the initial video.

[0097] S300 performs motion smoothing on global rotational motion information to generate a stabilized video of the initial video.

[0098] In practical applications, computer devices can use motion smoothing algorithms to smooth global rotational motion information and directly generate a stabilized video of the initial video. Alternatively, they can use motion smoothing algorithms to smooth global rotational motion information to obtain motion smoothing data, and then process the motion smoothing data to generate a stabilized video of the initial video.

[0099] Optionally, the motion smoothing algorithm described above can be moving window least squares polynomial smoothing, Kernel smoothing, additive smoothing, or depth smoothing, etc.

[0100] It should be noted that the video stabilization method described above, when applied to initial videos with significant parallax and scene depth, produces a more noticeable stabilized video compared to the original video. Optionally, parallax can be understood as the greater motion fluctuation between adjacent video frames captured by the camera, or the faster the camera zooms between adjacent frames, resulting in a larger difference between them. Scene depth can be understood as the varying distances between different objects and the camera lens, i.e., different depths of field, also resulting in a larger difference between adjacent video frames.

[0101] The technical solution in this application embodiment obtains the three-dimensional rotation information of the initial video, obtains the global rotation motion information of the initial video based on the three-dimensional rotation information, and performs motion smoothing processing on the global rotation motion information to generate a stabilized video of the initial video. Using the above method, the three-dimensional rotation information of the initial video can be obtained, and then processed to achieve three-dimensional stabilization of the initial video. This improves the accuracy of the stabilization processing and the stabilization effect of the video, and effectively enhances the stability and quality of the image in the initial video, avoiding problems such as video distortion and loss of quality, thereby improving the user experience. This improves user experience; moreover, the method requires no human intervention, saving labor costs associated with image stabilization and reducing human error, thus enhancing the accuracy, speed, and efficiency of video stabilization. Furthermore, the method can stabilize various types of initial videos, offering broad applicability and robustness. Moreover, by employing 3D stabilization, the method considers the actual spatial motion of each video frame in the initial video, resulting in more realistic stabilization and better performance on initial videos with significant parallax and scene depth.

[0102] The process of obtaining the three-dimensional rotation information of the initial video is described below. In one embodiment, as... Figure 3 As shown, the steps in S100 above can be implemented in the following ways:

[0103] S101. Obtain the set of matching point pairs between each two adjacent video frames in the initial video.

[0104] Specifically, the computer device can acquire each video frame in the initial video, and then, for any two adjacent video frames, use a feature point extraction algorithm to extract feature points from the two adjacent video frames respectively to obtain feature points in the two adjacent video frames. Then, perform feature point matching processing on the feature points in the two adjacent video frames to obtain matching point pairs between the two adjacent video frames. Finally, obtain a set of matching point pairs between the two adjacent video frames based on all matching point pairs between the two adjacent video frames.

[0105] Optionally, the aforementioned feature point extraction algorithm can be a grayscale-based corner detection algorithm, a fast corner detection algorithm, or a deep learning-based corner detection algorithm, etc. It should be noted that the total number of matching point pairs obtained through feature point matching processing can be less than or equal to the total number of feature points in any of the two adjacent video frames.

[0106] S102. For any two adjacent video frames, determine the target rotation matrix between the two adjacent video frames based on the matching point pair set of the two adjacent video frames.

[0107] In practical applications, for any two adjacent video frames, a computer device can pre-train an algorithm model, and then input the pixel coordinates of each matching point pair in the matching point pair set of the two adjacent video frames on the corresponding video frame into the algorithm model. The algorithm model outputs the target rotation matrix between the two adjacent video frames.

[0108] In addition, the computer device can perform rotation transformation processing on the pixel coordinates of each matching point pair in the matching point pair set of two adjacent video frames to obtain the target rotation matrix between the two adjacent video frames. Optionally, the target rotation matrix can represent the rotation relationship between two adjacent video frames in three-dimensional space.

[0109] S103. Determine the three-dimensional rotation information of the initial video based on the target rotation matrix between each two adjacent video frames.

[0110] In this embodiment, the target rotation matrix of the first video frame and the target rotation matrices between any two adjacent video frames can be determined as the three-dimensional rotation information of the initial video. Since the first video frame is stationary relative to the point before the initial video was captured, the target rotation matrix of the first video frame can be an identity matrix.

[0111] The technical solution in this application embodiment obtains the matching point pair set of each two adjacent video frames in the initial video. For any two adjacent video frames, the target rotation matrix between the two adjacent video frames is determined based on the matching point pair set, and the three-dimensional rotation information of the initial video is determined based on the target rotation matrix between each two adjacent video frames. This method can determine the three-dimensional rotation information of the initial video by using the matching point pair set of each two adjacent video frames in the initial video. The processing is relatively simple and can be implemented without complex algorithms. It can not only shorten the determination time of the three-dimensional rotation information, but also improve the determination speed of the three-dimensional rotation information.

[0112] The process of obtaining the matching point pair set of each two adjacent video frames in the initial video is described below. In one embodiment, as... Figure 4 As shown, the steps in S101 above may include:

[0113] S111. For any two adjacent video frames, extract the feature points from the previous video frame to obtain multiple original feature points.

[0114] In practical applications, for any two adjacent video frames in an initial video, a computer device can use a feature point extraction algorithm to extract feature points from the preceding video frame to obtain multiple feature points of the preceding video frame, i.e., multiple original feature points. In the embodiments of this application, the above-mentioned feature point extraction algorithm can be the Shi-Tomas corner detection algorithm.

[0115] S121. Based on each original feature point, perform optical flow tracing on the next video frame in two adjacent video frames to obtain multiple tracking feature points in the next video frame.

[0116] Among them, computer equipment can use optical flow tracing to perform optical flow tracing on the next video frame in two adjacent video frames based on each original feature point, thereby obtaining multiple feature points in the next video frame, i.e., multiple tracing feature points.

[0117] Optionally, the optical flow tracing method described above can be either a sparse optical flow tracing method or a dense optical flow tracing method; this embodiment of the application does not limit the specific method. Furthermore, the total number of traced feature points can be less than or equal to the total number of original feature points.

[0118] S131. Perform reflective flow tracing on the previous video frame based on each tracking feature point to obtain multiple anti-tracking feature points in the previous video frame.

[0119] Furthermore, based on the multiple tracking feature points in the subsequent video frame obtained in the preceding steps, optical flow tracing can be used to perform reverse optical flow tracing on the previous video frame to obtain multiple feature points in the previous video frame, i.e., multiple reverse tracking feature points. Optionally, the total number of the above reverse tracking feature points can be less than or equal to the total number of tracking feature points.

[0120] S141. Based on each anti-tracking feature point, determine the matching point pair set of two adjacent video frames.

[0121] Specifically, the computer device can combine each anti-tracking feature point in the previous video frame with the corresponding tracking feature point to obtain a matching point pair, and determine all matching point pairs as the matching point pair set of two adjacent video frames.

[0122] In addition, the computer device can also filter feature points that meet preset filtering conditions from each anti-tracking feature point in the previous video frame, and then combine the feature points that meet the preset conditions with the corresponding tracking feature points to obtain matching point pairs, and determine all matching point pairs as the matching point pair set of two adjacent video frames.

[0123] In one embodiment, such as Figure 5 As shown, the step in S141 above, which determines the set of matching point pairs between two adjacent video frames based on each anti-tracking feature point, can be implemented in the following way:

[0124] S1411. Obtain the distance between each anti-tracking feature point and its corresponding original feature point.

[0125] In practical applications, each anti-tracking feature point in the previous video frame has a corresponding original feature point. However, any anti-tracking feature point and its corresponding original feature point may or may not overlap. Naturally, for any anti-tracking feature point in the previous video frame, the computer device can use a distance calculation method to calculate the distance between the anti-tracking feature point and its corresponding original feature point.

[0126] Optionally, the distance calculation method can be Euclidean distance method, Manhattan distance method, Mahalanobis distance method, Chebyshev distance method, Minkowski distance method or Bach distance method, etc., and this application embodiment does not limit the method.

[0127] S1412. Determine the anti-tracking feature points whose interval distance is less than the preset distance threshold as valid anti-tracking feature points for optical flow tracing.

[0128] Furthermore, it can be determined whether the distance between the anti-tracking feature point and the corresponding original feature point is less than a preset distance threshold. If so, the anti-tracking feature point is determined as a valid feature point for optical flow tracing; otherwise, the anti-tracking feature point is determined as an invalid feature point for optical flow tracing.

[0129] S1413. Based on all valid anti-tracking feature points of optical flow tracing, determine the set of matching point pairs between two adjacent video frames.

[0130] Based on all the anti-tracking feature points that are valid for optical flow tracing in the previous video frame obtained in the previous steps, each valid anti-tracking feature point can be combined with the corresponding tracking feature point in the next video frame to obtain multiple matching point pairs that are valid for optical flow tracing. All matching point pairs that are valid for optical flow tracing can then be combined together to obtain a set of matching point pairs between two adjacent video frames.

[0131] In this embodiment, the computer device can use a distance calculation method to calculate the boundary distances between each effective anti-tracking feature point of optical flow tracing on the previous video frame and the long side and wide side of the previous video frame, respectively. If the boundary distance between the anti-tracking feature point and the long side of the previous video frame is greater than a preset long side distance threshold and the boundary distance between the anti-tracking feature point and the wide side of the previous video frame is greater than a preset wide side distance threshold, the anti-tracking feature point is determined as a candidate anti-tracking feature point. Then, each candidate anti-tracking feature point is combined with the corresponding tracking feature point on the next video frame to obtain multiple candidate matching point pairs. All candidate matching point pairs are combined together to obtain a set of matching point pairs between two adjacent video frames.

[0132] Optionally, the preset long side distance threshold can be equal to or different from the preset wide side distance threshold, and this embodiment does not limit this. The preset long side distance threshold and the preset wide side distance threshold can be user-defined or determined based on historical experience. For example, both the preset long side distance threshold and the preset wide side distance threshold can be equal to 5, 4, 6, etc. In this embodiment, the distance can be understood as the number of pixels.

[0133] For example, if two adjacent video frames are M i and M i+1 The obtained M i and M i+1 The set of matching point pairs P contains m candidate matching point pairs. Correspondingly, the set of matching point pairs P can be represented as...

[0134]

[0135] in, Let represent the j-th candidate matching pair in P (j is greater than or equal to 1 and less than or equal to m). Indicates in M i The pixel coordinates of the last pixel. Indicates in M i+1 The pixel coordinates of the pixel above.

[0136] The technical solution in this application embodiment involves extracting feature points from the preceding video frame to obtain multiple original feature points for any two adjacent video frames. Optical flow tracing is then performed on the following video frame based on these original feature points to obtain multiple tracking feature points. Reverse optical flow tracing is then performed on the preceding video frame based on these tracking feature points to obtain multiple reverse tracking feature points. Finally, a set of matching point pairs between the two adjacent video frames is determined based on these reverse tracking feature points. This method uses feature point extraction and optical flow tracing to determine the set of matching point pairs between adjacent video frames in the initial video. This process is relatively simple and does not require complex algorithms, thus improving the speed of obtaining the set of matching point pairs. Furthermore, this method does not involve complex algorithms, so users do not need to learn much professional knowledge to obtain the set of matching point pairs between adjacent video frames. This process not only saves users' learning costs but also reduces the amount of data processed through simple processing methods.

[0137] In some scenarios, even if matching point pairs exist between two adjacent video frames, the total number of matching point pairs is too small to determine whether the two adjacent video frames are valid for optical flow tracing. Therefore, the following describes the process of determining the optical flow tracing results between two adjacent video frames and the process of determining the optimal rotation matrix between them based on the optical flow tracing results. In one embodiment, as... Figure 6 As shown, the step in S102 above, which determines the target rotation matrix between two adjacent video frames based on the matching point pair set of two adjacent video frames, can be implemented in the following way:

[0138] S112. Obtain the optical flow tracing results of two adjacent video frames based on the matching point pair set of two adjacent video frames.

[0139] Specifically, the computer device can obtain the total number of all matching point pairs in the matching point pair set of two adjacent video frames, and determine whether the total number of all matching point pairs is less than a preset first number threshold. If so, it indicates that the total number of effective feature points for optical flow tracing on two adjacent video frames is small. In this case, the optical flow tracing result of the two adjacent video frames is determined to be invalid. If not, it indicates that the total number of effective feature points for optical flow tracing on two adjacent video frames is large. In this case, the optical flow tracing result of the two adjacent video frames is determined to be valid.

[0140] In addition, the computer device can also obtain the total number of anti-tracking feature points on the previous video frame in two adjacent video frames. If the total number of anti-tracking feature points on the previous video frame is less than a preset second number threshold, it indicates that the total number of effective optical flow tracking feature points on the two adjacent video frames is small. In this case, the optical flow tracking result of the two adjacent video frames is determined to be invalid. If not, it indicates that the total number of effective optical flow tracking feature points on the two adjacent video frames is large. In this case, the optical flow tracking result of the two adjacent video frames is determined to be valid.

[0141] The preset first quantity threshold and the preset second quantity threshold can be equal, can be user-defined, or can be determined based on historical experience values.

[0142] S122. If the optical flow tracing result of two adjacent video frames is that the optical flow tracing is invalid, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames; and if the optical flow tracing result of two adjacent video frames is that the optical flow tracing is valid, then the target rotation matrix between the two adjacent video frames is determined according to the shooting state corresponding to the two adjacent video frames.

[0143] In this embodiment, if it is determined that the optical flow tracing results of two adjacent video frames are invalid, the identity matrix can be determined as the target rotation matrix between the two adjacent video frames. It should be noted that the target rotation matrix is ​​the optimal rotation matrix between the two adjacent video frames.

[0144] The aforementioned identity matrix is ​​also a 3x3 square matrix, for two adjacent video frames M i and M i+1 In other words, the target rotation matrix between two adjacent video frames equals I 3×3 .

[0145] Meanwhile, if it is determined that the optical flow tracing results of two adjacent video frames are valid, the pixel coordinates of each matching point pair in the matching point pair set of the two adjacent video frames can be processed according to the shooting state of the two adjacent video frames captured by the shooting device to obtain the target rotation matrix between the two adjacent video frames.

[0146] The computer device can pre-train an algorithm model, and then input the shooting state corresponding to two adjacent video frames captured by the shooting device and the pixel coordinates of each matching point pair in the matching point pair set of the two adjacent video frames into the algorithm model. The algorithm model outputs the target rotation matrix between two adjacent video frames.

[0147] The technical solution in this application embodiment obtains the optical flow tracing results of two adjacent video frames based on the matching point pair set of two adjacent video frames. If the optical flow tracing results of two adjacent video frames are invalid, the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames. If the optical flow tracing results of two adjacent video frames are valid, the target rotation matrix between the two adjacent video frames is determined based on the shooting state corresponding to the two adjacent video frames. This method can obtain the optical flow tracing results of two adjacent video frames based on the matching point pair set of two adjacent video frames, and determine the target rotation matrix between the two adjacent video frames using corresponding methods based on the optical flow tracing results of the two adjacent video frames. This results in a high accuracy of the final target rotation matrix. At the same time, when determining that the optical flow tracing of two adjacent video frames is valid, this method can combine the shooting state corresponding to the two adjacent video frames to determine the target rotation matrix between the two adjacent video frames, thereby greatly improving the accuracy of the target rotation matrix and providing accurate basic information for subsequent processing, thus improving the accuracy of video stabilization.

[0148] The process of determining the target rotation matrix between two adjacent video frames based on their respective shooting states is described below. In one embodiment, the steps in S132 above may include: if the shooting states corresponding to two adjacent video frames are non-static shooting states, then obtaining the essential matrix of the matching point pair set of the two adjacent video frames, and obtaining the target rotation matrix between the two adjacent video frames based on the essential matrix of the matching point pair set; if the shooting states corresponding to two adjacent video frames are static shooting states, then determining the identity matrix as the target rotation matrix between the two adjacent video frames.

[0149] Specifically, the computer device can determine whether the shooting state corresponding to two adjacent video frames is a static shooting state. If it is determined that the shooting state corresponding to two adjacent video frames is a static shooting state, the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames. If it is determined that the shooting state corresponding to two adjacent video frames is not a static shooting state, the essential matrix of the matching point pair set of the two adjacent video frames can be obtained, and then arithmetic operations are performed on the essential matrix of the matching point pair set to obtain the target rotation matrix between the two adjacent video frames.

[0150] One way to determine whether the shooting state corresponding to two adjacent video frames is a static shooting state is to compare the pixel values ​​of each pixel in the previous video frame with the pixel values ​​of the corresponding pixels in the next video frame. If the pixel values ​​of all pixels in the previous video frame and the next video frame are equal, the shooting state corresponding to the two adjacent video frames is determined to be a static shooting state; otherwise, the shooting state corresponding to the two adjacent video frames is determined to be a non-static shooting state.

[0151] In addition, due to some calculation errors or shooting environment errors, it is necessary to reserve some error amount for equal pixel values ​​of some pixels to determine the shooting state corresponding to two adjacent video frames. Based on this, another way to determine whether the shooting state corresponding to two adjacent video frames is a static shooting state is to compare the pixel values ​​of each pixel in the previous video frame with the pixel values ​​of the corresponding pixels in the next video frame. If the pixel values ​​of a preset number of pixels in the previous video frame and the next video frame are equal, the shooting state corresponding to the two adjacent video frames is determined to be a static shooting state; otherwise, the shooting state corresponding to the two adjacent video frames is determined to be a non-static shooting state.

[0152] Optionally, the preset number can be less than the total number of pixels in the previous or next video frame, but the preset number cannot be too small. For example, the preset number can be equal to 90%, 93%, 95%, or 98% of the total number of pixels in the previous or next video frame, etc.

[0153] Meanwhile, the essential matrix of each candidate matching point pair in the matching point pair set of two adjacent video frames can be obtained by calculating the essential matrix of two adjacent video frames based on the pixel coordinates of each candidate matching point pair in the matching point pair set of two adjacent video frames using the epipolar constraint principle.

[0154] In practical applications, the epipolar constraint principle is used to determine the relationship between two adjacent video frames M. i and M i+1 Any candidate matching point pair in the set of matching point pairs All of them exist in the epipolar constraint equation in formula (1), that is:

[0155]

[0156] Where K represents the intrinsic parameter matrix of the shooting device, and E represents the essential matrix of two adjacent video frames. Further, the m epipolar constraint equations consisting of m candidate matching point pairs in the matching point pair set are simultaneously solved using an iterative method to obtain the optimal essential matrix E of two adjacent video frames. Optionally, the above iterative method can be the steepest descent method, conjugate iteration method, variable-scale iteration method, least squares method, etc., but in this embodiment, the above iterative method is the random sample consensus method.

[0157] In practical applications, due to measurement noise and calculation errors, the rotation matrix and translation information calculated using candidate matching points will also have slight errors. When this is fed back into the final video, a slightly jittery image that should be static may appear. To improve the accuracy of motion estimation for video frames in static states and the quality of the final video, it is necessary to determine the static shooting state of the video frames. Therefore, in one embodiment, before determining whether the shooting state corresponding to two adjacent video frames is a non-static shooting state or a static shooting state, as follows... Figure 7 As shown, the above method may include the following steps:

[0158] S142. Select the candidate matching point pairs that meet the preset stationary point conditions in the matching point pair set as stationary point pairs.

[0159] Specifically, for any candidate matching point pair in the matching point pair set, the computer device can determine whether the candidate matching point pair meets the preset stationary point condition based on the pixel coordinates of the candidate matching point pair. If so, the candidate matching point pair is determined as a stationary point pair; otherwise, the candidate matching point pair is determined as a non-stationary point pair.

[0160] In this embodiment of the application, for two adjacent video frames M i and M i+1For any candidate matching point pair j in the matching point pair set, it is necessary to satisfy the preset stationary point condition expressed in the following formula (2) in order to determine the candidate matching point pair j as a stationary point pair.

[0161]

[0162] Where, ‖*‖ represents taking the modulus length, th static This indicates a preset threshold.

[0163] S152. If the total number of all stationary point pairs in the matching point pair set is greater than the preset number threshold, then the shooting state corresponding to two adjacent video frames is determined to be a stationary shooting state; otherwise, the shooting state corresponding to two adjacent video frames is determined to be a non-stationary shooting state.

[0164] Ideally, when all candidate matching point pairs in the matching point pair set are stationary point pairs, it can be directly determined that the shooting state corresponding to two adjacent video frames is a stationary shooting state.

[0165] However, in practical applications, due to some calculation errors, a margin of error for stationary point pairs can be reserved to determine the shooting state corresponding to two adjacent video frames. Correspondingly, based on the total number of all stationary point pairs in the matching point pair set obtained in the above steps, it can be determined whether the total number of all stationary point pairs in the matching point pair set is greater than a preset threshold. If so, the shooting state corresponding to two adjacent video frames is determined to be a stationary shooting state; otherwise, the shooting state corresponding to two adjacent video frames is determined to be a non-stationary shooting state. In this case, the preset threshold can be equal to 87%, 92%, 95%, or 97% of the total number of candidate matching point pairs in the matching point pair set, etc.

[0166] The technical solution in this application embodiment obtains the essential matrix of each candidate matching point pair in the matching point pair set of the two adjacent video frames when the shooting state corresponding to the two adjacent video frames is a non-static shooting state, and obtains the target rotation matrix between the two adjacent video frames based on the essential matrix of each candidate matching point pair. When the shooting state corresponding to the two adjacent video frames is a static shooting state, the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames. This method can accurately determine the target rotation matrix between the two adjacent video frames according to the shooting state corresponding to the two adjacent video frames, so as to provide accurate basic information for subsequent further processing, thereby improving the accuracy of video stabilization.

[0167] In one embodiment, such as Figure 8 As shown, the steps described above for obtaining the target rotation matrix between two adjacent video frames based on the essential matrix of the matching point pair set can be implemented in the following way:

[0168] S132a. Perform singular value decomposition on the essential matrix of the matching point pair set to obtain the initial rotation matrix, three-dimensional translation information and depth information set between two adjacent video frames.

[0169] Specifically, the computer device can perform singular value decomposition on the essential matrix of the matching point pair set to obtain a first rotation matrix, a second rotation matrix, a first translation vector, and a second translation vector. The singular value decomposition results are then combined to obtain four sets of data (namely: the first rotation matrix and the first translation vector, the second rotation matrix and the first translation vector, the second rotation matrix and the second translation vector, and the first rotation matrix and the second translation vector). Based on each set of data, all candidate matching point pairs in the matching point pair set are triangulated to obtain a depth information set. Based on the depth information set corresponding to each set of data, the initial rotation matrix, three-dimensional translation information, and depth information set between two adjacent video frames are determined.

[0170] In practical applications, the depth information set includes m depth information pieces (m equals the total number of candidate matching point pairs in the matching point pair set). Based on the four depth information sets obtained, the total number of positive depth information pieces in the four depth information sets can be obtained. Then, the depth information set with the largest total number of positive depth information pieces is determined as the depth information set of two adjacent video frames. Furthermore, the rotation matrix and translation vector corresponding to the depth information set with the largest total number of positive depth information pieces are determined as the initial rotation matrix and 3D translation information between the two adjacent video frames, respectively. It should be noted that each depth information piece in the depth information set corresponds to a candidate matching point pair.

[0171] Among them, for two adjacent video frames M i and M i+1 As far as M is concerned, i and M i+1 When optical flow tracing is invalid, the 3D translation information of two adjacent video frames equal to 0 3×1 That is, the zero vector equal to 3*1.

[0172] S132b: Based on the initial rotation matrix, 3D translation information, and depth information set, obtain the target rotation matrix between two adjacent video frames.

[0173] The computer device can pre-train an algorithm model, and then input the initial rotation matrix, three-dimensional translation information and depth information set into the algorithm model. The algorithm model outputs the target rotation matrix between two adjacent video frames.

[0174] In addition, computer equipment can perform filtering, optimization, and / or comparison processing based on the initial rotation matrix, 3D translation information, and depth information set to obtain the target rotation matrix between two adjacent video frames.

[0175] In one embodiment, such as Figure 9 As shown, the step in S132b above, which obtains the target rotation matrix between two adjacent video frames based on the initial rotation matrix, 3D translation information, and depth information set, may include:

[0176] S1321. Based on the depth information set, select multiple target matching point pairs from all candidate matching point pairs in the matching point pair set.

[0177] Specifically, the computer device can acquire the positive depth information from the depth information set of two adjacent video frames, and then determine the candidate matching point pairs corresponding to the positive depth information as the target matching point pairs. Optionally, the total number of all target matching point pairs can be less than or equal to the total number of all candidate matching point pairs in the matching point pair set.

[0178] S1322. Based on the initial rotation matrix, three-dimensional translation information, and each target matching point pair, determine the target rotation matrix between two adjacent video frames.

[0179] In practical applications, computer equipment can process the initial rotation matrix, three-dimensional translation information, and each target matching point pair to obtain the target rotation matrix between two adjacent video frames.

[0180] In one embodiment, such as Figure 10 As shown, the step in S1322 above, which determines the target rotation matrix between two adjacent video frames based on the initial rotation matrix, 3D translation information, and each target matching point pair, may include:

[0181] S1322a. Map each target matching point pair to the coordinate system of the shooting device using the intrinsic parameters of the shooting device to obtain the mapped coordinates.

[0182] In this embodiment, the coordinate system of the imaging device is a three-dimensional spatial coordinate system. The computer device can map each target matching point pair to the imaging device coordinate system using the intrinsic parameters of the imaging device, thereby obtaining the corresponding mapped coordinates.

[0183] In one implementation, mapping each target matching point pair to the camera coordinate system using the camera's intrinsic parameters can be achieved by calculating the transformation relationship (i.e., the transformation matrix) between the pixel coordinate system and the camera coordinate system using the camera's intrinsic parameters. Then, this transformation matrix is ​​multiplied by each target matching point to complete the mapping of each target matching point pair to the camera coordinate system. It should be noted that the pixel coordinate system mentioned above is a two-dimensional planar coordinate system, which is the video frame coordinate system.

[0184] In another implementation, mapping each target matching point pair to the camera coordinate system using the camera's intrinsic parameters can be achieved by performing coordinate transformation on each target matching point pair using the camera's intrinsic parameters.

[0185] S1322b: Construct a nonlinear least squares model using the initial rotation matrix, three-dimensional translation information, and mapped coordinates.

[0186] In this embodiment of the application, the computer device can acquire a constructed network model, and then input the initial rotation matrix and three-dimensional translation information between two adjacent video frames, as well as the corresponding mapping coordinates of each target matching point pair, into the constructed network model. The constructed network model outputs a nonlinear least squares model.

[0187] Optionally, the network model can be constructed by combining at least one of the following: convolutional neural network model, fully connected neural network model, recurrent recurrent neural network model, deep belief network model, deep autoencoder and generative adversarial network model. This embodiment of the application does not limit the specific network model.

[0188] S1322c: Solve the nonlinear least squares model to obtain the target rotation matrix between two adjacent video frames.

[0189] Based on the nonlinear least squares model constructed using the above steps, the computer device can employ least squares problem-solving methods to solve the nonlinear least squares model and obtain the target rotation matrix between two adjacent video frames. Optionally, the aforementioned least squares problem-solving method can be at least one of Newton's method, gradient descent method, Gauss-Newton method, and Levenberg-Marquardt method.

[0190] The technical solution in this application embodiment performs singular value decomposition on the essential matrix of the matching point pair set to obtain the initial rotation matrix, 3D translation information, and depth information set between two adjacent video frames. Based on the initial rotation matrix, 3D translation information, and depth information set, the target rotation matrix between two adjacent video frames is obtained. This method determines the target rotation matrix between two adjacent video frames by constructing a nonlinear least squares problem. This process is relatively simple and can be implemented without complex algorithms, thereby improving the speed of obtaining the target rotation matrix. In addition, this method does not have complex algorithms and does not require users to learn too much professional knowledge to obtain the matching point pair set between two adjacent video frames. This process not only saves users' learning costs but also reduces the amount of data processed through simple processing methods.

[0191] In practical applications, to improve the accuracy of video stabilization, it is necessary to combine the motion of each video frame in the initial video throughout the entire video shooting process. Based on this, we can first obtain the global rotational motion information of each video frame in the initial video, and then use this global rotational motion information as reference information to achieve accurate stabilization. The process of obtaining the global rotational motion information of each video frame in the initial video is described below. In one embodiment, as... Figure 11 As shown, the step in S200 above, which obtains the global rotation motion information of the initial video based on the three-dimensional rotation information, may include:

[0192] S201. Based on the three-dimensional rotation information of the initial video, obtain the global rotation motion matrix of each video frame in the initial video.

[0193] Specifically, computer equipment can use the three-dimensional rotation information of the initial video to calculate the global rotation motion matrix of each video frame in the initial video by multiplication.

[0194] For example, if the initial video includes 4 video frames and the 3D rotation information of the initial video is [S1 S2 S3 S4], then the global rotation motion matrix of the first video frame in the initial video can be equal to S1, the global rotation motion matrix of the second video frame in the initial video can be equal to S1*S2, the global rotation motion matrix of the third video frame in the initial video can be equal to S1*S2*S3, and the global rotation motion matrix of the fourth video frame in the initial video can be equal to S1*S2*S3*S4.

[0195] S202. Determine the global rotation motion information of the initial video based on the global rotation motion matrix of each video frame.

[0196] Specifically, the global rotation motion matrix of each video frame in the initial video can be converted to a unit quaternion format to obtain the global rotation motion information corresponding to each video frame in the initial video. This global rotation motion information is then combined to obtain the overall global rotation motion information of the initial video. Optionally, the aforementioned global rotation motion information of the initial video can be understood as the motion estimation result of the initial video.

[0197] The technical solution in this application embodiment obtains the global rotation motion matrix of each video frame in the initial video based on the three-dimensional rotation information of the initial video, and determines the global rotation motion information of the initial video based on the global rotation motion matrix of each video frame. This method can calculate the global rotation motion information of the initial video through the three-dimensional rotation information of the initial video, which is equivalent to taking into account the motion state of each video frame during the entire video shooting process, thereby improving the motion estimation accuracy of each video frame in the initial video. Furthermore, by combining the global rotation motion information of the initial video, video stabilization processing is achieved, making the stabilization process more realistic and improving the accuracy and effect of video stabilization.

[0198] The process of performing motion smoothing on the global rotational motion information to generate the stabilized initial video is described below. In one embodiment, as... Figure 12 As shown, the steps in S300 above can be implemented in the following ways:

[0199] S301. Perform motion smoothing processing on the global rotational motion information to obtain motion smoothing data for the initial video.

[0200] Specifically, the computer device can use a motion smoothing algorithm to perform motion smoothing processing on the global rotational motion information of the initial video to obtain motion smoothed data of the initial video. In the embodiments of this application, the above-mentioned motion smoothing algorithm can be a spherical linear interpolation smoothing algorithm.

[0201] S302. Determine the target motion smoothing data for the initial video based on the motion smoothing weights of each video frame in the initial video, the global rotation motion information of the initial video, and the motion smoothing data.

[0202] In the initial video, there may be cases where two adjacent video frames are not optically traced and the 3D rotation matrix between the two adjacent video frames cannot be calculated. In such cases, the identity matrix is ​​directly determined as the 3D rotation matrix between the two adjacent video frames. Therefore, the final target motion smoothing result needs to consider the motion smoothing weight of each video frame in order to improve the accuracy of the target motion smoothing data of the initial video.

[0203] In practical applications, computer equipment can first obtain the motion smoothing weights of each video frame in the initial video. Then, based on the motion smoothing weights of each video frame in the initial video, it can process the global rotational motion information and / or motion smoothing data of the initial video to obtain the target motion smoothing data of the initial video. It should be noted that the target motion smoothing data of the initial video includes the target motion smoothing results of each video frame in the initial video.

[0204] For example, if the initial video consists of p video frames, the motion smoothing weight set of the initial video is set as φ(w).

[0205] φ(W)={w1,w2,...,w n , ..., w p} (3);

[0206] Among them, w n This represents the motion smoothing weight of the nth video frame in the initial video. If optical flow tracing of the nth video frame is invalid, w n It can be set to 1, and at the same time, with w n Centered on a radius r, take w from φ(w) n A set consisting of r consecutive motion smoothing weights

[0207]

[0208] Optionally, according to The motion smoothing weights and w in the middle n The distance between them can be set with different weights, and the weight values ​​are in the range [0, 1]; the above r is less than (p-1) / 2, for any video frame, if w n-r , ..., w n-1 or w n+1 , ..., w n+r Not in w1, w2, ..., w n , ..., w p If it falls within the range, then it will not be within w1, w2, ..., w n , ..., w p Removing motion smoothing weights within a certain range only requires updating the motion smoothing weights within that range. It's important to note that if optical flow tracing is effective in the nth video frame, the corresponding motion smoothing weight w... n It can be 0.

[0209] In this embodiment of the application, if the motion smoothing weight w of the nth video frame n If the value is 1, then the global rotation motion information of that video frame in the global rotation motion information of the initial video is determined as the target motion smoothing result of that video frame; if the motion smoothing weight w of the nth video frame is... n If the value is 0, then the motion smoothing result of that video frame in the initial video motion smoothing information is determined as the target motion smoothing result of that video frame; if the motion smoothing weight w of the nth video frame is... n Within the range (0, 1), spherical linear interpolation is performed on the global rotation motion information of the video frame and the motion smoothing result of the video frame to obtain the target motion smoothing result of the video frame.

[0210] S303. Generate a stabilized video based on the target motion smoothing data and the initial video.

[0211] The computer equipment can perform data conversion, overlay, and / or data enhancement processing on each video frame in the initial video based on the target motion smoothing data of the initial video to obtain processed video frames. Then, according to the shooting sequence of each video frame in the initial video, the processed video frames are fused to obtain the stabilized video.

[0212] In one embodiment, such as Figure 13 As shown, the step of generating a stabilized video based on the target motion smoothing data and the initial video in S303 above may include:

[0213] S313. Determine the rotation compensation amount of the initial video based on the target motion smoothing data and global rotation motion information.

[0214] Optionally, the computer device can perform arithmetic operations on the target motion smoothing data and the global rotational motion information of the initial video to obtain the rotation compensation amount of the initial video.

[0215] If the initial video consists of p video frames, the target motion smoothing data of the initial video is φ(q″) = {q″1, q″2, ..., q″}. i ,...,q″ p},q″ i Let represent the target motion smoothing result of the i-th video frame in the initial video. The global rotational motion information of the initial video is φ(q) = {q1, q2, ..., q...}. i , ..., q p}, then the rotation compensation amount φ(Δq) of the initial video can be expressed by formula (5):

[0216]

[0217] Among them, symbols This represents quaternion multiplication, Δq i This represents the rotation compensation amount for the i-th video frame in the initial video.

[0218] For example, if a pixel M in two adjacent video frames rotates 90 degrees around the x-axis in the spatial coordinate system, corresponding to a motion smoothing data of 80 degrees, then 10 (i.e., 90-80) can be the rotation compensation amount for the pixel M rotating around the x-axis; at the same time, if a pixel M in two adjacent video frames rotates 90 degrees around the y-axis in the spatial coordinate system, corresponding to a motion smoothing data of 85 degrees, then 5 (i.e., 90-85) can be the rotation compensation amount for the pixel M rotating around the y-axis; and if a pixel M in two adjacent video frames rotates 95 degrees around the z-axis in the spatial coordinate system, corresponding to a motion smoothing data of 80 degrees, then 15 (i.e., 95-80) can be the rotation compensation amount for the pixel M rotating around the z-axis.

[0219] Correspondingly, the rotation compensation amount of pixel M in space between two adjacent video frames can include the rotation compensation amount of pixel M rotating around the x-axis, the rotation compensation amount of pixel M rotating around the y-axis, and the rotation compensation amount of pixel M rotating around the z-axis. It should be noted that the rotation compensation amount of the initial video mentioned above can include the rotation compensation amount of each pixel in each video frame of the initial video. Naturally, the rotation compensation amount of each video frame in the initial video can be represented in matrix form.

[0220] S323. Map the pixels in each video frame according to the rotation compensation amount of the initial video to generate a stabilized video.

[0221] In this embodiment, the computer device can perform mapping processing on the pixels in each video frame according to the rotation compensation amount of each video frame in the initial video, so as to map each pixel in each video frame in the initial video to the stabilized video frame, obtain the pixel coordinates of each pixel in each video frame on the stabilized video frame, and then generate the corresponding stabilized video frame according to the pixel coordinates of each pixel in each video frame on the stabilized video frame. Then, according to the shooting sequence of each video frame in the initial video, the stabilized video frames are fused to obtain the stabilized video.

[0222] In practical applications, the pixels mapped from each video frame in the initial video may fall outside the stabilized video frame. Therefore, after obtaining the mapping coordinates of each pixel in each video frame, the stabilized video frame is first generated based on the mapping coordinates of each pixel. Then, the stabilized video frames corresponding to each video frame are cropped to a uniform size, and the cropped video frames are then fused to obtain the stabilized video.

[0223] In one implementation, the smallest video frame among all stabilized video frames is used as a reference. Other stabilized video frames are cropped to the same size as the smallest video frame, and then the cropped video frames are merged to obtain the stabilized video.

[0224] The technical solution in this application embodiment performs motion smoothing processing on global rotational motion information to obtain motion smoothing data for the initial video. Based on the motion smoothing weights of each video frame in the initial video, the global rotational motion information of the initial video, and the motion smoothing data, the target motion smoothing data for the initial video is determined. Based on the target motion smoothing data and the initial video, a stabilized video is generated. The above method can perform motion smoothing processing on the global rotational motion information of each video frame in the initial video, i.e., the three-dimensional motion estimation result, thereby improving the accuracy of the determined motion smoothing data for the initial video. At the same time, considering that video frames in the initial video where optical flow tracing is invalid cannot achieve motion estimation, this method can incorporate the motion smoothing weights of each video frame in the initial video, thereby determining the optimal motion smoothing data for the initial video, i.e., the target motion smoothing data. Then, based on the target motion smoothing data and the initial video, a stabilized video with better stabilization effect is generated, improving the stability of the video frame display in the initial video and enhancing the picture quality of the initial video.

[0225] In one embodiment, this application also provides a video stabilization method, which includes the following steps:

[0226] (1) For any two adjacent video frames, extract the feature points in the previous video frame to obtain multiple original feature points.

[0227] (2) Perform optical flow tracing on the next video frame in two adjacent video frames based on each original feature point to obtain multiple tracking feature points in the next video frame.

[0228] (3) Perform reflective flow tracing on the previous video frame based on each tracking feature point to obtain multiple anti-tracking feature points in the previous video frame.

[0229] (4) Obtain the distance between each anti-tracking feature point and the corresponding original feature point.

[0230] (5) The anti-tracking feature points with an interval distance less than the preset distance threshold are identified as valid anti-tracking feature points for optical flow tracing.

[0231] (6) Based on all valid anti-tracking feature points of optical flow tracing, determine the matching point pair set of two adjacent video frames.

[0232] (7) For any two adjacent video frames, obtain the optical flow tracing results of the two adjacent video frames based on the matching point pair set of the two adjacent video frames.

[0233] (8) If the optical flow tracing result of two adjacent video frames is invalid, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames.

[0234] (9) If the optical flow tracing results of two adjacent video frames are valid, then determine the target rotation matrix between the two adjacent video frames based on the shooting state corresponding to the two adjacent video frames.

[0235] Among them, the step (9) above, which determines the target rotation matrix between two adjacent video frames based on the shooting states corresponding to two adjacent video frames, includes:

[0236] (91) Determine the candidate matching point pairs that meet the preset stationary point conditions in the matching point pair set as stationary point pairs;

[0237] (92) If the total number of all stationary point pairs in the matching point pair set is greater than the preset number threshold, then the shooting state corresponding to two adjacent video frames is determined to be a stationary shooting state; otherwise, the shooting state corresponding to two adjacent video frames is determined to be a non-stationary shooting state.

[0238] (93) If the shooting state corresponding to two adjacent video frames is a non-static shooting state, then the singular value decomposition is performed on the essential matrix of the matching point pair set to obtain the initial rotation matrix, three-dimensional translation information and depth information set between the two adjacent video frames. Based on the depth information set, multiple target matching point pairs are selected from all candidate matching point pairs in the matching point pair set.

[0239] (94) Map each target matching point pair to the coordinate system of the shooting device using the intrinsic parameters of the shooting device to obtain the mapped coordinates;

[0240] (95) A nonlinear least squares model is constructed using the initial rotation matrix, three-dimensional translation information, and mapped coordinates;

[0241] (96) Solve the nonlinear least squares model to obtain the target rotation matrix between two adjacent video frames;

[0242] Meanwhile, the step (9) above, which determines the target rotation matrix between two adjacent video frames based on the shooting states corresponding to two adjacent video frames, also includes:

[0243] (97) If the shooting state corresponding to two adjacent video frames is a static shooting state, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames.

[0244] (10) Determine the three-dimensional rotation information of the initial video based on the target rotation matrix between each two adjacent video frames.

[0245] (11) Based on the three-dimensional rotation information of the initial video, obtain the global rotation motion matrix of each video frame in the initial video.

[0246] (12) Determine the global rotation motion information of the initial video based on the global rotation motion matrix of each video frame.

[0247] (13) Perform motion smoothing on the global rotation motion information to obtain motion smoothing data of the initial video.

[0248] (14) Determine the target motion smoothing data of the initial video based on the motion smoothing weights of each video frame in the initial video, the global rotation motion information of the initial video, and the motion smoothing data.

[0249] (15) Determine the rotation compensation amount of the initial video based on the target motion smoothing data and global rotation motion information.

[0250] (16) Map the pixels in each video frame according to the rotation compensation amount of the initial video to generate a stabilized video.

[0251] For details of the execution process of (1) to (16) above, please refer to the description of the above embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0252] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0253] Based on the same inventive concept, this application also provides a video stabilization processing apparatus for implementing the video stabilization processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more video stabilization processing apparatus embodiments provided below can be found in the limitations of the video stabilization processing method described above, and will not be repeated here.

[0254] In one embodiment, Figure 14 This is a schematic diagram of the structure of a video stabilization processing device in one embodiment of this application. The video stabilization processing device provided in this embodiment can be applied to computer equipment. Figure 14 As shown, the video stabilization processing device of this application embodiment may include: a rotation information acquisition module 11, a global information acquisition module 12, and a smoothing processing module 13, wherein:

[0255] Rotation information acquisition module 11 is used to acquire the three-dimensional rotation information of the initial video;

[0256] The global information acquisition module 12 is used to acquire the global rotation motion information of the initial video based on the three-dimensional rotation information;

[0257] The smoothing module 13 is used to perform motion smoothing on the global rotational motion information to generate a stabilized video of the initial video.

[0258] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0259] In one embodiment, the rotation information acquisition module 11 includes: a matching point pair set acquisition unit, a target rotation matrix determination unit, and a three-dimensional rotation information determination unit, wherein:

[0260] The matching point pair set acquisition unit is used to acquire the matching point pair set of each two adjacent video frames in the initial video;

[0261] The target rotation matrix determination unit is used to determine the target rotation matrix between any two adjacent video frames based on the matching point pair set of the two adjacent video frames.

[0262] The three-dimensional rotation information determination unit is used to determine the three-dimensional rotation information of the initial video based on the target rotation matrix between each two adjacent video frames.

[0263] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0264] In one embodiment, the matching point pair set acquisition unit includes: a feature point extraction subunit, an optical flow tracing subunit, a reflected optical flow tracing subunit, and a matching point pair set determination subunit, wherein:

[0265] The feature point extraction subunit is used to extract feature points from the previous video frame for any two adjacent video frames to obtain multiple original feature points.

[0266] The optical flow tracing subunit is used to perform optical flow tracing on the next video frame between two adjacent video frames based on each original feature point, so as to obtain multiple tracing feature points in the next video frame.

[0267] The reflective flow tracing subunit is used to perform reflective flow tracing on the previous video frame based on each tracing feature point to obtain multiple anti-tracing feature points in the previous video frame.

[0268] The matching point pair determination subunit is used to determine the matching point pair set between two adjacent video frames based on each anti-tracking feature point.

[0269] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0270] In one embodiment, the matching point pair set determination subunit is specifically used for:

[0271] Obtain the distance between each anti-tracking feature point and its corresponding original feature point;

[0272] Anti-tracking feature points with an interval distance less than a preset distance threshold are identified as valid anti-tracking feature points for optical flow tracing;

[0273] Based on all valid anti-tracking feature points of optical flow tracing, determine the set of matching point pairs between two adjacent video frames.

[0274] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0275] In one embodiment, the target rotation matrix determination unit includes: a tracking result acquisition subunit, a first determination subunit, and a second determination subunit, wherein:

[0276] The tracking result acquisition subunit is used to acquire the optical flow tracking results of two adjacent video frames based on the matching point pair set of two adjacent video frames;

[0277] The first determining subunit is used to determine the identity matrix as the target rotation matrix between the two adjacent video frames when the optical flow tracing result of two adjacent video frames is invalid.

[0278] The second determining subunit is used to determine the target rotation matrix between two adjacent video frames based on the shooting state corresponding to the two adjacent video frames when the optical flow tracing result of the two adjacent video frames is valid.

[0279] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0280] In one embodiment, the second determining subunit includes: a first acquiring subunit and a third determining subunit, wherein:

[0281] The first acquisition subunit is used to acquire the essential matrix of the matching point pair set of two adjacent video frames when the shooting state corresponding to two adjacent video frames is a non-static shooting state, and to acquire the target rotation matrix between two adjacent video frames based on the essential matrix of the matching point pair set.

[0282] The third determining subunit is used to determine the identity matrix as the target rotation matrix between two adjacent video frames when the shooting state corresponding to two adjacent video frames is a static shooting state.

[0283] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0284] In one embodiment, the first acquisition subunit includes: a singular value decomposition subunit and a second acquisition subunit, wherein:

[0285] The singular value decomposition subunit is used to perform singular value decomposition on the essential matrix of the matching point pair set to obtain the initial rotation matrix, 3D translation information and depth information set between two adjacent video frames;

[0286] The second acquisition subunit is used to acquire the target rotation matrix between two adjacent video frames based on the initial rotation matrix, three-dimensional translation information, and depth information set.

[0287] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0288] In one embodiment, the second acquisition subunit is specifically used for:

[0289] Based on the depth information set, multiple target matching point pairs are selected from all candidate matching point pairs in the matching point pair set;

[0290] Based on the initial rotation matrix, 3D translation information, and each target matching point pair, determine the target rotation matrix between two adjacent video frames;

[0291] Specifically, based on the initial rotation matrix, 3D translation information, and each target matching point pair, the target rotation matrix between two adjacent video frames is determined, including:

[0292] By mapping each target matching point pair to the coordinate system of the shooting device using the intrinsic parameters of the shooting device, the mapped coordinates are obtained;

[0293] A nonlinear least squares model is constructed using the initial rotation matrix, 3D translation information, and mapped coordinates.

[0294] Solve the nonlinear least squares model to obtain the target rotation matrix between two adjacent video frames.

[0295] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0296] In one embodiment, the second determining subunit further includes: a stationary point pair determining subunit and a shooting state determining subunit, wherein:

[0297] The stationary point pair determination sub-unit is used to determine the candidate matching point pairs that meet the preset stationary point conditions in the matching point pair set as stationary point pairs;

[0298] The shooting state determination subunit is used to determine the shooting state of two adjacent video frames as a static shooting state when the total number of all static point pairs in the matching point pair set is greater than a preset number threshold; otherwise, it determines the shooting state of two adjacent video frames as a non-static shooting state.

[0299] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0300] In one embodiment, the global information acquisition module 12 includes: a global motion matrix acquisition unit and a global information determination unit, wherein:

[0301] The global motion matrix acquisition unit is used to acquire the global rotation motion matrix of each video frame in the initial video based on the three-dimensional rotation information of the initial video.

[0302] The global information determination unit is used to determine the global rotation motion information of the initial video based on the global rotation motion matrix of each video frame.

[0303] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0304] In one embodiment, the smoothing module 13 includes: a motion smoothing processing unit, a target smoothing data determination unit, and a stabilized video generation unit, wherein:

[0305] The motion smoothing unit is used to perform motion smoothing on global rotational motion information to obtain motion smoothing data for the initial video.

[0306] The target smoothing data determination unit is used to determine the target motion smoothing data of the initial video based on the motion smoothing weights of each video frame in the initial video, the global rotation motion information of the initial video, and the motion smoothing data.

[0307] The image stabilization video generation unit is used to generate image stabilization video based on target motion smoothing data and the initial video.

[0308] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0309] In one embodiment, the image stabilization video generation unit includes: a compensation amount determination subunit and a mapping processing subunit, wherein:

[0310] The compensation amount determination subunit is used to determine the rotation compensation amount of the initial video based on the target motion smoothing data and global rotation motion information;

[0311] The mapping processing subunit is used to map the pixels in each video frame according to the rotation compensation amount of the initial video to generate a stabilized video.

[0312] The video stabilization processing device provided in this application embodiment can be used to execute the technical solutions in the above-described video stabilization processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0313] Specific limitations regarding the video stabilization processing device can be found in the limitations of the video stabilization processing method described above, and will not be repeated here. Each module in the aforementioned video stabilization processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0314] In one embodiment, a video stabilization system is also provided, see further. Figure 1 As shown, the system includes:

[0315] Filming equipment;

[0316] A computer device connected to the shooting device, the computer device being used to perform the steps of any embodiment of the video stabilization processing method described above.

[0317] The video stabilization system provided in this application can be used to execute the technical solutions in the above-described video stabilization method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0318] In one embodiment, a computer device is also provided, the internal structure of which can be as follows: Figure 15 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides processing power. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores initial video and stabilized video. The network interface communicates with external endpoints via a network connection. When executed by the processor, the computer program implements a video stabilization method.

[0319] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0320] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0321] Obtain the 3D rotation information of the initial video;

[0322] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0323] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0324] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0325] Obtain the 3D rotation information of the initial video;

[0326] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0327] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0328] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0329] Obtain the 3D rotation information of the initial video;

[0330] Based on the 3D rotation information, obtain the global rotation motion information of the initial video;

[0331] The global rotational motion information is processed to smooth the motion and generate a stabilized video of the initial video.

[0332] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0333] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0334] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video stabilization method, characterized in that, The method includes: The three-dimensional rotation information of the initial video is obtained, including the target rotation matrix between each two adjacent video frames in the initial video. The target rotation matrix is ​​determined based on the shooting state corresponding to the two adjacent video frames. If the shooting state corresponding to the two adjacent video frames is a non-static shooting state, the essential matrix of the matching point pair set of the two adjacent video frames is obtained, and the target rotation matrix between the two adjacent video frames is obtained based on the essential matrix of the matching point pair set. If the shooting state corresponding to the two adjacent video frames is a static shooting state, the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames. Based on the three-dimensional rotation information, obtain the global rotation motion information of the initial video; The global rotational motion information is processed to smooth the motion and generate a stabilized version of the initial video.

2. The method according to claim 1, characterized in that, The process of obtaining the three-dimensional rotation information of the initial video includes: Obtain the set of matching point pairs between each two adjacent video frames in the initial video; For any two adjacent video frames, the target rotation matrix between the two adjacent video frames is determined based on the set of matching point pairs between the two adjacent video frames. The three-dimensional rotation information of the initial video is determined based on the target rotation matrix between each two adjacent video frames.

3. The method according to claim 2, characterized in that, The step of obtaining the set of matching point pairs between each two adjacent video frames in the initial video includes: For any two adjacent video frames, extract the feature points from the previous video frame to obtain multiple original feature points. Optical flow tracing is performed on the next video frame in two adjacent video frames based on the multiple original feature points to obtain multiple tracking feature points in the next video frame; Based on each tracking feature point, reflective flow tracing is performed on the previous video frame to obtain multiple anti-tracking feature points in the previous video frame; Based on each anti-tracking feature point, determine the set of matching point pairs between two adjacent video frames.

4. The method according to claim 3, characterized in that, The step of determining the set of matching point pairs between two adjacent video frames based on each anti-tracking feature point includes: Obtain the distance between each of the anti-tracking feature points and the corresponding original feature points; Anti-tracking feature points with an interval distance less than a preset distance threshold are identified as valid anti-tracking feature points for optical flow tracing; Based on all valid anti-tracking feature points of optical flow tracing, the set of matching point pairs between two adjacent video frames is determined.

5. The method according to any one of claims 2-4, characterized in that, Determining the target rotation matrix between two adjacent video frames based on the matching point pair set of two adjacent video frames includes: Based on the set of matching point pairs between two adjacent video frames, obtain the optical flow tracing results of the two adjacent video frames; If the optical flow tracing result of two adjacent video frames is invalid, then the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames; If the optical flow tracing results of two adjacent video frames are valid, then the target rotation matrix between the two adjacent video frames is determined according to the shooting state corresponding to the two adjacent video frames.

6. The method according to claim 1, characterized in that, The step of obtaining the target rotation matrix between two adjacent video frames based on the essential matrix of the matching point pair set includes: Singular value decomposition is performed on the essential matrix of the matching point pair set to obtain the initial rotation matrix, three-dimensional translation information and depth information set between two adjacent video frames; Based on the initial rotation matrix, the three-dimensional translation information, and the depth information set, the target rotation matrix between two adjacent video frames is obtained.

7. The method according to claim 6, characterized in that, The step of obtaining the target rotation matrix between two adjacent video frames based on the initial rotation matrix, the three-dimensional translation information, and the depth information set includes: Based on the depth information set, multiple target matching point pairs are selected from all candidate matching point pairs in the matching point pair set; Based on the initial rotation matrix, the three-dimensional translation information, and each target matching point pair, the target rotation matrix between two adjacent video frames is determined.

8. The method according to claim 7, characterized in that, The step of determining the target rotation matrix between two adjacent video frames based on the initial rotation matrix, the three-dimensional translation information, and each target matching point pair includes: By mapping each target matching point pair to the coordinate system of the shooting device using the intrinsic parameters of the shooting device, the mapped coordinates are obtained; A nonlinear least squares model is constructed using the initial rotation matrix, the three-dimensional translation information, and the mapped coordinates. Solve the nonlinear least squares model to obtain the target rotation matrix between two adjacent video frames.

9. The method according to claim 1, characterized in that, The method further includes: Candidate matching point pairs that satisfy the preset static point conditions in the set of matching point pairs are determined as static point pairs; If the total number of all stationary point pairs in the matching point pair set is greater than a preset threshold, then the shooting state corresponding to two adjacent video frames is determined to be the stationary shooting state; otherwise, the shooting state corresponding to two adjacent video frames is determined to be the non-stationary shooting state.

10. The method according to any one of claims 1-4, characterized in that, The step of obtaining the global rotation motion information of the initial video based on the three-dimensional rotation information includes: Based on the three-dimensional rotation information of the initial video, obtain the global rotation motion matrix of each video frame in the initial video; The global rotation motion information of the initial video is determined based on the global rotation motion matrix of each video frame.

11. The method according to any one of claims 1-4, characterized in that, The step of performing motion smoothing processing on the global rotational motion information to generate a stabilized video of the initial video includes: The global rotational motion information is subjected to motion smoothing processing to obtain the motion smoothing data of the initial video; Based on the motion smoothing weights of each video frame in the initial video, the global rotation motion information of the initial video, and the motion smoothing data, the target motion smoothing data of the initial video is determined. The stabilized video is generated based on the target motion smoothing data and the initial video.

12. The method according to claim 11, characterized in that, The step of generating the stabilized video based on the target motion smoothing data and the initial video includes: Based on the target motion smoothing data and the global rotation motion information, determine the rotation compensation amount of the initial video; The pixels in each video frame are mapped according to the rotation compensation amount of the initial video to generate the stabilized video.

13. A video stabilization processing device, characterized in that, The device includes: A rotation information acquisition module is used to acquire three-dimensional rotation information of an initial video. The three-dimensional rotation information includes a target rotation matrix between each pair of adjacent video frames in the initial video. The target rotation matrix is ​​determined based on the shooting state corresponding to the two adjacent video frames. If the shooting state corresponding to the two adjacent video frames is a non-static shooting state, the essential matrix of the matching point pair set of the two adjacent video frames is acquired, and the target rotation matrix between the two adjacent video frames is acquired based on the essential matrix of the matching point pair set. If the shooting state corresponding to the two adjacent video frames is a static shooting state, the identity matrix is ​​determined as the target rotation matrix between the two adjacent video frames. A global information acquisition module is used to acquire global rotation motion information of the initial video based on the three-dimensional rotation information; The smoothing module is used to perform motion smoothing on the global rotational motion information to generate a stabilized video of the initial video.

14. A video stabilization system, characterized in that, The system includes: Filming equipment; A computer device connected to the shooting device, the computer device being used to perform the steps of the method according to any one of claims 1-12.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-12.

Images