Panoramic video anti-shake method and device and storage medium

Abstract

The application relates to the technical field of image processing, and provides a panoramic video anti-shake method and device and a storage medium. The method comprises the following steps: a lookup table is constructed; based on the lookup table, current frames and adjacent frames are resampled to obtain corresponding groups of distortion-free planar images, feature point detection and optical flow tracking are performed on the groups of distortion-free planar images, and a planar two-dimensional feature point pair is obtained; the planar two-dimensional feature point pair is back-projected to a three-dimensional space to form a three-dimensional unit ray direction vector pair; based on the three-dimensional unit ray direction vector pair, a pose rotation matrix between adjacent frames is estimated by using a random sample consensus algorithm combined with singular value decomposition under three-dimensional rotation constraints; the pose rotation matrix is converted into a unit quaternion to update an original cumulative pose trajectory, and a compensation rotation matrix is applied to perform reverse resampling on panoramic video frames to generate stable frames after anti-shaking.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a panoramic video stabilization method, apparatus, and storage medium. Background Technology

[0002] With the rapid development of virtual reality, panoramic live streaming, and security monitoring, panoramic video has gained widespread attention for its ability to provide a 360-degree immersive visual experience. Panoramic video is typically stored and transmitted using an equidistant projection format, which unfolds a spherical scene into a rectangular planar image, offering advantages such as well-organized data structure and good encoding compatibility. However, in actual shooting, camera shake often results in shaky and blurry images in panoramic videos, severely impacting the viewing experience. Therefore, electronic image stabilization technology for panoramic video has become a research hotspot.

[0003] Existing panoramic video stabilization solutions generally fall into two categories: hardware-assisted stabilization based on inertial measurement units (IMUs) and software-based stabilization based on pure vision. Hardware-assisted stabilization relies on sensors such as gyroscopes, which are costly and susceptible to electromagnetic interference, limiting its applicability. Software-based stabilization solutions typically include feature point extraction and matching, motion model estimation, and inter-frame compensation. However, most of these solutions are designed for ordinary planar videos and have significant shortcomings when directly applied to panoramic videos. Specifically, panoramic video frames in equidistant projection formats suffer from severe geometric distortion in the polar regions, making it difficult to stably match feature points extracted in these regions across different frames. Furthermore, the motion in panoramic videos is essentially a three-dimensional rotational motion on a sphere, which traditional affine transformations or homography matrix models for planar videos cannot accurately describe, resulting in significant motion estimation errors and poor stabilization performance. Summary of the Invention

[0004] This application provides a panoramic video stabilization method, device, and storage medium that eliminates the effects of polar distortion and accurately estimates spherical rotational motion, thereby achieving high-quality panoramic video stabilization.

[0005] On the one hand, this application provides a panoramic video stabilization method, the method comprising: Construct a lookup table to map panoramic video frames in isometric projection format to each distortion-free plane of a cubemap; Based on the lookup table, the current frame and adjacent frames are resampled to obtain multiple sets of distortion-free planar images. Feature point detection and optical flow tracing are performed on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs. The planar two-dimensional feature point pairs are back-projected into three-dimensional space to form three-dimensional unit ray direction vector pairs. Based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, the attitude rotation matrix between adjacent frames is estimated by combining the random sampling consensus algorithm with singular value decomposition. The attitude rotation matrix is ​​converted into a unit quaternion to update the original cumulative attitude trajectory, and spherical linear interpolation filtering is performed on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, a compensation rotation matrix is ​​calculated based on the smooth attitude trajectory and the original cumulative attitude trajectory, and the compensation rotation matrix is ​​applied to reverse resample the panoramic video frame to generate a stabilized frame after image stabilization.

[0006] On the other hand, this application provides a panoramic video stabilization device, the device comprising: The module is used to build a lookup table for mapping panoramic video frames in isometric projection format to the distortion-free planes of a cubemap; The acquisition module is used to resample the current frame and adjacent frames based on the lookup table to obtain multiple sets of distortion-free planar images, and to perform feature point detection and optical flow tracing on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs. The estimation module is used to back-project the two-dimensional feature point pairs on the plane to three-dimensional space to form three-dimensional unit ray direction vector pairs, and based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, estimate the attitude rotation matrix between adjacent frames through a random sampling consensus algorithm combined with singular value decomposition. The generation module is used to convert the attitude rotation matrix into a unit quaternion to update the original cumulative attitude trajectory, and to perform spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, it calculates a compensation rotation matrix based on the smooth attitude trajectory and the original cumulative attitude trajectory, and applies the compensation rotation matrix to perform reverse resampling on the panoramic video frame to generate a stabilized frame after image stabilization.

[0007] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the panoramic video stabilization method described above.

[0008] Fourthly, this application provides a storage medium storing a computer program that, when executed by a processor, implements the panoramic video stabilization method described above.

[0009] As can be seen from the technical solutions provided in this application, on the one hand, by constructing a lookup table to map panoramic video frames in equidistant projection format to each distortion-free plane of the cube map, the influence of polar distortion on subsequent processing is eliminated, enabling feature point detection and optical flow tracing to be performed on an approximately planar image, significantly improving the stability and accuracy of feature matching; on the other hand, by back-projecting planar two-dimensional feature point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, and combining random sampling consensus algorithm and singular value decomposition under three-dimensional rotation constraints to perform attitude estimation, it is possible to accurately fit the inherent spherical rotation motion model of panoramic video, overcoming the model mismatch problem of traditional planar motion models when describing spherical motion, thereby obtaining a more accurate inter-frame attitude rotation matrix; thirdly, by converting the attitude rotation matrix into a unit quaternion and updating the original cumulative attitude trajectory, and combining spherical linear interpolation filtering to obtain a smooth attitude trajectory, and then calculating the compensation rotation matrix and performing reverse resampling, high-quality stabilization processing of panoramic video frames is achieved, effectively suppressing the jitter phenomenon of video images and improving the visual experience of panoramic video. In summary, the technical solution of this application eliminates the influence of polar distortion and accurately estimates the spherical rotation motion, thereby achieving high-quality panoramic video stabilization. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of the panoramic video stabilization method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the panoramic video stabilization device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the device provided in the embodiments of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0014] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0015] Existing panoramic video stabilization solutions generally fall into two categories: hardware-assisted stabilization based on inertial measurement units (IMUs) and software-based stabilization based on pure vision. Hardware-assisted stabilization relies on sensors such as gyroscopes, which are costly and susceptible to electromagnetic interference, limiting its applicability. Software-based stabilization solutions typically include feature point extraction and matching, motion model estimation, and inter-frame compensation. However, most of these solutions are designed for ordinary planar videos and have significant shortcomings when directly applied to panoramic videos. Specifically, panoramic video frames in equidistant projection formats suffer from severe geometric distortion in the polar regions, making it difficult to stably match feature points extracted in these regions across different frames. Furthermore, the motion in panoramic videos is essentially a three-dimensional rotational motion on a sphere, which traditional affine transformations or homography matrix models for planar videos cannot accurately describe, resulting in significant motion estimation errors and poor stabilization performance.

[0016] To address the aforementioned problems in the prior art, this application proposes a panoramic video stabilization method, the flowchart of which is attached. Figure 1 As shown, it mainly includes steps S1 to S4, which are detailed below: Step S1: Construct a lookup table for mapping panoramic video frames in isometric projection format to the distortion-free planes of a cubemap.

[0017] In this embodiment, the isometric projection format (ERP) panoramic video frame is a rectangular image formed by unfolding a spherical panoramic image according to equally spaced latitude and longitude, with its width corresponding to 360 degrees of longitude and its height corresponding to 180 degrees of latitude. Under this projection method, regions near the poles (North and South Poles) are severely stretched horizontally, resulting in significant geometric distortion. This causes traditional planar feature detection and optical flow tracing algorithms to fail in these regions, leading to numerous false matches in feature matching. Therefore, a preprocessing method is needed to convert the ERP image into multiple distortion-free planes for subsequent processing on the approximate planar image. Existing solutions either process directly on the ERP image, but polar distortion cannot be avoided; or they use spherical or cylindrical projection, but spherical projection is still non-linear, and cylindrical projection still has distortion in the vertical direction. Furthermore, neither can uniformly divide the sphere into six approximate planar sub-regions like cube mapping, which is detrimental to the subsequent use of mature planar image processing algorithms. In short, panoramic video frames in the isometric projection format suffer from severe geometric distortion in the polar regions. Existing technologies make it difficult for feature points extracted in the polar regions to be stably matched between different frames, which in turn affects the accuracy of subsequent motion estimation.

[0018] To address the aforementioned issues, this application constructs a lookup table for mapping isometric projection format panoramic video frames to each distortion-free plane of a cube map. Here, the lookup table (LUT) is a pre-computed coordinate mapping matrix that records the correspondence between each pixel in the ERP image and each pixel on the six faces of the cube map.

[0019] Specifically, as an embodiment of this application, the construction of a lookup table for mapping isometric projection format panoramic video frames to each distortion-free plane of a cube map can be achieved through the following steps S1-1 to S1-3: Step S1-1: For a panoramic video frame in equidistant projection format with a resolution of , calculate its pixel coordinates ( x , y ) and latitude and longitude angle The mapping relationship, where, , .

[0020] here, , These are the column index and row index of the pixel, respectively. Longitude, with a value range of 100. , For latitude, the range of values ​​is: This mapping relationship is the foundation for the definition of the ERP format, namely longitude. with pixel column coordinates x The relationship is linear, latitude with pixel row coordinatesy The relationship is linear.

[0021] Step S1-2: For each face of the cube map, calculate the normalized planar coordinates of that face. Calculate the corresponding three-dimensional direction vector The three-dimensional direction vectors are normalized and mapped back to latitude and longitude angles to determine the corresponding pixel coordinates on the panoramic video frame in the equidistant projection format.

[0022] Taking the +X face of a cube map as an example, its normalized planar coordinates The mapping relationship with the three-dimensional direction vector is as follows: Then, unitize. Normalized direction vector d Pointing to a point on the sphere, its corresponding latitude and longitude can be obtained from... d The components were calculated to obtain: , Based on the aforementioned... , The corresponding pixel coordinates on the ERP image can be obtained by inverse mapping with the ERP pixel coordinates (i.e., the inverse functions of formulas (1) and (2)). The same principle applies to the other five faces (-X, +Y, -Y, +Z, -Z); simply adjust the formula for calculating the direction vector according to the orientation of each face. For example, for the -Y face (facing down), the direction vector is... And so on.

[0023] Step S1-3: Save the corresponding pixel coordinates in advance as a lookup table so that the distortion-free planar image can be directly extracted by bilinear interpolation during subsequent resampling.

[0024] Since the lookup table is pre-calculated, when processing video frames, the sampling coordinates on the ERP image can be obtained directly by looking up the table based on the pixel coordinates of the target cube map face. Then, bilinear interpolation can be performed to quickly obtain a distortion-free planar image without repeating trigonometric function calculations, which greatly reduces the computational overhead.

[0025] By constructing a lookup table to map isometric projection format panoramic video frames to the distortion-free planes of a cube map, the technical solution exemplified in steps S1-1 to S1-3 efficiently converts the ERP image into six distortion-free planes of the cube map, effectively eliminating the impact of polar distortion on subsequent processing. Each cube face is an approximate plane, enabling feature point detection and optical flow tracing to be performed with minimal geometric distortion, significantly improving the stability and reliability of feature matching. Simultaneously, the pre-computation method of the lookup table ensures the real-time performance of the resampling process while maintaining algorithm efficiency.

[0026] Step S2: Based on the lookup table used to map panoramic video frames in equidistant projection format to each distortion-free plane of the cube map, the current frame and adjacent frames are resampled to obtain multiple sets of corresponding distortion-free plane images. Feature point detection and optical flow tracing are performed on each set of distortion-free plane images to obtain two-dimensional feature point pairs on the plane.

[0027] After obtaining the distortion-free planar images, feature correspondences need to be established between adjacent frames to estimate inter-frame motion. Since panoramic videos may contain dynamic objects, occlusions, or lighting changes, feature point detection and tracking need to be robust, and specific implementation details are required to ensure that those skilled in the art can implement them. Therefore, this application uses a lookup table to map isometric projection format panoramic video frames to distortion-free planes of a cubemap, resampling the current frame and adjacent frames to obtain multiple sets of distortion-free planar images. Specifically, for the current frame (frame t) and its adjacent frames (e.g., frame t-1 or t+1), the lookup table is used to resample the isometric projection format panoramic video frame to six distortion-free planar images of a cubemap, each set containing six planes (+X, -X, +Y, -Y, +Z, -Z). Resampling can be performed as follows: for each pixel position on each cube face, the corresponding ERP image coordinates are read from a lookup table, and then bilinear interpolation is used to extract the pixel value from the ERP image and fill it into the corresponding position on the cube face. In this way, six distortion-free planar images are obtained for each frame. Then, feature point detection and optical flow tracing are performed on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs. Specifically, step S2 is implemented as shown in steps S2-1 and S2-2: Step S2-1: Extract feature points from each group of distortion-free planar images using the minimum eigenvalue algorithm.

[0028] The implementation steps of the minimum eigenvalue algorithm (MinEigen, also known as Shi-Tomasi corner detection) are as follows: First, for each pixel in the image, calculate the gradient matrix within its local window (e.g., a 3×3 or 5×5 window). M as follows:

[0029] in, and The images are respectively in x direction and y The gradient in the direction (which can be calculated using the Sobel operator). Then, the gradient matrix is ​​calculated. M Two eigenvalues and The minimum value among these values ​​is taken as the corner response value of the pixel. Finally, non-maximum suppression is applied to the entire image, retaining pixels with response values ​​greater than a preset threshold and local maxima as feature points. This algorithm can extract corner points with rich textures that are easy to track, and it has a certain robustness to changes in illumination. In this embodiment, feature point extraction is performed independently on each cube face to ensure that each plane obtains sufficient feature points.

[0030] Step S2-2: Use the pyramid optical flow method to track feature points on the corresponding distortion-free planar images of adjacent frames to obtain planar two-dimensional feature point pairs.

[0031] The specific implementation of the Pyramid Optical Flow (KLT) method is as follows: First, an image pyramid (e.g., 3 or 4 layers) is constructed for the current frame and adjacent frames, with the bottom layer being the original image and the top layer being the downsampled image. Then, starting from the top layer of the pyramid, at the feature point location in the top layer image of the current frame, the best match is searched for near the corresponding location in the top layer image of the adjacent frame. The initial displacement is obtained by minimizing the sum of squares of the brightness difference. Next, this displacement is passed to the next layer, and a fine-grained search is performed in the next layer. This process is repeated until the bottom layer, finally obtaining the sub-pixel precision optical flow displacement. In this way, even if there is a large pixel displacement between adjacent frames (e.g., caused by rapid rotation), it is possible to successfully track the image. For a feature point on a certain plane in the current frame, the above tracking is performed on the same plane in adjacent frames to obtain the corresponding two-dimensional coordinates, thus forming a pair of planar two-dimensional feature points.

[0032] To improve the quality of feature points, this application also filters the extracted feature points. Specifically, step S2 may further include: filtering the feature points extracted from each group of distortion-free planar images, removing feature points located within a preset pixel range at the edge of the distortion-free planar image, and retaining reliable feature points located in the central region for optical flow tracing. In specific implementation, an edge clipping width (e.g., 10 pixels) can be set for the feature point coordinates. ,like or ,or or (Assuming the plane dimensions are) If a feature point fails to reach its designated edge, it will be removed. This is because the edge regions of each plane in the cube map may have slight stitching errors or residual distortions, and feature points at the edges are prone to moving out of the plane in the next frame, leading to tracking failure. By setting edge clipping regions, invalid tracking can be effectively reduced, improving overall robustness.

[0033] Furthermore, since there are boundaries between the planes of the cube map, optical flow tracing within a single plane will fail when a feature point moves from one plane to an adjacent plane. To address this, this application further introduces a cross-plane feature point tracking and fusion mechanism, as detailed in steps S10a to S10c, as follows: Step S10a: For the overlapping region between adjacent distortion-free planar images, the feature points are simultaneously projected onto the adjacent planes, and optical flow tracing is performed on the corresponding planes respectively.

[0034] In practice, the first step is to determine the overlapping regions between the planes of the cube map. For example, the +X and +Y planes share an edge, and a strip-shaped overlapping region exists near their boundary (usually determined by the cube map's resolution and sampling method). When a feature point is located within this overlapping region, given its coordinates on the +X plane in the current frame, it can be projected onto the +Y plane using coordinate transformation (utilizing the cube map's geometric relationships) to obtain its corresponding coordinates on the +Y plane. Then, the feature point is traced to the next frame using pyramid optical flow on both the +X and +Y planes, yielding two candidate tracking results.

[0035] Step S10b: Compare the tracking results of the same physical feature point on adjacent planes. If the difference in tracking displacement is less than a preset threshold, the average value is taken as the final tracking coordinate.

[0036] Here, the difference in tracking displacement refers to the magnitude of the difference between the displacement vectors obtained from tracking on the two planes. If the difference is very small (e.g., less than 1 pixel), it means that the tracking results on the two planes are consistent. Taking their average value can improve accuracy and eliminate any possible minor systematic biases.

[0037] Step S10c: If the difference in tracked displacement is greater than or equal to a preset threshold, discard the feature point to avoid errors introduced by boundary jumps. This situation usually occurs when there is occlusion or discontinuity in motion near the boundary. In this case, the tracking results on the two planes are inconsistent, and forcibly using one of the results may introduce a large error. Discarding the point can prevent incorrect matching from contaminating subsequent pose estimation.

[0038] The cross-plane feature point tracking and fusion mechanism exemplified in steps S10a to S10c above projects feature points onto adjacent planes simultaneously for parallel tracking within the overlapping area of ​​adjacent planes. It then determines whether to fuse or discard the feature points by comparing the differences in the tracking results. This cross-plane collaborative tracking strategy fully utilizes the spatial continuity of cube maps, thereby effectively solving the tracking breakage problem at the plane boundary and increasing the number and stability of trackable feature points. Especially in scenarios where rapid camera rotation causes a large number of feature points to cross the boundary, it significantly improves the reliability and continuity of motion estimation.

[0039] In another embodiment of this application, the pyramid optical flow method described above further includes the following cross-plane feature point fusion scheme based on optical flow confidence weighting: For overlapping regions between adjacent distortion-free planar images, feature points are simultaneously projected onto adjacent planes, and pyramid optical flow tracing is performed on each plane to obtain multiple candidate tracking displacements; the optical flow confidence of each candidate tracking displacement is calculated, wherein the optical flow confidence is jointly determined based on the minimum eigenvalue of the image gradient covariance matrix of the tracking window in the plane and the tracking residual; each candidate tracking displacement is weighted and averaged according to its optical flow confidence to obtain the final fused tracking coordinates; if the confidence of all candidate tracking displacements is lower than a preset confidence lower limit, the feature point is discarded. This scheme uses weighted average fusion instead of simple arithmetic average or binary selection, making the tracking results with high confidence contribute more and the results with low confidence contribute less, resulting in a smoother and more accurate fusion result. Meanwhile, the confidence lower bound mechanism retains the function of discarding the technical solutions in the examples of steps S10a to S10c, but the decision basis is more scientific. It combines quantitative evaluation of optical flow reliability with multi-view fusion, which significantly improves the robustness and accuracy of feature point tracking near the boundary.

[0040] Step S3: Back-project the two-dimensional feature point pairs on the plane to the three-dimensional space to form three-dimensional unit ray direction vector pairs. Based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, estimate the attitude rotation matrix between adjacent frames by combining the random sampling consensus algorithm with singular value decomposition.

[0041] After obtaining the two-dimensional feature point pairs in the plane, it is necessary to estimate the motion between adjacent frames. Since the inter-frame motion of panoramic video is essentially the rotational motion of the camera in three-dimensional space, traditional two-dimensional planar models (such as affine transformations or homography matrices) cannot accurately describe this spherical motion, leading to systematic biases. Furthermore, optical flow tracing inevitably produces mismatches (e.g., due to dynamic objects, repetitive textures, or occlusion), requiring a robust estimation algorithm to remove outliers. Therefore, an estimation method that can accurately fit the spherical rotational motion and is robust to mismatches is needed. Existing solutions include: some directly estimating the affine transformation on the two-dimensional plane, but ignoring the nonlinearity of spherical projection; some attempting to map two-dimensional point pairs to a three-dimensional sphere and then directly solve the least squares problem, but lacking robust handling of mismatches, a small number of mismatches can severely pollute the estimation results; and others using standard random sample consensus (RANSAC) with a fixed threshold, but the fixed threshold is difficult to adapt to changes in noise levels under different scenes.

[0042] To address the aforementioned issues, this application backprojects two-dimensional feature point pairs onto three-dimensional space, forming three-dimensional unit ray direction vector pairs. Based on these three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, the attitude rotation matrix between adjacent frames is estimated using a random sample consensus algorithm combined with singular value decomposition. This scheme, by backprojecting two-dimensional feature point pairs onto three-dimensional space to form three-dimensional unit ray direction vector pairs and establishing a rigorous mathematical model based on three-dimensional rotation constraints, reduces the motion estimation problem to the essence of spherical rotation. Combining the robustness of the random sample consensus algorithm in eliminating mismatches with the accurate solution of the optimal rotation matrix by singular value decomposition, it can robustly estimate the attitude rotation matrix between adjacent frames, achieving high-precision motion estimation results even with a certain proportion of mismatches or noise.

[0043] Specifically, in order to robustly estimate the three-dimensional rotational motion between panoramic video frames from two-dimensional feature point pairs in the presence of mismatches and noise, step S3 above can be implemented by steps S3-1 and S3-2, as detailed below: Step S3-1: Backproject each pair of points in the two-dimensional feature point pair in the plane to the three-dimensional space to obtain the corresponding two three-dimensional unit ray direction vectors. and And establish ideal rotational constraints ,in, R Let be the attitude rotation matrix to be determined.

[0044] The backprojection process utilizes the mapping relationship between the cube map and the 3D orientation vector established in step S1. Specifically, for feature points on a distortion-free plane in the current frame... According to the mapping relationship corresponding to the plane (e.g., The corresponding 3D direction vector is calculated and normalized. Similarly, for the corresponding points tracked in adjacent frames... Calculate And normalize it. This gives us a pair of three-dimensional unit ray direction vectors. Under ideal pure rotational motion, these two rays should satisfy... ,in, R It is An orthogonal rotation matrix that satisfies and This constraint forms the basis for subsequent estimations.

[0045] Step S3-2: In each iteration of the random sampling consensus algorithm, at least two pairs of three-dimensional unit ray direction vectors are randomly selected. The number of interior points is evaluated based on the residual angle between the direction vectors. The attitude rotation matrix that minimizes the error of the interior point set is solved by singular value decomposition.

[0046] The specific implementation process is as follows: First, set the maximum number of RANSAC iterations (e.g., 500) and the interior point determination threshold (initially set to a fixed value, e.g., 0.01 radians). In each iteration, randomly select two (or more) pairs of 3D unit ray direction vectors. Note that two pairs are the minimum number required to solve the rotation matrix, because each pair provides a 3D vector constraint, and the two pairs provide a total of 6 equations, which are sufficient to uniquely determine the 3-DOF rotation matrix. R Then, using the selected two pairs of direction vectors, the rotation matrix is ​​solved via singular value decomposition (SVD). Specifically, the matrix is ​​constructed. (in, n (For selected point logarithms), then on A Perform SVD decomposition Then the rotation matrix To ensure ,like Then invert the last column and multiply them.

[0047] After obtaining the candidate rotation matrices, calculate the angular residuals for all direction vector pairs: That is, after rotation and The angle between the points is calculated. Pairs of points whose angle residuals are less than a preset threshold are marked as interior points, and the number of interior points is counted. The set of interior points and the corresponding rotation matrix for this iteration are recorded.

[0048] After all iterations are completed, the rotation matrix corresponding to the iteration with the largest number of inliers is selected as the initial estimate. Then, using all direction vector pairs in this inlier set, a globally optimal rotation matrix is ​​solved again using SVD, which is the final estimated attitude rotation matrix. This "RANSAC+SVD" combined strategy not only eliminates mismatches but also improves accuracy by utilizing all inlier information.

[0049] The technical solutions described in steps S3-1 and S3-2 back-project the planar two-dimensional feature point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, establishing a strict spherical rotation constraint model instead of using the approximation of the traditional planar model. At the same time, the robustness of RANSAC and the accuracy of SVD are organically combined, which not only eliminates mismatches but also obtains the optimal solution in the least squares sense, thereby significantly improving the robustness and accuracy of motion estimation. Even in scenarios with low feature point quality or many mismatches, a reliable attitude rotation matrix can still be obtained, providing high-quality input for subsequent smooth compensation.

[0050] In another embodiment of this application, before backprojecting the planar two-dimensional feature point pairs to three-dimensional space, the following technical solution can be adopted: Pre-screening the planar two-dimensional feature point pairs based on epipolar geometric constraints to remove abnormal point pairs that do not conform to the epipolar geometric relationship between panoramic video frames; back-projecting the pre-screened point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, and using a weighted iterative reprojection least squares method instead of singular value decomposition (SVD). The weights are jointly determined based on the angular residuals of each direction vector pair and the feature point response intensity. The weights are updated and the pose rotation matrix is ​​resolved in each iteration until convergence. This solution uses a weighted iterative reprojection least squares method instead of standard SVD. The weights not only consider the residuals but also incorporate the feature point response intensity, which reflects the reliability of the features, resulting in a greater contribution from high-quality feature points and more robust estimation results, especially in sparse texture regions.

[0051] Considering that traditional RANSAC uses a fixed threshold to determine inliers, it cannot adapt to changes in noise levels under different scenarios, resulting in the loss of valid inliers in low-noise scenarios or the introduction of too many false matches in noisy scenarios. Therefore, as an embodiment of this application, the evaluation of the number of inliers based on the residual angle between direction vectors in the above embodiment can be implemented by steps S9a and S9b, as detailed below: Step S9a: In each iteration, dynamically calculate the interior point determination threshold based on the median or standard deviation of the angular residuals of the three-dimensional unit ray direction vector pairs of the current interior point set.

[0052] In practice, during each iteration, an initial threshold (e.g., 0.02 radians) is used to initially filter inlier points. Then, the median or standard deviation of the angular residuals of the current inlier set is calculated, and a new threshold is set to twice the median or three times the standard deviation. This allows the threshold to adaptively adjust based on the noise level of the current data, avoiding the problem of a fixed threshold being too lenient in low-noise scenarios and too strict in high-noise scenarios.

[0053] Step S9b: Mark the direction vector pairs whose included angle residual is less than the in-point determination threshold as in-points, and count the number of in-points.

[0054] By using dynamic thresholding, interior points can be identified more accurately in each iteration, thus obtaining better candidate rotation matrices in subsequent iterations.

[0055] Furthermore, to prevent accidental outliers in single-frame estimation from affecting the overall trajectory, this application performs temporal consistency checks on the attitude rotation matrices of consecutive frames. Specifically, it checks the temporal consistency of the attitude rotation matrices estimated from consecutive frames by calculating the rotation angle between adjacent frames. If the rotation angle exceeds a preset threshold, the attitude rotation matrix of the current frame is replaced with the attitude rotation matrix of the previous frame or interpolated for correction. In practice, the rotation matrix of the current frame... Rotation matrix of the previous frame Relative rotation between Converting to axis-angle representation yields the rotation angle. .like If the value exceeds a preset threshold (e.g., 30 degrees), the estimation result of the current frame is considered unreliable, possibly due to feature tracking failure or sudden large fluctuations. In this case, the rotation matrix of the previous frame can be used to directly replace the result of the current frame. Alternatively, the rotation matrix of the previous few frames can be interpolated (e.g., spherical linear interpolation) to obtain a correction value. This mechanism effectively prevents single-frame errors from propagating to subsequent pose accumulation and trajectory smoothing. By back-projecting planar two-dimensional feature point pairs to three-dimensional space and establishing three-dimensional rotation constraints, this application accurately fits the spherical rotation motion nature of panoramic video, overcoming the mismatch problem of traditional two-dimensional models. Combining RANSAC's robust elimination of mismatches and SVD's accurate solution capability, a high-precision pose rotation matrix can be obtained even in dynamic scenes or when feature point quality is low. The adaptive interior point decision threshold further improves RANSAC's adaptability to different noise environments, while temporal consistency verification provides additional security for the system. These measures together ensure that the pose rotation matrix output in step S3 has high reliability and high accuracy, laying a solid foundation for subsequent trajectory smoothing and compensatory resampling.

[0056] The technical solutions described in steps S9a and S9b above dynamically calculate the inlier determination threshold based on the median or standard deviation of the included angle residuals of the current inlier set, allowing the threshold to adaptively adjust with the data distribution. This method eliminates the need for manual threshold pre-setting and exhibits better generalization ability for scenarios with varying noise levels. While maintaining RANSAC robustness, it improves the accuracy of inlier determination, thereby enhancing the estimation accuracy of the attitude rotation matrix, especially in real-world shooting scenarios with significant noise level variations.

[0057] In another embodiment of this application, the above-mentioned random sampling consensus algorithm adopts the following sampling strategy guided by the spatial distribution uniformity of feature points: In each iteration, samples are preferentially extracted from the subset of feature points with the most uniform spatial distribution in the distortion-free planar image. The spatial distribution uniformity is measured by calculating the coefficient of variation of the distance from each feature point to its nearest neighbor feature point. The smaller the coefficient of variation, the more uniform the distribution. At the same time, the sampling bias of the next iteration is dynamically adjusted according to the spatial coverage of the inlier set in the current iteration, so that the sampling gradually tends to cover the image area that has not been fully sampled. Finally, the pose rotation matrix is ​​solved by jointly solving the entire inlier set.

[0058] Step S4: Convert the attitude rotation matrix into a unit quaternion to update the original cumulative attitude trajectory, and perform spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, calculate the compensation rotation matrix based on the smooth attitude trajectory and the original cumulative attitude trajectory, and apply the compensation rotation matrix to reverse resample the panoramic video frame to generate a stabilized frame after image stabilization.

[0059] Step S3 estimates the relative rotation matrix between adjacent frames, but image stabilization requires the absolute pose of each frame relative to a reference coordinate system (usually the first frame). Directly accumulating relative rotations will accumulate errors, and the original pose trajectory contains jitter components, requiring smoothing. Furthermore, how to convert the smoothed pose into actual image compensation is a key problem that needs to be solved. Existing solutions either directly apply linear filtering to the Euler angles of the rotation matrix, but Euler angles suffer from gimbal lock and the interpolation path is unnatural; or they use Kalman filtering, but this requires complex parameter tuning and has poor adaptability to nonlinear motion. Regarding resampling, forward mapping (from input frame to output frame) is prone to holes and overlaps, requiring additional post-processing. Therefore, this application converts the pose rotation matrix into a unit quaternion to update the original accumulated pose trajectory, performs spherical linear interpolation filtering on the original accumulated pose trajectory to obtain a smooth pose trajectory, calculates the compensation rotation matrix based on the smoothed pose trajectory and the original accumulated pose trajectory, and applies the compensation rotation matrix to perform reverse resampling of the panoramic video frame to generate a stabilized frame. The specific implementation is as follows: First, the inter-frame pose rotation matrix obtained in step S3 (representing the relative rotation from frame i to frame j) is converted into a unit quaternion representation. Quaternions are a compact, singular-free rotation representation suitable for accumulation and interpolation. The conversion method is a standard algorithm: given the rotation matrix, calculate its trace, then select different formulas based on its size to calculate the four components of the quaternion, ensuring normalization.

[0060] Then, attitude accumulation and spherical linear interpolation filtering are performed, where the spherical linear interpolation filtering uses a smoothing factor. The smooth attitude trajectory is obtained through the following steps S4a and S4b: Step S4a: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] k The pose rotation matrix of a frame is converted into a unit quaternion. and in accordance with Update the original cumulative attitude trajectory.

[0061] in, express k The cumulative pose of a frame relative to the world coordinate system (usually based on the first frame). This represents quaternion multiplication. Initially, it is set... (Unit quaternion, representing zero rotation). The original cumulative pose for each frame can be obtained through recursion. The trajectory includes both intentional camera movement and unintentional camera shake.

[0062] Step S4b: Calculate the quaternions of adjacent time points in the original accumulated attitude trajectory. and Perform spherical linear interpolation, i.e. This yields a smooth attitude trajectory.

[0063] Spherical Linear Interpolation (SLERP) is an interpolation performed along geodesics on a quaternion manifold, ensuring the geometric naturalness of the interpolation path. Smoothing factor. Controlling smoothing intensity: The smaller the value, the stronger the smoothing effect, meaning the trajectory is closer to the historical smoothing value, but the greater the delay. The larger the value, the weaker the smoothing effect, meaning the trajectory is closer to the original value, but the smaller the delay. In practical applications, A fixed value between 0.1 and 0.3 is typically used. Through frame-by-frame SLERP filtering, high-frequency jitter is effectively suppressed, while intentional low-frequency motion is preserved.

[0064] The smooth attitude trajectory acquisition methods in steps S4a and S4b above use unit quaternions to represent rotations, avoiding the singularity of Euler angles. Spherical linear interpolation (SLEP) is used to smooth the quaternion trajectory, ensuring the interpolation path follows geodesics and has clear physical meaning. Compared to traditional linear interpolation or Kalman filtering, SLEP better conforms to the Riemannian geometry of rotation space. The generated smooth attitude trajectory maintains a natural motion transition in rotation space, without abrupt changes or jitter, resulting in a smoother and more natural compensated video image and reducing visual artifacts introduced by improper smoothing.

[0065] In another embodiment of this application, the spherical linear interpolation filtering of the above embodiment employs an adaptive smoothing factor. The adaptive smoothing factor is dynamically adjusted according to the motion amplitude of the current frame as follows: when the rotation angle between adjacent frames is greater than a first threshold, the smoothing factor is reduced to retain more real motion; when the rotation angle is less than a second threshold, the smoothing factor is increased to enhance the smoothing effect; and a sliding window statistical analysis is performed on the deviation between the original cumulative attitude trajectory and the smoothed attitude trajectory. If the variance of the deviation exceeds a preset fluctuation limit, a local reset is triggered, and the smoothed attitude trajectory is rolled back to the most recent anchor frame with a confidence level higher than the confidence threshold, and spherical linear interpolation filtering is re-performed from that anchor frame. This scheme introduces an adaptive smoothing factor to dynamically adjust the smoothing intensity according to the motion amplitude, avoiding the defects of a fixed factor causing delay due to over-smoothing during rapid motion and residual jitter due to insufficient smoothing during stillness. On the other hand, an anomaly detection and local reset mechanism based on sliding window variance is added. When the smoothed trajectory deviates too much from the original trajectory (the excessive deviation may be due to long-term accumulated errors or sudden large jitter), it automatically rolls back to a reliable anchor frame for re-smoothing, thereby preventing trajectory drift.

[0066] In another embodiment of this application, the reverse resampling of the above embodiments can also adopt the following adaptive sampling strategy based on mesh deformation and content awareness: The compensation rotation matrix is ​​applied to the global direction vector mesh to obtain preliminary sampling coordinates; the preliminary sampling coordinates are locally fine-tuned, the goal of which is to minimize the structural similarity loss between adjacent pixels in the output frame while maintaining the overall constraint of the compensation rotation matrix. This local fine-tuning is achieved by solving a constrained energy optimization equation, where the energy term includes a fidelity term, a smoothness term, and a boundary consistency term; bilinear interpolation is performed based on the locally fine-tuned sampling coordinates to generate a stable frame after image stabilization. This scheme introduces content-aware mesh deformation optimization, considering the structural information of the image (e.g., edges, texture), so that the stable frame after resampling retains image details and visual quality to the maximum extent while performing geometric correction, avoiding jagged edges or blurring that may be caused by simple geometric mapping. It is a more refined and intelligent sampling strategy.

[0067] To efficiently and completely resample isometric projection format panoramic video frames, avoiding holes or overlaps that may occur during forward resampling, and ensuring complete viewpoint coverage of the output frames, this application adopts a strategy based on a global direction vector grid in the reverse resampling stage. That is, applying a compensation rotation matrix to reverse resample isometric projection format panoramic video frames can be achieved through steps S4c to S4e, as described in detail below: Step S4c: Pre-construct a global isometric projection format orientation vector grid covering the entire panoramic view. This grid is a two-dimensional grid with the same resolution as the output frame, and each grid point corresponds to a three-dimensional orientation vector. The orientation vector is determined by the point's position in the ERP image. and The inverse mapping is calculated. This mesh only needs to be constructed once during the entire video processing.

[0068] Step S4d: Apply the compensation rotation matrix to the global isometric projection format orientation vector grid to obtain the rotated orientation vector, and then calculate its sampling coordinates on the isometric projection format image based on the rotated orientation vector.

[0069] The compensation rotation matrix is ​​calculated as follows: First, based on the smooth posture... and original posture Calculate the compensated quaternion Then convert it into a rotation matrix. .Will Acts on each direction vector in the global direction vector grid To obtain the rotated direction vector Then, according to The coordinates are obtained by inverse operations of formulas (5) and (6) to get the latitude and longitude, and then by formula... and The inverse operation yields the sampling coordinates on the ERP image. This "output-to-input" reverse mapping strategy naturally avoids the hole problem that may occur with forward mapping.

[0070] Step S4e: Perform bilinear interpolation on the sampling coordinates, extract pixel values ​​from the isometric projection format panoramic video frame of the current frame, and generate a stabilized frame after image stabilization.

[0071] Due to sampling coordinates Typically, the values ​​are floating-point numbers. Bilinear interpolation is used to extract pixel values ​​from the ERP image of the current frame and fill them into the corresponding positions in the output frame. By iterating through all pixels in the entire output frame, a complete stabilized frame can be generated.

[0072] The technical solution described in steps S4c to S4e above pre-constructs a global isometric projection format orientation vector grid covering the entire panoramic view. A compensation rotation matrix is ​​then applied to this grid, and the sampling coordinates are calculated inversely. This is a reverse mapping strategy "from output to input," naturally avoiding the hole problem of forward mapping. Simultaneously, bilinear interpolation ensures a smooth transition of pixel values, resulting in stabilized frames without holes or overlaps, maintaining the complete viewpoint of the original panoramic video, minimizing image quality loss, and providing a good user experience.

[0073] From the above Figure 1As can be seen from the example of the panoramic video stabilization method, on the one hand, by constructing a lookup table to map panoramic video frames in equidistant projection format to each distortion-free plane of the cube map, the influence of polar distortion on subsequent processing is eliminated, enabling feature point detection and optical flow tracing to be performed on an approximately planar image, significantly improving the stability and accuracy of feature matching. On the other hand, by back-projecting planar two-dimensional feature point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, and combining random sampling consensus algorithm and singular value decomposition under three-dimensional rotation constraints for attitude estimation, it can accurately fit the inherent spherical rotation motion model of panoramic video, overcoming the model mismatch problem of traditional planar motion models when describing spherical motion, thus obtaining a more accurate inter-frame attitude rotation matrix. On the third hand, by converting the attitude rotation matrix into a unit quaternion and updating the original cumulative attitude trajectory, and combining it with spherical linear interpolation filtering to obtain a smooth attitude trajectory, and then calculating the compensation rotation matrix and performing reverse resampling, high-quality stabilization processing of panoramic video frames is achieved, effectively suppressing video image jitter and improving the visual experience of panoramic video. In summary, the technical solution of this application eliminates the influence of polar distortion and accurately estimates the spherical rotation motion, thereby achieving high-quality panoramic video stabilization.

[0074] Please see the appendix Figure 2 This application provides a panoramic video stabilization device, which includes a construction module 201, an acquisition module 202, an estimation module 203, and a generation module 204, as detailed below: Module 201 is used to build a lookup table for mapping panoramic video frames in isometric projection format to each distortion-free plane of a cube map; The acquisition module 202 is used to resample the current frame and adjacent frames based on a lookup table for mapping panoramic video frames in isometric projection format to each distortion-free plane of a cube map, to obtain multiple sets of corresponding distortion-free plane images, and to perform feature point detection and optical flow tracing on each set of distortion-free plane images to obtain two-dimensional feature point pairs of the plane. The estimation module 203 is used to back-project the two-dimensional feature point pairs on the plane to the three-dimensional space to form three-dimensional unit ray direction vector pairs, and based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, estimate the attitude rotation matrix between adjacent frames through a random sampling consensus algorithm combined with singular value decomposition. The generation module 204 is used to convert the attitude rotation matrix into a unit quaternion to update the original cumulative attitude trajectory, and to perform spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, it calculates the compensation rotation matrix based on the smooth attitude trajectory and the original cumulative attitude trajectory, and applies the compensation rotation matrix to perform reverse resampling on the panoramic video frame to generate a stabilized frame after image stabilization.

[0075] From the above Figure 2As can be seen from the example of the panoramic video stabilization device, on the one hand, by constructing a lookup table to map panoramic video frames in equidistant projection format to each distortion-free plane of the cube map, the influence of polar distortion on subsequent processing is eliminated, enabling feature point detection and optical flow tracing to be performed on an approximately planar image, significantly improving the stability and accuracy of feature matching. On the other hand, by back-projecting planar two-dimensional feature point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, and combining random sampling consensus algorithm and singular value decomposition under three-dimensional rotation constraints for attitude estimation, it can accurately fit the inherent spherical rotation motion model of panoramic video, overcoming the model mismatch problem of traditional planar motion models when describing spherical motion, thus obtaining a more accurate inter-frame attitude rotation matrix. On the third hand, by converting the attitude rotation matrix into a unit quaternion and updating the original cumulative attitude trajectory, and combining it with spherical linear interpolation filtering to obtain a smooth attitude trajectory, and then calculating the compensation rotation matrix and performing reverse resampling, high-quality stabilization processing of panoramic video frames is achieved, effectively suppressing video image jitter and improving the visual experience of panoramic video. In summary, the technical solution of this application eliminates the influence of polar distortion and accurately estimates the spherical rotation motion, thereby achieving high-quality panoramic video stabilization.

[0076] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a panoramic video stabilization method. When the processor 30 executes the computer program 32, it implements the steps described in the panoramic video stabilization method embodiment, for example... Figure 1 Steps S1 to S4 are shown. Alternatively, when processor 30 executes computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the construction module 201, acquisition module 202, estimation module 203, and generation module 204 are shown.

[0077] For example, the computer program 32 of the panoramic video stabilization method mainly includes: constructing a lookup table for mapping panoramic video frames in isometric projection format to each distortion-free plane of a cube map; based on the lookup table, resampling the current frame and adjacent frames to obtain multiple sets of corresponding distortion-free plane images, and performing feature point detection and optical flow tracing on each set of distortion-free plane images to obtain planar two-dimensional feature point pairs; back-projecting the planar two-dimensional feature point pairs to three-dimensional space to form three-dimensional unit ray direction vector pairs, and estimating the attitude rotation matrix between adjacent frames based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints through a random sampling consensus algorithm combined with singular value decomposition; converting the attitude rotation matrix into unit quaternions to update the original cumulative attitude trajectory, and performing spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory, then calculating a compensation rotation matrix based on the smooth attitude trajectory and the original cumulative attitude trajectory, and applying the compensation rotation matrix to back-resample the panoramic video frames to generate stabilized frames after stabilization. Computer program 32 can be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 32 in electronic device 3. For example, computer program 32 can be divided into the functions of a construction module 201, an acquisition module 202, an estimation module 203, and a generation module 204 (a module in a virtual device). The specific functions of each module are as follows: Construction module 201 is used to construct a lookup table for mapping panoramic video frames in isometric projection format to each distortion-free plane of a cube map; Acquisition module 202 uses this lookup table to resample the current frame and adjacent frames to obtain multiple sets of distortion-free plane images, and performs feature point detection and optical flow tracing on each set of distortion-free plane images to obtain two-dimensional feature point pairs of the plane; Estimation module 203 is used to estimate the plane... Two-dimensional feature point pairs are back-projected into three-dimensional space to form three-dimensional unit ray direction vector pairs. Based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, the attitude rotation matrix between adjacent frames is estimated by combining random sampling consensus algorithm with singular value decomposition. The generation module 204 is used to convert the attitude rotation matrix into unit quaternions to update the original cumulative attitude trajectory, and to perform spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, the compensation rotation matrix is ​​calculated based on the smooth attitude trajectory and the original cumulative attitude trajectory, and the compensation rotation matrix is ​​applied to reverse resample the panoramic video frames to generate stabilized frames after image stabilization.

[0078] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0079] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0080] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0082] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the panoramic video stabilization method can be stored in a storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above, namely, constructing a lookup table for mapping panoramic video frames in equidistant projection format to each distortion-free plane of a cube map; based on this lookup table, resampling the current frame and adjacent frames to obtain multiple sets of corresponding distortion-free plane images; and performing feature point detection and optical flow tracing on each set of distortion-free plane images to obtain the plane. Two-dimensional feature point pairs are back-projected onto three-dimensional space to form three-dimensional unit ray direction vector pairs. Based on these pairs and under three-dimensional rotation constraints, the attitude rotation matrix between adjacent frames is estimated using a random sampling consensus algorithm combined with singular value decomposition. The attitude rotation matrix is ​​then converted into unit quaternions to update the original accumulated attitude trajectory. Spherical linear interpolation filtering is applied to the original accumulated attitude trajectory to obtain a smoothed attitude trajectory. A compensation rotation matrix is ​​then calculated based on the smoothed attitude trajectory and the original accumulated attitude trajectory. This compensation rotation matrix is ​​then applied to reverse-sample the panoramic video frames to generate stabilized frames. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The storage medium can include any entity or device capable of carrying computer program code, recording media, USB flash drives, external hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of the storage medium may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the storage medium may not include electrical carrier signals and telecommunication signals.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A panoramic video stabilization method, characterized in that, The method includes: Construct a lookup table to map panoramic video frames in isometric projection format to each distortion-free plane of a cubemap; Based on the lookup table, the current frame and adjacent frames are resampled to obtain multiple sets of distortion-free planar images. Feature point detection and optical flow tracing are performed on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs. The planar two-dimensional feature point pairs are back-projected into three-dimensional space to form three-dimensional unit ray direction vector pairs. Based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, the attitude rotation matrix between adjacent frames is estimated by combining the random sampling consensus algorithm with singular value decomposition. The attitude rotation matrix is ​​converted into a unit quaternion to update the original cumulative attitude trajectory, and spherical linear interpolation filtering is performed on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, a compensation rotation matrix is ​​calculated based on the smooth attitude trajectory and the original cumulative attitude trajectory, and the compensation rotation matrix is ​​applied to reverse resample the panoramic video frame to generate a stabilized frame after image stabilization.

2. The panoramic video stabilization method according to claim 1, characterized in that, Based on the lookup table, the current frame and adjacent frames are resampled to obtain multiple sets of distortion-free planar images. Feature point detection and optical flow tracing are performed on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs, including: Feature points are extracted from the distortion-free planar images in each group using the minimum eigenvalue algorithm. The feature points are traced on the corresponding distortion-free planar images of adjacent frames using the pyramid optical flow method to obtain the planar two-dimensional feature point pairs.

3. The panoramic video stabilization method according to claim 2, characterized in that, The step of back-projecting the two-dimensional feature point pairs onto three-dimensional space to form three-dimensional unit ray direction vector pairs, and estimating the pose rotation matrix between adjacent frames based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints using a random sampling consensus algorithm combined with singular value decomposition, includes: By back-projecting each pair of points in the two-dimensional feature point pair onto three-dimensional space, we obtain the corresponding two three-dimensional unit ray direction vectors. and And establish ideal rotational constraints The R Let be the attitude rotation matrix to be determined; In each iteration of the random sampling consensus algorithm, at least two pairs of the three-dimensional unit ray direction vectors are randomly selected, the number of inliers is evaluated based on the residual angle between the direction vectors, and the attitude rotation matrix that minimizes the error of the inlier set is solved by singular value decomposition.

4. The panoramic video stabilization method according to claim 3, characterized in that, The spherical linear interpolation filter uses a smoothing factor. The smooth attitude trajectory is obtained in the following way: The first k The pose rotation matrix of the frame is converted into a unit quaternion. and in accordance with Update the original accumulated attitude trajectory; Quaternions of adjacent moments in the original cumulative attitude trajectory and Perform spherical linear interpolation, i.e. The smooth attitude trajectory is obtained.

5. The panoramic video stabilization method according to claim 4, characterized in that, The process of applying the compensation rotation matrix to perform reverse resampling of the panoramic video frame includes: Pre-construct a global isometric projection format orientation vector grid covering the entire panoramic view; The compensation rotation matrix is ​​applied to the global isometric projection format direction vector grid to obtain the rotated direction vector, and the sampling coordinates on the isometric projection format image are calculated based on the rotated direction vector. Bilinear interpolation is performed on the sampled coordinates to extract pixel values ​​from the isometric projection format panoramic video frame of the current frame, generating the stabilized frame after image stabilization.

6. The panoramic video stabilization method according to claim 3, characterized in that, After estimating the pose rotation matrix between adjacent frames, the method further includes: The attitude rotation matrix estimated from multiple consecutive frames is subjected to a time consistency check. The rotation angle between adjacent frames is calculated. If the rotation angle exceeds a preset threshold, the attitude rotation matrix of the current frame is replaced with the attitude rotation matrix of the previous frame or interpolation correction is performed.

7. The panoramic video stabilization method according to claim 3, characterized in that, The method for evaluating the number of interior points based on the residual of the angle between direction vectors includes: In each iteration, the in-point decision threshold is dynamically calculated based on the median or standard deviation of the angle residuals between the three-dimensional unit ray direction vector pairs of the current in-point set. The direction vector pairs with an included angle residual less than the in-point determination threshold are marked as in-points, and the number of in-points is counted.

8. A panoramic video stabilization device, characterized in that, The device includes: The module is used to build a lookup table for mapping panoramic video frames in isometric projection format to the distortion-free planes of a cubemap; The acquisition module is used to resample the current frame and adjacent frames based on the lookup table to obtain multiple sets of distortion-free planar images, and to perform feature point detection and optical flow tracing on each set of distortion-free planar images to obtain planar two-dimensional feature point pairs. The estimation module is used to back-project the two-dimensional feature point pairs on the plane to three-dimensional space to form three-dimensional unit ray direction vector pairs, and based on the three-dimensional unit ray direction vector pairs under three-dimensional rotation constraints, estimate the attitude rotation matrix between adjacent frames through a random sampling consensus algorithm combined with singular value decomposition. The generation module is used to convert the attitude rotation matrix into a unit quaternion to update the original cumulative attitude trajectory, and to perform spherical linear interpolation filtering on the original cumulative attitude trajectory to obtain a smooth attitude trajectory. Then, it calculates a compensation rotation matrix based on the smooth attitude trajectory and the original cumulative attitude trajectory, and applies the compensation rotation matrix to perform reverse resampling on the panoramic video frame to generate a stabilized frame after image stabilization.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the panoramic video stabilization method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the panoramic video stabilization method as described in any one of claims 1 to 7.

Images