A panoramic video anti-shake method and device, a storage medium and an electronic device

Abstract

The application provides a panoramic video anti-shake method and device, a storage medium and an electronic device, which are applied to the technical field of video processing, and the motion information of a panoramic camera at a current moment and the images of each lens are obtained; the motion information and the lens parameters of the panoramic camera are processed to obtain the motion compensation strategy of each lens; the motion compensation strategy of each lens is applied to perform one-frame multi-lens synchronous compensation on the image sequence of each lens, and meanwhile, anti-shake processing is performed on multiple lenses to obtain the compensation images of each lens at the current moment, and based on the types of the image pickup sensors of each lens, the compensation images of each lens at different moments are cropped and spliced to obtain an anti-shake panoramic video. The motion information of the panoramic camera and the images of each lens are fused, and meanwhile, anti-shake is performed on multiple lenses, so that the picture motion of multiple lenses is synchronized, the problems of ghosting and faulting in picture splicing are avoided, and the picture quality of the panoramic video is improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a panoramic video stabilization method and apparatus, storage medium and electronic device. Background Technology

[0002] Panoramic cameras, capable of 360° omnidirectional imaging, are widely used in sports photography, scene reconstruction, VR content creation, and other fields. Their core working principle involves capturing images from different perspectives using at least two symmetrically arranged lenses (such as front and rear dual lenses or multi-lens arrays), and then stitching them together using a stitching algorithm to create a complete panoramic image. Unlike ordinary single-lens cameras, the image stabilization effect of panoramic cameras directly impacts the image stitching accuracy and overall visual experience, presenting them with far greater challenges than ordinary cameras.

[0003] Most current image stabilization solutions use inertial measurement unit (IMU) data or image data alone, and only perform image stabilization processing on a single lens. This cannot adapt to the needs of coordinating and stitching multiple lens images, resulting in asynchronous movement of multiple lens images and problems such as stitching ghosting and discontinuity that affect the image quality of panoramic videos. Summary of the Invention

[0004] In view of this, embodiments of this application provide a panoramic video stabilization method and apparatus, storage medium and electronic device. The panoramic video stabilization solution provided by this application fuses the motion information of the panoramic camera with the images of each lens and simultaneously stabilizes multiple lenses, so that the motion of the images of multiple lenses is synchronized, avoiding the problems of ghosting and discontinuity in image stitching, and improving the image quality of panoramic video.

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions: A panoramic video stabilization method includes: Acquire motion information of the panoramic camera corresponding to the current moment and images from each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens; Obtain the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens; By applying the motion compensation strategy of each of the aforementioned shots, multi-shot synchronous compensation is performed on the image of each of the aforementioned shots in one frame to obtain the compensated image of each of the aforementioned shots at the current moment. Based on the camera sensor type of each lens, the compensation images of each lens at different times are cropped and stitched together to output a stabilized panoramic video.

[0006] Optionally, in the above method, acquiring the motion information of the panoramic camera corresponding to the current moment and the images from each lens of the panoramic camera includes: The IMU data of the panoramic camera at the current moment is acquired through the IMU sensor, and the images of each of the lenses at the current moment are also acquired. The IMU data is denoised, the IMU sensor is calibrated, and IMU dynamic zero bias calibration data is output. The denoised IMU data and the images of each of the lenses are timestamped together. The motion trajectory of the panoramic camera is obtained by integrating the denoised IMU data. Based on a preset IMU dynamic zero-bias calibration frequency, the motion trajectory of the panoramic camera is corrected using the IMU dynamic zero-bias calibration data. Based on the corrected motion trajectory and the lens layout parameters of the panoramic camera, the short-term motion estimation result of the panoramic camera's IMU is obtained. The short-term motion estimation result of the panoramic camera includes the overall motion parameters of the panoramic camera and the relative motion deviation between each lens. The short-term motion estimation results of the IMU are processed using lightweight Kalman filtering and the IMU dynamic zero-bias calibration data to obtain the motion information of the panoramic camera. The motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens.

[0007] Optionally, in the above method, obtaining the motion compensation strategy for each lens based on the lens parameters and the motion information includes: The overall motion information and the relative motion information between each shot in the motion information are decomposed to obtain the independent motion information of each shot. The independent motion information includes the rotation matrix and translation vector of the shot. The lens parameters and target pixel coordinates are processed to obtain a lens distortion correction strategy, wherein the target pixel coordinates are the pixel coordinates of the pixels in the panoramic camera that participate in the panoramic image motion mapping calculation. The lens distortion correction strategy and the independent motion information of each lens are processed to obtain the motion compensation strategy for each lens.

[0008] Optionally, in the above method, processing the lens parameters and target pixel coordinates to obtain a lens distortion correction strategy includes: By applying the lens focal length from the lens parameters, the target pixel coordinates are converted into coordinates in the camera coordinate system to obtain normalized coordinates; The radial distortion parameter and tangential distortion parameter in the lens parameters are used to correct the normalized coordinates to obtain the corrected normalized coordinates; The corrected normalized coordinates are converted into coordinates in the pixel coordinate system to obtain the lens distortion correction strategy.

[0009] Optionally, in the above method, processing the lens distortion correction strategy and the independent motion information of each lens to obtain the motion compensation strategy for each lens includes: Extract the two-dimensional components of the rotation matrix from the independent motion information of each lens to obtain the two-dimensional rotation matrix of each lens; The translation component in the independent motion information of each lens is transformed from three dimensions to two dimensions to obtain the two-dimensional translation component of each lens; A matrix is ​​constructed by applying the two-dimensional rotation matrix and two-dimensional translation component of each lens to obtain the initial affine transformation matrix of each lens; Acquire IMU dynamic zero-bias calibration data, and use the IMU dynamic zero-bias calibration data to calibrate each of the initial affine transformation matrices to obtain the affine transformation matrix of each lens; Based on the lens distortion correction strategy and the affine transformation matrix of each lens, a motion compensation strategy for each lens is obtained.

[0010] Optionally, the above method involves applying motion compensation strategies to each of the lenses to perform one-frame multi-lens synchronous compensation on the images of each lens, obtaining compensated images of each lens at the current moment, including: For each image from the lens, the motion compensation strategy of the lens is used to perform a compensation operation on each pixel in the image one by one; For each lens image undergoing compensation, the compensation progress of each lens is acquired in real time; each lens is paired to obtain a lens combination, and the compensation progress deviation of each lens combination is determined. The compensation progress deviation is the absolute value of the difference between the compensation progress of the two lenses in the lens combination; when the compensation progress deviation of any lens combination is greater than a preset progress deviation threshold, the compensation rate of the lens with the larger compensation progress in the lens combination is reduced, and the compensation progress deviation of the lens combination is continuously monitored until it is less than or equal to the preset progress deviation threshold, at which point the adjustment stops. After the images of each lens are compensated, the residual jitter is obtained based on the compensated images of each lens. When the residual jitter is not greater than a preset residual jitter threshold, the compensated images of each lens at the current time are output. When the residual jitter is greater than the preset residual jitter threshold, the IMU dynamic zero-bias calibration frequency is adjusted, and the motion compensation strategy of each lens is reacquired. The reacquired motion compensation strategies are used to perform one-frame multi-lens synchronous compensation on the images of each lens again until the residual jitter is not greater than the preset residual jitter threshold, and then the compensated images of each lens at the current time are output.

[0011] Optionally, in the above method, the step of using the lens's motion compensation strategy to compensate for each pixel in the image individually includes: For each pixel in the image, the lens distortion correction strategy in the motion compensation strategy is applied to correct the distortion of the pixel to obtain the distortion correction coordinates corresponding to the pixel. The distortion correction coordinates of the pixel and the affine transformation matrix in the motion compensation strategy are processed to obtain the compensated coordinates of the pixel; When the compensated coordinates of the pixel are integer coordinates, the compensation operation for the pixel ends; when the compensated coordinates of the pixel are non-integer coordinates, a bilinear interpolation algorithm is used to fill the grayscale values ​​of the compensated coordinates of the pixel, and the compensation operation for the pixel ends.

[0012] A panoramic video stabilization device includes: The first acquisition unit is used to acquire motion information of the panoramic camera corresponding to the current time and images from each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens. The second acquisition unit is used to acquire the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens. The synchronization compensation unit is used to apply the motion compensation strategy of each of the lenses to perform one-frame multi-lens synchronization compensation on the image of each of the lenses, so as to obtain the compensated image of each of the lenses at the current time. The stitching unit is used to crop and stitch the compensated images of each of the lenses at different times based on the camera sensor type of each lens, and output a stabilized panoramic video.

[0013] Optionally, in the aforementioned apparatus, the first acquisition unit performs the process of acquiring motion information of the panoramic camera corresponding to the current time and images from each lens of the panoramic camera, including: The IMU data of the panoramic camera at the current moment is acquired through the IMU sensor, and the images of each of the lenses at the current moment are also acquired. The IMU data is denoised, the IMU sensor is calibrated, and IMU dynamic zero bias calibration data is output. The denoised IMU data and the images of each of the lenses are timestamped together. The motion trajectory of the panoramic camera is obtained by integrating the denoised IMU data. Based on a preset IMU dynamic zero-bias calibration frequency, the motion trajectory of the panoramic camera is corrected using the IMU dynamic zero-bias calibration data. Based on the corrected motion trajectory and the lens layout parameters of the panoramic camera, the short-term motion estimation result of the panoramic camera's IMU is obtained. The short-term motion estimation result of the panoramic camera includes the overall motion parameters of the panoramic camera and the relative motion deviation between each lens. The short-term motion estimation results of the IMU are processed using lightweight Kalman filtering and the IMU dynamic zero-bias calibration data to obtain the motion information of the panoramic camera. The motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens.

[0014] Optionally, in the aforementioned apparatus, the second acquisition unit performs a process of obtaining motion compensation strategies for each lens based on the lens parameters and the motion information, including: The overall motion information and the relative motion information between each shot in the motion information are decomposed to obtain the independent motion information of each shot. The independent motion information includes the rotation matrix and translation vector of the shot. The lens parameters and target pixel coordinates are processed to obtain a lens distortion correction strategy, wherein the target pixel coordinates are the pixel coordinates of the pixels in the panoramic camera that participate in the panoramic image motion mapping calculation. The lens distortion correction strategy and the independent motion information of each lens are processed to obtain the motion compensation strategy for each lens.

[0015] Optionally, in the aforementioned apparatus, the second acquisition unit performs a process of processing the lens parameters and target pixel coordinates to obtain a lens distortion correction strategy, including: By applying the lens focal length from the lens parameters, the target pixel coordinates are converted into coordinates in the camera coordinate system to obtain normalized coordinates; The radial distortion parameter and tangential distortion parameter in the lens parameters are used to correct the normalized coordinates to obtain the corrected normalized coordinates; The corrected normalized coordinates are converted into coordinates in the pixel coordinate system to obtain the lens distortion correction strategy.

[0016] Optionally, in the aforementioned apparatus, the second acquisition unit performs a process of processing the lens distortion correction strategy and the independent motion information of each lens to obtain the motion compensation strategy for each lens, including: Extract the two-dimensional components of the rotation matrix from the independent motion information of each lens to obtain the two-dimensional rotation matrix of each lens; The translation component in the independent motion information of each lens is transformed from three dimensions to two dimensions to obtain the two-dimensional translation component of each lens; A matrix is ​​constructed by applying the two-dimensional rotation matrix and two-dimensional translation component of each lens to obtain the initial affine transformation matrix of each lens; Acquire IMU dynamic zero-bias calibration data, and use the IMU dynamic zero-bias calibration data to calibrate each of the initial affine transformation matrices to obtain the affine transformation matrix of each lens; Based on the lens distortion correction strategy and the affine transformation matrix of each lens, a motion compensation strategy for each lens is obtained.

[0017] Optionally, in the aforementioned apparatus, the process by which the synchronization compensation unit executes a motion compensation strategy applied to each of the lenses, performs one-frame multi-lens synchronization compensation on the images of each of the lenses, and obtains the compensated images of each of the lenses at the current moment includes: For each image from the lens, the motion compensation strategy of the lens is used to perform a compensation operation on each pixel in the image one by one; For each lens image undergoing compensation, the compensation progress of each lens is acquired in real time; each lens is paired to obtain a lens combination, and the compensation progress deviation of each lens combination is determined. The compensation progress deviation is the absolute value of the difference between the compensation progress of the two lenses in the lens combination; when the compensation progress deviation of any lens combination is greater than a preset progress deviation threshold, the compensation rate of the lens with the larger compensation progress in the lens combination is reduced, and the compensation progress deviation of the lens combination is continuously monitored until it is less than or equal to the preset progress deviation threshold, at which point the adjustment stops. After the images of each lens are compensated, the residual jitter is obtained based on the compensated images of each lens. When the residual jitter is not greater than a preset residual jitter threshold, the compensated images of each lens at the current time are output. When the residual jitter is greater than the preset residual jitter threshold, the IMU dynamic zero-bias calibration frequency is adjusted, and the motion compensation strategy of each lens is reacquired. The reacquired motion compensation strategies are used to perform one-frame multi-lens synchronous compensation on the images of each lens again until the residual jitter is not greater than the preset residual jitter threshold, and then the compensated images of each lens at the current time are output.

[0018] Optionally, in the aforementioned apparatus, the synchronization compensation unit performs a process of compensating each pixel in the image one by one using the motion compensation strategy of the lens, including: For each pixel in the image, the lens distortion correction strategy in the motion compensation strategy is applied to correct the distortion of the pixel to obtain the distortion correction coordinates corresponding to the pixel. The distortion correction coordinates of the pixel and the affine transformation matrix in the motion compensation strategy are processed to obtain the compensated coordinates of the pixel; When the compensated coordinates of the pixel are integer coordinates, the compensation operation for the pixel ends; when the compensated coordinates of the pixel are non-integer coordinates, a bilinear interpolation algorithm is used to fill the grayscale values ​​of the compensated coordinates of the pixel, and the compensation operation for the pixel ends.

[0019] A storage medium comprising stored instructions, wherein, when the instructions are executed, the device on which the storage medium resides executes the panoramic video stabilization method as described above.

[0020] An electronic device includes a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors to implement the panoramic video stabilization method described above.

[0021] Compared with the prior art, this application has the following beneficial effects: This application provides a panoramic video stabilization method, apparatus, storage medium, and electronic device. It acquires the motion information of the panoramic camera and images from each lens at the current moment; processes the motion information and lens parameters of the panoramic camera to obtain a motion compensation strategy for each lens; applies the motion compensation strategy for each lens to perform one-frame multi-lens synchronous compensation on the image sequence of each lens, thereby simultaneously stabilizing multiple lenses and obtaining compensated images for each lens at the current moment. Furthermore, based on the camera sensor type of each lens, the compensated images of each lens at different times are cropped and stitched together to obtain a stabilized panoramic video. By fusing the motion information of the panoramic camera with the images from each lens and simultaneously stabilizing multiple lenses, the motion of the multi-lens images is synchronized, avoiding ghosting and discontinuity issues during image stitching, and improving the image quality of the panoramic video. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 A flowchart of a panoramic video stabilization method provided in this application embodiment; Figure 2 A flowchart illustrating the motion compensation strategy for each lens based on lens parameters and motion information, provided in another embodiment of this application; Figure 3 A flowchart illustrating a lens distortion correction strategy provided in another embodiment of this application; Figure 4 A flowchart illustrating a method for obtaining motion compensation strategies for each shot, provided in another embodiment of this application; Figure 5 An example diagram is provided for applying motion compensation strategies for each lens to perform multi-lens compensation on the images of each lens in one frame, as an embodiment of this application. Figure 6 This is a schematic diagram of the structure of a panoramic video stabilization device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] As the background technology shows, current video stabilization solutions only stabilize individual lenses, failing to meet the demands of multi-lens image collaboration and stitching. This leads to asynchronous movement across multiple lenses, resulting in issues like ghosting and image fragmentation that negatively impact the quality of panoramic videos. Besides these problems, existing video stabilization technologies also utilize optical or electronic image stabilization (EIS) techniques. Optical stabilization compensates for shake through mechanical structures, a costly and power-consuming method that struggles to simultaneously compensate for the coordinated shake of multiple lenses in a panoramic camera. It also struggles to handle high-frequency shake common in panoramic shooting (such as handheld or rapid movement in action scenes). Electronic stabilization relies primarily on image feature matching. However, in low-light conditions and simple textures (such as skies or solid-color walls) during panoramic shooting, feature point extraction is difficult, limiting stabilization effectiveness and potentially causing misalignment and blurring in stitched areas, affecting the integrity of the panoramic image. Meanwhile, current video stabilization technologies do not take into account the sensor type of panoramic cameras (square or rectangular CMOS). The sensor type affects the margin of the stabilization algorithm. Traditional video stabilization solutions do not take this difference into account, resulting in poor adaptability of stabilization methods and poor power consumption control.

[0027] To address the aforementioned issues, this application provides a panoramic video stabilization solution. This solution acquires the motion information of the panoramic camera and images from each lens at the current moment; processes the motion information and lens parameters of the panoramic camera to obtain a motion compensation strategy for each lens; applies the motion compensation strategies for each lens to perform one-frame multi-lens synchronous compensation on the images of each lens, thereby simultaneously stabilizing multiple lenses and obtaining compensated images for each lens at the current moment. Furthermore, based on the camera sensor type of each lens, the compensated images of each lens at different times are cropped and stitched together to obtain a stabilized panoramic video. The stabilization process fuses the motion information of the panoramic camera with the images from each lens and simultaneously stabilizes multiple lenses, ensuring synchronized motion across multiple lenses, avoiding ghosting and discontinuity issues during image stitching, and improving the image quality of the panoramic video.

[0028] The solution provided in this application can be applied to processing devices that output panoramic video, and these processing devices can be set up in a variety of general-purpose or special-purpose computer device environments or configurations. Specifically, the processing device can be a server computer, a handheld or portable device, a tablet device, a multiprocessor device, a distributed computing environment including any of the above devices or equipment, etc.

[0029] Reference Figure 1 The following is a flowchart of a panoramic video stabilization method provided in an embodiment of this application, and is described in detail below.

[0030] S101. Acquire the motion information of the panoramic camera corresponding to the current moment and the images from each lens of the panoramic camera.

[0031] It should be noted that this involves acquiring the motion information of the panoramic camera at the current moment, as well as the images from each lens at the current moment. Furthermore, "current moment" can be understood as the current timestamp or the current video frame.

[0032] The timestamps of images acquired at different times are different; furthermore, images from each shot are acquired at different timestamps or video frames.

[0033] Motion information includes the overall motion information of the panoramic camera and the relative motion information between the individual lenses.

[0034] In the embodiments provided in this application, motion information of the panoramic camera is obtained by processing the lens layout parameters of the panoramic camera and the IMU data collected by the IMU sensor integrated in the panoramic camera.

[0035] The IMU sensor, which includes a gyroscope and an accelerometer, is integrated into the panoramic camera. Its installation position is close to the center of the lens group to ensure that the collected motion data can accurately reflect the overall motion state of the panoramic camera.

[0036] When acquiring IMU data, the IMU sensor collects gyroscope (angular velocity) and accelerometer (linear acceleration) data at a frequency higher than the image frame rate of each lens (e.g., 200Hz-1000Hz). The acquisition frequency can be dynamically adjusted according to the lens frame rate and CMOS type. For square CMOS, the sampling rate can be appropriately reduced to save power, while for rectangular CMOS, the sampling rate needs to be increased to compensate for insufficient imaging margin, ensuring that high-frequency shaking (such as handheld shaking, rapid turning) in panoramic shooting can be captured. At the same time, the shooting parameters of each lens, such as focal length and exposure time, are also acquired.

[0037] The acquired IMU data undergoes denoising and IMU sensor calibration. Low-pass filtering is applied to the acquired IMU data to remove environmental and sensor noise. Specifically, Kalman filtering or moving average filtering is used for low-pass filtering to denoise the IMU data. Sensor calibration algorithms are used to perform zero-bias and scale calibration on the IMU sensor, thereby eliminating zero-bias and scale errors and ensuring the accuracy of the acquired data. During the IMU sensor calibration process, dynamic zero-bias calibration data is output. Preferably, the dynamic zero-bias calibration data can be the calibration results tracked and output during the zero-bias calibration of the IMU sensor; the dynamic zero-bias calibration data includes, but is not limited to, gyroscope zero-bias estimates and accelerometer zero-bias estimates.

[0038] In the embodiments provided in this application, for multi-lens panoramic cameras, additional IMU data and relative position calibration of each lens are added to compensate for motion data deviation caused by lens layout. At the same time, combined with lens distortion parameter calibration, it lays the foundation for subsequent motion estimation.

[0039] Because panoramic cameras have slight time differences in multi-lens shooting and IMU data and image acquisition have temporal discrepancies, microsecond-level time synchronization algorithms can be used to precisely align the denoised IMU data with the images from each lens's timestamp. Furthermore, visual feature point matching and IMU data correlation analysis can be employed to ensure accurate correspondence between the denoised IMU data and the images at the corresponding moments. By aligning IMU data and images, the temporal discrepancies between IMU data and each image can be effectively eliminated, ensuring the temporal consistency of multi-lens motion estimation and avoiding misalignment of image stabilization compensation and stitching errors caused by temporal discrepancies. Synchronization accuracy is controlled within ±1ms, thus adapting to the needs of multi-lens collaborative shooting.

[0040] In the embodiments provided in this application, an IMU data integration algorithm is used to integrate the denoised IMU data to obtain IMU integrated data. The motion change parameters between two adjacent frames of the panoramic camera can be obtained from the IMU integrated data, thereby estimating the motion trajectory of the panoramic camera. To improve data accuracy, based on a preset IMU dynamic zero-bias calibration frequency, the motion trajectory of the panoramic camera is corrected using IMU dynamic zero-bias calibration data. The IMU dynamic zero-bias calibration frequency can be achieved by setting a preset frame interval, for example, error correction is performed every preset frame interval. Preferably, the preset frame interval can be adjusted as needed, for example, it can be 5 frames, 7 frames, 8 frames, etc.

[0041] The process of using IMU dynamic zero-bias calibration data to correct errors in the motion trajectory of a panoramic camera is as follows: the camera attitude change is obtained by integrating the angular velocity data collected by the IMU, and a filtering algorithm is used to smooth the camera attitude change during the integration process. The attitude estimation state (i.e., the motion trajectory of the panoramic camera) is periodically updated to reduce the attitude drift caused by the accumulation of integration, so that the motion compensation parameters calculated in the end are more accurate.

[0042] By correcting the cumulative drift error generated during the IMU data integration calculation process, the cumulative error generated during the IMU integration process is effectively suppressed, ensuring that the panoramic camera can maintain high motion estimation accuracy in long shooting scenarios (such as continuous shooting for more than 30 seconds), avoiding the drift phenomenon in the panoramic image, and ensuring the stability of the image stabilization effect.

[0043] Furthermore, quaternion conversion algorithms, Euler angle or direction cosine matrix attitude identification algorithms can be used to convert the angular velocity data collected by the IMU gyroscope into attitude data obtained by the panoramic camera. Here, the attitude data is the motion trajectory of the panoramic camera mentioned above. Preferably, the quaternion conversion algorithm has the following advantages: (1) There is no Euler angle gimbal lock problem, which is suitable for panoramic camera large-angle rotation scenarios. (2) The amount of computation is small and the calculation efficiency is high, which is suitable for real-time anti-interference. (3) The numerical stability is good and the error is small after multiple integrations. (4) It has good compatibility with subsequent attitude interpolation and motion compensation calculations. Therefore, the quaternion conversion algorithm can be used to estimate the attitude data, thereby optimizing the accuracy of the attitude data.

[0044] Based on the corrected motion trajectory and the lens layout parameters of the panoramic camera, the short-term motion estimation results of the IMU for the panoramic camera are obtained. These results include the overall motion parameters of the panoramic camera and the relative motion deviations between each lens. Lens distribution parameters include, but are not limited to, lens spacing and viewing angle. By integrating the IMU data, the motion trajectory of the panoramic camera is estimated in real time. Combining the motion trajectory and lens layout parameters, the overall motion parameters of the panoramic camera between two adjacent frames are accurately calculated, and the relative motion deviations between each lens are derived. The overall motion parameters include rotational and translational motion parameters.

[0045] The short-term motion estimation method adopted in this application does not rely on any image feature point extraction process or deep learning model when obtaining IMU short-term motion estimation results. It has the technical advantages of fast computing speed and low processing latency. Even in harsh shooting scenarios such as low light and simple textures, it can still stably provide high-precision motion estimation data for multiple lenses, effectively avoiding image stabilization failure caused by single-lens image feature loss and deep learning model computing lag. It is especially suitable for panoramic shooting needs in fast-moving scenarios.

[0046] In another embodiment provided in this application, after obtaining the short-term motion estimation results of the IMU, the short-term motion estimation results of the IMU are processed by lightweight Kalman filtering and IMU dynamic zero-bias calibration data to obtain the motion information of the panoramic camera. The motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens.

[0047] This application employs a lightweight Kalman filter (simplifying computational complexity and improving speed) to fuse and optimize only the short-term motion estimation results of the IMU, eliminating the need for additional panoramic characteristic constraint optimization. This yields a smooth and accurate overall motion trajectory of the panoramic camera (i.e., the overall motion information of the panoramic camera) and the relative motion trajectories of each lens (i.e., the relative motion information between each lens). For panoramic stitching requirements, the focus is on optimizing the motion fusion accuracy of the stitching area to ensure synchronized motion compensation for each lens. Simultaneously, the fusion weights are adjusted according to the CMOS type—square CMOS allows for reduced computational complexity to save power, while rectangular CMOS requires increased fusion accuracy to compensate for insufficient imaging margin, avoiding stitching ghosting and discontinuities caused by asynchronous movement of multiple lenses. Spherical linear interpolation using the Slerp algorithm improves the smoothness of the motion trajectory. Furthermore, dynamic zero-bias calibration data of the IMU is introduced during the fusion and optimization of the short-term motion estimation results using the lightweight Kalman filter to optimize fusion parameters in real time, further improving the stability and accuracy of motion estimation while balancing computational efficiency and image stabilization, ensuring the overall algorithm latency is controlled within 8ms.

[0048] S102. Obtain the lens parameters of the panoramic camera, and based on the lens parameters and motion information, obtain the motion compensation strategy for each lens.

[0049] The lens parameters of a panoramic camera include, but are not limited to, focal length, principal point coordinates, radial distortion coefficient, and tangential distortion coefficient. Furthermore, the focal lengths of different lenses in a panoramic camera can be different or the same.

[0050] The overall motion information of the panoramic camera and the relative motion information between each lens in the lens parameters and motion information of the panoramic camera are processed to obtain the motion compensation strategy for each lens. The motion compensation strategy is used to perform motion compensation on the image acquired by the lens.

[0051] S103. Apply motion compensation strategies for each shot to perform one-frame multi-shot synchronous compensation on the images of each shot, and obtain the compensated images of each shot at the current moment.

[0052] When performing multi-lens synchronous compensation on images from various lenses, compensation is applied to images from multiple lenses simultaneously to ensure that the motion states of the images from each lens are synchronized. This avoids subsequent stitching misalignment and discontinuity, thereby improving the image quality of the stitched panoramic image.

[0053] The motion compensation strategy of the lens at the current moment is used to compensate the image of the lens at the current moment, thereby obtaining the motion compensation strategy at the current moment.

[0054] S104. Based on the camera sensor type of each lens, the compensation images of each lens at different times are cropped and stitched together to output a stabilized panoramic video.

[0055] At each different moment, each lens has a corresponding compensated image. For any given moment, based on the camera sensor type of each lens, the compensated images of each lens at that moment are cropped and stitched together to obtain the panoramic image at that moment; the panoramic images from each moment are then combined to create a panoramic video.

[0056] The camera sensor types of the various lenses in a panoramic camera may differ or be the same. For example, camera sensor types include, but are not limited to, square CMOS and rectangular CMOS. Different camera sensor types have different imaging characteristics. By cropping and stitching the supplemented images based on the camera sensor type, the effective imaging area after compensation can be effectively preserved, while invalid edge areas caused by compensation are eliminated. This further avoids discontinuities and ghosting in the stitched panoramic image, resulting in a high-quality image without discontinuities or ghosting.

[0057] For example, adaptive intelligent cropping is performed on each compensated image from each lens, retaining only the effective imaging area after compensation and removing invalid edge areas caused by compensation. The cropping process adapts to the motion compensation results. For lenses with square CMOS sensors, which have a circular image field, the cropping boundary is circular with a radius of... Calculation formula: Where W is the image width and H is the image height; this cropping method ensures the preservation of the core area of ​​the circular image field, reducing cropping loss. For lenses with square CMOS sensors, the cropping boundary of the compensated image is rectangular, and the boundary coordinates are determined by the effective pixel area after compensation. Priority is given to preserving the image center and the stitching overlap area to avoid cropping critical areas.

[0058] In the embodiments provided in this application, after cropping, the images after compensation and cropping of each lens are combined into a complete and stable panoramic image using a conventional panoramic stitching algorithm. The stitching process only adapts to the image state after motion compensation and does not perform detailed calculations, thereby ensuring that the stitching error is controlled within 0.2mm, and finally outputting panoramic images and videos that meet the requirements.

[0059] In another embodiment provided in this application, in order to maintain the integrity of the panoramic image and the 360° field of view, adaptive intelligent cropping is performed on each image of each compensated lens. During the cropping process, the importance of the content in the image is analyzed, and key areas of the panoramic image (such as faces, main objects, etc.) and overlapping areas are prioritized for preservation. The cropping boundary is dynamically adjusted to avoid cropping key areas. For the compensated image of a square CMOS lens, its circular image field advantage can be fully utilized to reduce cropping loss and improve image utilization. For the compensated image of a rectangular CMOS lens, the cropping algorithm is optimized to reduce the cropping of invalid areas and ensure that the stitched panoramic image has no obvious discontinuities or ghosting. Finally, the panoramic stitching algorithm is used to synthesize the compensated and cropped images of each lens into a complete and stable panoramic image, thereby improving the quality of the panoramic image.

[0060] The method provided in this application combines IMU dynamic zero-bias calibration and panoramic characteristic constraints, which not only solves the problem of accumulated error in traditional short-term motion estimation IMU integration, but also avoids the slow speed and stuttering issues caused by deep learning models. Furthermore, it achieves multi-lens synchronous image stabilization, solving the stitching misalignment problem caused by asynchronous motion of multiple lenses in a panoramic camera. Further, this solution targets the core characteristics of multi-lens collaborative shooting and image stitching in panoramic cameras. By efficiently fusing IMU motion data with multi-lens image information, it achieves multi-lens synchronous image stabilization and precise compensation of the stitching area, solving problems such as image blurring and stitching misalignment caused by shaking in panoramic shooting. This results in more accurate, adaptive, and low-latency shaking compensation, improving the stability and stitching quality of panoramic images and videos in various shooting scenarios.

[0061] Reference Figure 2 The flowchart of obtaining motion compensation strategies for each lens based on lens parameters and motion information, which is another embodiment provided in this application, is described in detail below.

[0062] S201. Decompose the overall motion information and the relative motion information between each shot in the motion information to obtain the independent motion information of each shot.

[0063] In the embodiments provided in this application, the coordinate system involved in the process of obtaining the motion compensation strategy is defined, including: the world coordinate system of the panoramic camera is... The camera coordinate system is The image pixel coordinate system is .

[0064] The overall motion information includes, but is not limited to, the overall motion parameters of the panoramic camera. The overall motion parameters of the panoramic camera include, but are not limited to, the overall rotation matrix R of adjacent frames and the overall translation vector t. The overall rotation matrix R of adjacent frames is used to describe the camera's attitude change and can be a 3×3 matrix; the overall translation vector t is used to describe the camera's position change and can be a 3×1 vector.

[0065] The relative motion information between the lenses includes, but is not limited to, the offset parameters of each lens relative to the camera coordinate system, which include the rotation offset matrix. Translation offset vector The value of i is related to the number of lenses in the panoramic camera, and i is a positive integer. For example, when the panoramic camera is a dual-lens panoramic camera, i = 1, 2. Preferably, the rotation offset matrix... It is a 3×3 matrix, and the translation offset vector is... It is a 3×1 vector.

[0066] Based on the overall motion parameters of the panoramic camera and the offset parameters of each lens relative to the camera coordinate system, the independent motion information of each lens is obtained by decomposition. The independent motion information includes the rotation matrix and translation vector of the lens.

[0067] The rotation matrix and translation vector of the lens can be expressed by formulas, as shown in equations (1) and (2).

[0068] Equation (1) Equation (2) in, Let be the rotation matrix of the i-th shot, reflecting the pose change of the shot relative to the previous frame (corresponding to the rotation component of the jitter). Let be the translation vector of the i-th shot, reflecting the positional change of the shot relative to the previous frame (corresponding to the translation component of the shake); R is the overall rotation matrix of adjacent frames; t is the overall translation vector; Let be the rotation offset matrix for the i-th lens; Let be the translation offset vector of the i-th lens. Matrix multiplication uses standard matrix operations of 3×3 and 3×3, and 3×3 and 3×1 to ensure computational accuracy. Fixed-point arithmetic optimization is used during the calculation process to improve computational speed and adapt to the real-time requirements of panoramic cameras.

[0069] S202. Process the lens parameters and target pixel coordinates to obtain the lens distortion correction strategy.

[0070] The target pixel coordinates are the pixel coordinates of the pixels in the panoramic camera that participate in the panoramic image motion mapping calculation. Furthermore, using the camera pose change parameters mentioned above (which can be understood as the motion trajectory of the panoramic camera), combined with lens calibration parameters and panoramic stitching mapping relationships, the pixel mapping coordinates of each lens image during the motion compensation process can be determined; these coordinates are the target pixel coordinates.

[0071] Lens parameters include, but are not limited to, focal length. Image principal point coordinates Radial distortion coefficient Tangential distortion coefficient Furthermore, the principal point coordinates of the image are the offset of the origin of the pixel coordinate system relative to the image center. Here, the focal length *f* is in pixels. Radial distortion coefficients... Tangential distortion coefficient All parameters are calibrated before the camera leaves the factory and pre-stored in the control module.

[0072] Reference Figure 3 The following is a flowchart of another embodiment of the present application for obtaining a lens distortion correction strategy, which is described in detail below.

[0073] S301. Using the lens focal length in the lens parameters, the target pixel coordinates are converted into coordinates in the camera coordinate system to obtain normalized coordinates.

[0074] By using the target pixel coordinates The normalized coordinates are obtained by calculating the focal length from the lens parameters. .

[0075] Normalized coordinates The calculation formula is shown in equation (3).

[0076] Equation (3) It should be noted that, For the focal length, when different lenses have different focal lengths, the focal length in equation (3) needs to be calculated using the focal length of the lens corresponding to the target pixel coordinates.

[0077] S302. Apply the radial and tangential distortion parameters from the lens parameters to correct the normalized coordinates and obtain the corrected normalized coordinates.

[0078] The radial distortion parameter is the radial distortion coefficient mentioned above. The tangential distortion parameter is the tangential distortion coefficient mentioned above. .

[0079] The correction amounts for radial and tangential distortions are calculated to correct the normalized coordinates. The formula for calculating the correction amounts is shown in equation (4).

[0080] Equation (4) in, The squared distance from the origin to the normalized coordinates.

[0081] The correction amount, normalized coordinates, radial distortion parameters, and tangential distortion parameters are calculated to obtain the corrected normalized coordinates. Correcting normalized coordinates The formula for expressing this is as follows: Equation (5) Equation (6) S303. Convert the normalized coordinates to pixel coordinates to obtain the lens distortion correction strategy.

[0082] Correct the normalized coordinates Convert to pixel coordinates to obtain lens distortion correction coordinates. Lens distortion correction coordinates The formula for expressing this is as follows: Equation (7) Equation (8) The obtained lens distortion correction strategy includes the expression formula of lens distortion correction coordinates. It should be noted that the expression formula of lens distortion correction coordinates can be applied to any pixel that needs to be corrected for lens distortion. The corresponding lens distortion correction coordinates can be obtained by simply substituting the relevant values ​​of the pixel into equations (7) and (8).

[0083] In the embodiments provided in this application, the panoramic camera lens has inherent optical distortions, namely radial distortion and tangential distortion. Directly performing motion compensation on the lens would lead to compensation deviations. Therefore, distortion correction of the lens imaging is necessary first. Preferably, the correction process only targets the pixel coordinates required for motion compensation, i.e., the target pixel coordinates mentioned above. When performing distortion correction, a lens distortion correction strategy is applied to reduce motion compensation deviations, improve the accuracy of motion compensation, and enhance the quality of the subsequently obtained panoramic video and images.

[0084] S203. Process the lens distortion correction strategy and the independent motion information of each lens to obtain the motion compensation strategy for each lens.

[0085] Reference Figure 4 The following is a flowchart of a method for obtaining motion compensation strategies for each shot according to another embodiment of this application, which is described in detail below.

[0086] S401. Extract the two-dimensional components of the rotation matrix from the independent motion information of each shot to obtain the two-dimensional rotation matrix of each shot.

[0087] For each shot, a rotation matrix derived from the independent motion information of that shot. Extracting the two-dimensional components yields the two-dimensional rotation matrix of the lens. Preferably, the Z-axis effect is ignored when extracting the two-dimensional components, since the jitter mainly affects the two-dimensional rotation matrix of the image plane lens. The expression for is shown in equation (9).

[0088] Equation (9) in, for The first two rows of elements; for The first two columns of elements correspond to the rotation angle of the image plane.

[0089] S402. Convert the translation component in the independent motion information of each shot from three dimensions to two dimensions to obtain the two-dimensional translation component of each shot.

[0090] For each shot, the translation component in the shot's independent motion information. Convert to translation components in pixel coordinates, that is, translation components From 3D to 2D, we obtain the 2D translation components. Two-dimensional translation components The expressions are shown in equations (10) and (11). Equation (10) Equation (11) Z0 is the average object distance captured by the panoramic camera. This average object distance is a preset value, ranging from 0.5m to 5m. It can be dynamically adjusted according to the scene to balance the translation compensation range and avoid over- or under-compensation.

[0091] S403. Construct a matrix using the two-dimensional rotation matrix and two-dimensional translation component of each lens to obtain the initial affine transformation matrix of each lens.

[0092] For each shot, use the two-dimensional rotation matrix of that shot. and two-dimensional translation components Construct a matrix to obtain the initial affine transformation matrix of the lens. Initial affine transformation matrix The expression is as shown in equation (12).

[0093] Equation (12) Furthermore, an initial affine transformation matrix can be used. Lens distortion calibration coordinates Motion compensation is performed on the pixels to obtain their compensated coordinates. Pixel-compensated coordinates The expression is shown in equations (13) and (14).

[0094] Equation (13) Equation (14) Furthermore, expanding the initial affine transformation matrix yields equations (13) and (14). Motion compensation for the lens pixels can be performed using the initial affine transformation matrix and lens distortion calibration strategy. To improve the accuracy of motion compensation, the calibrated affine transformation matrix is ​​used to perform motion compensation for the pixels, thereby counteracting the rotational and translational shifts caused by jitter.

[0095] Since each lens has a slight installation deviation, directly using the initial affine transformation matrix for motion compensation will result in errors. Therefore, it is necessary to calibrate the initial affine transformation matrix and use the calibrated affine transformation matrix for motion compensation. The steps for calibrating the initial affine transformation matrix are shown in S404.

[0096] S404. Obtain IMU dynamic zero-bias calibration data, and use the IMU dynamic zero-bias calibration data to calibrate each initial affine transformation matrix to obtain the affine transformation matrix of each lens.

[0097] The IMU dynamic zero-bias calibration data obtained here is the same as the IMU dynamic zero-bias calibration data described in the S101 section above; that is, this IMU dynamic zero-bias calibration data is the calibration result tracked and output during zero-bias calibration of the IMU sensor. Furthermore, the IMU dynamic zero-bias calibration data here... The data is in matrix form, and can also be called the IMU dynamic zero bias calibration deviation matrix.

[0098] For each shot, the initial affine transformation matrix for that shot After calibration, the affine transformation matrix of the lens is obtained. In other words, the affine transformation matrix is ​​the calibrated matrix. The expression for is shown in equation (15).

[0099] Equation (15) in, I is the calibrated affine transformation matrix; I is a 3×3 identity matrix; This is for IMU dynamic zero-bias calibration data. An affine transformation matrix is ​​used. Pixels can be compensated, and the matrix can be expanded to obtain the coordinates of the compensated pixels.

[0100] Furthermore, the obtained affine transformation matrix can be understood as the affine transformation matrix of the lens at the current moment, and this affine transformation matrix is ​​subsequently used to compensate the image acquired by the lens at the current moment.

[0101] By calibrating the initial affine transformation matrix, the compensation error caused by the installation deviation can be eliminated. The resulting affine transformation matrix contains rotation and translation components, which can be directly used for orientation compensation of image pixels to counteract the effects of jitter.

[0102] S405. Based on the lens distortion correction strategy and the affine transformation matrix of each lens, the motion compensation strategy for each lens is obtained.

[0103] It should be noted that for each lens, the lens distortion calibration coordinates are combined with the affine transformation matrix of the lens to obtain the coordinates of the pixel after motion compensation. The calculation process of the coordinates of the pixel after motion compensation can be referred to equations (13) and (14).

[0104] The generated motion compensation strategy includes a lens distortion correction strategy and an affine transformation matrix for the lens.

[0105] In the method provided in this application embodiment, based on the overall motion trajectory of the fused panoramic camera and the relative motion trajectory of each lens, and combined with the shooting parameters (focal length, angle of view) of each lens, the affine transformation matrix (including rotation, translation, and scaling) that needs to be compensated for each frame of image of each lens is calculated separately. This ensures that the compensation direction and magnitude of each lens are synchronized. The calculation of the compensation matrix needs to be combined with the lens distortion parameters to eliminate the influence of optical distortion on the compensation effect. At the same time, it adapts to the circular image field characteristics of the square CMOS to improve the compensation accuracy and effective imaging area utilization. The transformation matrix of each lens can be derived through matrix operations to achieve synchronous compensation of multiple lenses.

[0106] In the solution provided in this application embodiment, a series of processes are performed on lens parameters, motion information, etc., to obtain a lens distortion correction strategy and an initial affine transformation matrix for each lens. The initial affine transformation matrix of each lens is then corrected to obtain a corrected affine transformation matrix for each lens. Using the lens distortion correction strategy and the corrected affine transformation matrix to perform motion compensation on pixels can effectively eliminate the influence of lens distortion. Correcting the affine transformation matrix can eliminate lens installation errors. Using the corrected affine transformation matrix to perform motion compensation on pixels further improves the accuracy of compensation and enhances the quality of subsequent panoramic images.

[0107] In another embodiment provided in this application, when using the motion compensation strategy of each lens to perform multi-lens compensation on the image of each lens, there are three main operations: (1) single-lens frame-by-frame compensation; (2) multi-lens synchronous control; (3) compensation effect verification. It should be noted that single-lens frame-by-frame compensation and multi-lens synchronous control are performed simultaneously, and compensation effect verification is performed when all lenses have completed compensation.

[0108] Reference Figure 5 The following is an example diagram illustrating the application of motion compensation strategies for each lens in this embodiment of the application, which performs multi-lens compensation on the images of each lens in one frame. The relevant explanation is as follows.

[0109] The single-shot frame-by-frame compensation shown in the figure specifically includes: for each shot's image, using the shot's motion compensation strategy to compensate each pixel in the image one by one. Furthermore, for each shot, using the shot's motion compensation strategy, each pixel of the shot's image is compensated individually.

[0110] This application provides a specific example of frame-by-frame compensation for a single lens, using the k-th frame of the i-th lens as an example for illustration. For details, please refer to S501-S503.

[0111] S501, Traverse all pixels of the image. For each pixel in the image, the lens distortion correction strategy in the motion compensation strategy is applied to correct the distortion of the pixel, and the distortion correction coordinates corresponding to the pixel are obtained.

[0112] It should be noted that distortion correction is performed on each pixel in the image one by one. For each pixel, equations (7) and (8) in the lens distortion correction strategy are used to complete the distortion correction of each pixel, thereby obtaining the distortion correction coordinates of each pixel. .

[0113] S502. Process the distortion correction coordinates of the pixel and the affine transformation matrix in the motion compensation strategy to obtain the compensated coordinates of the pixel.

[0114] For each pixel, the affine transformation matrix and the pixel's distortion correction coordinates from the motion compensation strategy are used for processing. For example, the pixel's distortion correction coordinates... Substitute the affine transformation matrix in the motion compensation strategy The expanded formula can be used to calculate the compensated coordinates of the pixel, i.e., the compensated coordinates of the pixel. .

[0115] S503. When the compensated coordinates of a pixel are integer coordinates, the pixel compensation operation ends; when the compensated coordinates of a pixel are non-integer coordinates, the bilinear interpolation algorithm is used to fill the grayscale values ​​of the compensated coordinates of the pixel, and the pixel compensation operation ends.

[0116] For each pixel, determine if its compensated coordinates are integers. If they are integers, the compensation operation ends, and the compensated coordinates are directly output. If the compensated coordinates are non-integers, use a bilinear interpolation algorithm to fill in the grayscale values, and then end the compensation operation.

[0117] When the compensated coordinates of a pixel are non-integer coordinates, the compensated coordinates of the pixel are filled with gray values ​​before output, thereby avoiding jagged edges and blurriness in the compensated image.

[0118] For example, assume the compensated coordinates for: ,in, Integer coordinates Then the pixel grayscale value As shown in equation (16).

[0119]

[0120] in, The values ​​are the pixel grayscale values ​​corresponding to integer coordinates in the image after distortion correction. This interpolation algorithm ensures the clarity of the compensated image, adapting to the high-resolution shooting requirements of panoramic cameras.

[0121] Because there is a slight time difference in the acquisition of images by multiple lenses in a panoramic camera, if each lens is compensated independently, the motion state of the images will be out of sync, which will lead to stitching misalignment. Therefore, a synchronization control mechanism, namely multi-lens synchronization control, needs to be introduced.

[0122] The multi-lens synchronous control includes: for images from each lens undergoing compensation, real-time acquisition of the compensation progress of each lens; pairing each lens in pairs to obtain lens combinations, determining the compensation progress deviation of each lens combination, where the compensation progress deviation is the absolute value of the difference between the compensation progress of the two lenses within the lens combination; when the compensation progress deviation of any lens combination exceeds a preset progress deviation threshold, reducing the compensation rate of the lens with the larger compensation progress in that lens combination, continuously monitoring the compensation progress deviation of that lens combination, and stopping the adjustment when it is less than or equal to the preset progress deviation threshold.

[0123] This application provides an example of one-to-many lens synchronization control, and the relevant process is described in S601-S603.

[0124] S601: Real-time acquisition of compensation progress for each lens.

[0125] Define compensation schedule Let be the ratio of the number of pixels that have been compensated for the i-th lens to the total number of pixels. The calculation formula is shown in Equation (17).

[0126] Equation (17) in, Let be the number of pixels for which compensation has been completed for the i-th lens. This represents the total number of pixels in the lens.

[0127] S602. Calculate the compensation progress deviation between each lens.

[0128] Use formula (18) to calculate the compensation progress deviation between each shot. .

[0129] Equation (18) in, Let i be the compensation progress for the i-th shot. Let j be the compensation progress for the j-th shot, where j ≠ i.

[0130] S603, Preset schedule deviation threshold ,when At that time, reduce the compensation rate of lenses with fast compensation progress until... Stop adjusting.

[0131] Methods to reduce the compensation rate of lenses with fast compensation progress include, but are not limited to, reducing the processing speed of that lens and reducing the pixel traversal frequency. S601-S603 ensure that the compensation progress of all lenses is synchronized, with the compensation completion time difference controlled within ±1ms.

[0132] In another embodiment provided in this application, after the images of each lens at the current moment have been compensated, the compensation effect needs to be verified in real time. Further, it can be understood that after each frame of multi-lens synchronous compensation is completed, the compensation effect is verified in real time, thereby avoiding compensation failure, that is, the compensation effect needs to be verified in real time.

[0133] The real-time verification of the compensation effect includes the following steps: After the images of each lens are compensated, the residual jitter is obtained based on the compensated images of each lens; when the residual jitter is not greater than the preset residual jitter threshold, the compensated images of each lens at the current moment are output; when the residual jitter is greater than the preset residual jitter threshold, the IMU dynamic zero-bias calibration frequency is adjusted, and the motion compensation strategy for each lens is reacquired. The reacquired motion compensation strategies are used to perform one-frame multi-lens synchronous compensation on the images of each lens again until the residual jitter is not greater than the preset residual jitter threshold, at which point the compensated images of each lens at the current moment are output.

[0134] This application provides an example of real-time verification of compensation effect, and the relevant process is described in S701-S702.

[0135] S701. Select the central reference region of the compensated image and calculate the variance of the pixel grayscale values ​​in that region. .

[0136] The size of the central reference area can be 200×200 pixels to avoid edge distortion areas.

[0137] Calculate the variance of pixel grayscale values ​​using equation (19). .

[0138] Equation (19) in, For the reference area pixel count, Let g be the gray value of the k-th pixel within the reference region, and g be the average gray value of the reference region.

[0139] S702, Preset jitter residual threshold ,like This indicates that significant jitter still exists after compensation, indicating that the compensation has failed. In this case, the affine transformation matrix of the lens corresponding to that frame is recalculated, and the IMU dynamic zero-bias calibration frequency is increased until... To ensure that the compensation achieves the desired effect; if Once the compensation is deemed valid, the process proceeds to the subsequent cutting and splicing stages.

[0140] Preferably, if Increase the IMU dynamic zero-bias calibration frequency and recalculate the affine transformation matrix for each lens, update the motion compensation strategy for each lens, and then re-perform synchronization compensation using the motion compensation strategy for each lens until... The output image is compensated. Increasing the IMU dynamic zero-bias calibration frequency can reduce the number of preset frame intervals.

[0141] In the embodiments provided in this application, multi-lens synchronous image stabilization is achieved, which solves the stitching misalignment problem caused by asynchronous movement of multiple lenses in a panoramic camera.

[0142] The solution provided in this application performs geometric transformation on the image of each lens and, based on the corresponding affine transformation matrix, reverses the image shift caused by panoramic camera shake. During the compensation process, the compensation progress of each lens is synchronized in real time to ensure that the motion state of the multi-lens images is consistent and to avoid stitching misalignment caused by asynchronous compensation. Targeted compensation optimization is performed on the overlapping area to ensure the image alignment accuracy of the overlapping area, reduce stitching gaps, and unify the compensated images of each lens to the same coordinate system through the transformation matrix, thereby improving stitching consistency.

[0143] Although this invention describes various operations in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0144] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0145] and Figure 1 In contrast to the method shown, embodiments of the present invention provide a panoramic video stabilization device, which is used to support... Figure 1 The method shown is applied in practice; this device is used in computing processing equipment or computers. A schematic diagram of the device is shown below. Figure 6 As shown, the specific explanation is as follows.

[0146] The first acquisition unit 801 is used to acquire motion information of the panoramic camera corresponding to the current time and images of each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens. The second acquisition unit 802 is used to acquire the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens. The synchronization compensation unit 803 is used to apply the motion compensation strategy of each of the lenses to perform one-frame multi-lens synchronization compensation on the image of each of the lenses, so as to obtain the compensated image of each of the lenses at the current time. The stitching unit 804 is used to crop and stitch the compensation images of each of the lenses at different times based on the camera sensor type of each lens, and output a stabilized panoramic video.

[0147] The apparatus provided in this application, when stabilizing panoramic video, fuses the motion information of the panoramic camera with the images from each lens to avoid a disconnect between the stabilization compensation and the stitching process, thereby improving the accuracy and robustness of stabilization. Furthermore, it simultaneously performs motion compensation on multiple lenses to ensure synchronized movement of the images from multiple lenses, preventing ghosting and discontinuities in the stitched images, thus meeting the shooting needs of panoramic cameras in complex scenes. Further, when cropping and stitching the compensated images, the impact of the lens's camera sensor type on video stabilization is considered, improving the adaptability of this solution and reducing power consumption, thereby enhancing the stability and stitching quality of panoramic video and images in various shooting scenarios.

[0148] In another embodiment of the apparatus provided in this application, the first acquisition unit performs a process of acquiring motion information of the panoramic camera corresponding to the current time and images from each lens of the panoramic camera, including: The IMU data of the panoramic camera at the current moment is acquired through the IMU sensor, and the images of each of the lenses at the current moment are also acquired. The IMU data is denoised, the IMU sensor is calibrated, and IMU dynamic zero bias calibration data is output. The denoised IMU data and the images of each of the lenses are timestamped together. The motion trajectory of the panoramic camera is obtained by integrating the denoised IMU data. Based on a preset IMU dynamic zero-bias calibration frequency, the motion trajectory of the panoramic camera is corrected using the IMU dynamic zero-bias calibration data. Based on the corrected motion trajectory and the lens layout parameters of the panoramic camera, the short-term motion estimation result of the panoramic camera's IMU is obtained. The short-term motion estimation result of the panoramic camera includes the overall motion parameters of the panoramic camera and the relative motion deviation between each lens. The short-term motion estimation results of the IMU are processed using lightweight Kalman filtering and the IMU dynamic zero-bias calibration data to obtain the motion information of the panoramic camera. The motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens.

[0149] In another embodiment of the apparatus provided in this application, the second acquisition unit performs a process of obtaining motion compensation strategies for each lens based on the lens parameters and the motion information, including: The overall motion information and the relative motion information between each shot in the motion information are decomposed to obtain the independent motion information of each shot. The independent motion information includes the rotation matrix and translation vector of the shot. The lens parameters and target pixel coordinates are processed to obtain a lens distortion correction strategy, wherein the target pixel coordinates are the pixel coordinates of the pixels in the panoramic camera that participate in the panoramic image motion mapping calculation. The lens distortion correction strategy and the independent motion information of each lens are processed to obtain the motion compensation strategy for each lens.

[0150] In another embodiment of the apparatus provided in this application, the second acquisition unit performs a process of processing the lens parameters and target pixel coordinates to obtain a lens distortion correction strategy, including: By applying the lens focal length from the lens parameters, the target pixel coordinates are converted into coordinates in the camera coordinate system to obtain normalized coordinates; The radial distortion parameter and tangential distortion parameter in the lens parameters are used to correct the normalized coordinates to obtain the corrected normalized coordinates; The corrected normalized coordinates are converted into coordinates in the pixel coordinate system to obtain the lens distortion correction strategy.

[0151] In another embodiment of the apparatus provided in this application, the second acquisition unit performs a process of processing the lens distortion correction strategy and the independent motion information of each lens to obtain the motion compensation strategy for each lens, including: Extract the two-dimensional components of the rotation matrix from the independent motion information of each lens to obtain the two-dimensional rotation matrix of each lens; The translation component in the independent motion information of each lens is transformed from three dimensions to two dimensions to obtain the two-dimensional translation component of each lens; A matrix is ​​constructed by applying the two-dimensional rotation matrix and two-dimensional translation component of each lens to obtain the initial affine transformation matrix of each lens; Acquire IMU dynamic zero-bias calibration data, and use the IMU dynamic zero-bias calibration data to calibrate each of the initial affine transformation matrices to obtain the affine transformation matrix of each lens; Based on the lens distortion correction strategy and the affine transformation matrix of each lens, a motion compensation strategy for each lens is obtained.

[0152] In another embodiment of the apparatus provided in this application, the process by which the synchronization compensation unit executes a motion compensation strategy applied to each of the lenses, performs one-frame multi-lens synchronization compensation on the images of each of the lenses, and obtains the compensated images of each of the lenses at the current moment includes: For each image from the lens, the motion compensation strategy of the lens is used to perform a compensation operation on each pixel in the image one by one; For each lens image undergoing compensation, the compensation progress of each lens is acquired in real time; each lens is paired to obtain a lens combination, and the compensation progress deviation of each lens combination is determined. The compensation progress deviation is the absolute value of the difference between the compensation progress of the two lenses in the lens combination; when the compensation progress deviation of any lens combination is greater than a preset progress deviation threshold, the compensation rate of the lens with the larger compensation progress in the lens combination is reduced, and the compensation progress deviation of the lens combination is continuously monitored until it is less than or equal to the preset progress deviation threshold, at which point the adjustment stops. After the images of each lens are compensated, the residual jitter is obtained based on the compensated images of each lens. When the residual jitter is not greater than a preset residual jitter threshold, the compensated images of each lens at the current time are output. When the residual jitter is greater than the preset residual jitter threshold, the IMU dynamic zero-bias calibration frequency is adjusted, and the motion compensation strategy of each lens is reacquired. The reacquired motion compensation strategies are used to perform one-frame multi-lens synchronous compensation on the images of each lens again until the residual jitter is not greater than the preset residual jitter threshold, and then the compensated images of each lens at the current time are output.

[0153] In another embodiment of the apparatus provided in this application, the synchronization compensation unit performs a process of compensating each pixel in the image one by one using the motion compensation strategy of the lens, including: For each pixel in the image, the lens distortion correction strategy in the motion compensation strategy is applied to correct the distortion of the pixel to obtain the distortion correction coordinates corresponding to the pixel. The distortion correction coordinates of the pixel and the affine transformation matrix in the motion compensation strategy are processed to obtain the compensated coordinates of the pixel; When the compensated coordinates of the pixel are integer coordinates, the compensation operation for the pixel ends; when the compensated coordinates of the pixel are non-integer coordinates, a bilinear interpolation algorithm is used to fill the grayscale values ​​of the compensated coordinates of the pixel, and the compensation operation for the pixel ends.

[0154] The various units and functions in the device provided in the embodiments of the present invention can be referred to the relevant descriptions in the embodiments above, and will not be repeated here.

[0155] This invention also provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device containing the storage medium is controlled to perform the above-described panoramic video stabilization method.

[0156] This invention also provides an electronic device, the structural schematic of which is shown below. Figure 7As shown, it specifically includes a memory 901 and one or more instructions 902, wherein one or more instructions 902 are stored in the memory 901 and configured to be executed by one or more processors 903 to perform the following operations: Acquire motion information of the panoramic camera corresponding to the current moment and images from each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens; Obtain the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens; By applying the motion compensation strategy of each of the aforementioned shots, multi-shot synchronous compensation is performed on the image of each of the aforementioned shots in one frame to obtain the compensated image of each of the aforementioned shots at the current moment. Based on the camera sensor type of each lens, the compensation images of each lens at different times are cropped and stitched together to output a stabilized panoramic video.

[0157] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions.

[0158] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of this invention.

[0159] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0161] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A panoramic video stabilization method, characterized in that, include: Acquire motion information of the panoramic camera corresponding to the current moment and images from each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens; Obtain the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens; By applying the motion compensation strategy of each of the aforementioned shots, multi-shot synchronous compensation is performed on the image of each of the aforementioned shots in one frame to obtain the compensated image of each of the aforementioned shots at the current moment. Based on the camera sensor type of each lens, the compensation images of each lens at different times are cropped and stitched together to output a stabilized panoramic video.

2. The method according to claim 1, characterized in that, The acquisition of motion information of the panoramic camera corresponding to the current moment and images from each lens of the panoramic camera includes: The IMU data of the panoramic camera at the current moment is acquired through the IMU sensor, and the images of each of the lenses at the current moment are also acquired. The IMU data is denoised, the IMU sensor is calibrated, and IMU dynamic zero bias calibration data is output. The denoised IMU data and the images of each of the lenses are timestamped together. The motion trajectory of the panoramic camera is obtained by integrating the denoised IMU data. Based on a preset IMU dynamic zero-bias calibration frequency, the motion trajectory of the panoramic camera is corrected using the IMU dynamic zero-bias calibration data. Based on the corrected motion trajectory and the lens layout parameters of the panoramic camera, the short-term motion estimation result of the panoramic camera's IMU is obtained. The short-term motion estimation result of the panoramic camera includes the overall motion parameters of the panoramic camera and the relative motion deviation between each lens. The short-term motion estimation results of the IMU are processed using lightweight Kalman filtering and the IMU dynamic zero-bias calibration data to obtain the motion information of the panoramic camera. The motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens.

3. The method according to claim 1, characterized in that, The motion compensation strategy for each lens, based on the lens parameters and the motion information, includes: The overall motion information and the relative motion information between each shot in the motion information are decomposed to obtain the independent motion information of each shot. The independent motion information includes the rotation matrix and translation vector of the shot. The lens parameters and target pixel coordinates are processed to obtain a lens distortion correction strategy, wherein the target pixel coordinates are the pixel coordinates of the pixels in the panoramic camera that participate in the panoramic image motion mapping calculation. The lens distortion correction strategy and the independent motion information of each lens are processed to obtain the motion compensation strategy for each lens.

4. The method according to claim 3, characterized in that, The process of processing the lens parameters and target pixel coordinates to obtain a lens distortion correction strategy includes: By applying the lens focal length from the lens parameters, the target pixel coordinates are converted into coordinates in the camera coordinate system to obtain normalized coordinates; The radial distortion parameter and tangential distortion parameter in the lens parameters are used to correct the normalized coordinates to obtain the corrected normalized coordinates; The corrected normalized coordinates are converted into coordinates in the pixel coordinate system to obtain the lens distortion correction strategy.

5. The method according to claim 3, characterized in that, The process of processing the lens distortion correction strategy and the independent motion information of each lens to obtain the motion compensation strategy for each lens includes: Extract the two-dimensional components of the rotation matrix from the independent motion information of each lens to obtain the two-dimensional rotation matrix of each lens; The translation component in the independent motion information of each lens is transformed from three dimensions to two dimensions to obtain the two-dimensional translation component of each lens; A matrix is ​​constructed by applying the two-dimensional rotation matrix and two-dimensional translation component of each lens to obtain the initial affine transformation matrix of each lens; Acquire IMU dynamic zero-bias calibration data, and use the IMU dynamic zero-bias calibration data to calibrate each of the initial affine transformation matrices to obtain the affine transformation matrix of each lens; Based on the lens distortion correction strategy and the affine transformation matrix of each lens, a motion compensation strategy for each lens is obtained.

6. The method according to claim 5, characterized in that, The motion compensation strategy applied to each of the aforementioned shots performs one-frame multi-shot synchronous compensation on the images of each of the aforementioned shots to obtain the compensated images of each of the aforementioned shots at the current moment, including: For each image from the aforementioned lens, the motion compensation strategy of the lens is used to compensate each pixel in the image one by one; For each lens image undergoing compensation, the compensation progress of each lens is acquired in real time; each lens is paired to obtain a lens combination, and the compensation progress deviation of each lens combination is determined. The compensation progress deviation is the absolute value of the difference between the compensation progress of the two lenses in the lens combination; when the compensation progress deviation of any lens combination is greater than a preset progress deviation threshold, the compensation rate of the lens with the larger compensation progress in the lens combination is reduced, and the compensation progress deviation of the lens combination is continuously monitored until it is less than or equal to the preset progress deviation threshold, at which point the adjustment stops. After the images of each lens are compensated, the residual jitter is obtained based on the compensated images of each lens. When the residual jitter is not greater than a preset residual jitter threshold, the compensated images of each lens at the current time are output. When the residual jitter is greater than the preset residual jitter threshold, the IMU dynamic zero-bias calibration frequency is adjusted, and the motion compensation strategy of each lens is reacquired. The reacquired motion compensation strategies are used to perform one-frame multi-lens synchronous compensation on the images of each lens again until the residual jitter is not greater than the preset residual jitter threshold, and then the compensated images of each lens at the current time are output.

7. The method according to claim 6, characterized in that, The motion compensation strategy using the lens compensates for each pixel in the image individually, including: For each pixel in the image, the lens distortion correction strategy in the motion compensation strategy is applied to correct the distortion of the pixel to obtain the distortion correction coordinates corresponding to the pixel. The distortion correction coordinates of the pixel and the affine transformation matrix in the motion compensation strategy are processed to obtain the compensated coordinates of the pixel; When the compensated coordinates of the pixel are integer coordinates, the compensation operation for the pixel ends; when the compensated coordinates of the pixel are non-integer coordinates, a bilinear interpolation algorithm is used to fill the grayscale values ​​of the compensated coordinates of the pixel, and the compensation operation for the pixel ends.

8. A panoramic video stabilization device, characterized in that, include: The first acquisition unit is used to acquire motion information of the panoramic camera corresponding to the current time and images from each lens of the panoramic camera; the motion information includes the overall motion information of the panoramic camera and the relative motion information between each lens. The second acquisition unit is used to acquire the lens parameters of the panoramic camera, and based on the lens parameters and the motion information, obtain the motion compensation strategy for each lens. The synchronization compensation unit is used to apply the motion compensation strategy of each of the lenses to perform one-frame multi-lens synchronization compensation on the image of each of the lenses, so as to obtain the compensated image of each of the lenses at the current time. The stitching unit is used to crop and stitch the compensated images of each of the lenses at different times based on the camera sensor type of each lens, and output a stabilized panoramic video.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device containing the storage medium is controlled to perform the panoramic video stabilization method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors to implement the panoramic video stabilization method as described in any one of claims 1-7.

Images