IMU-based camera motion compensation method and device and storage medium

Abstract

The invention relates to the technical field of camera motion compensation, and discloses an IMU-based camera motion compensation method and device and a storage medium, and the method comprises the steps: processing original data collected by a plurality of IMU sensors, and obtaining six-degree-of-freedom IMU fusion data; processing through a linear convolution aliasing model to obtain a high-dimensional motion feature vector; performing fusion to obtain a camera motion parameter prediction value; constructing an advanced network compensation model, carrying out jitter intensity evaluation, and dynamically adjusting a compensation model parameter matrix; performing geometric transformation to obtain an initial stable image sequence and a foreground mask; according to the method, IMU data and image information can be effectively fused, camera motion parameters can be accurately estimated, and a compensation strategy can be dynamically adjusted according to scene features to realize camera motion compensation.

Description

Technical Field

[0001] The present invention relates to the technical field of camera motion compensation, and in particular to an IMU-based camera motion compensation method, device and storage medium. Background Art

[0002] Camera stability is a key factor affecting image and video quality. Traditional optical image stabilization and digital image stabilization technologies often fail to meet the needs of high-quality image stabilization in complex motion scenes. In recent years, camera motion compensation methods based on inertial measurement units (IMUs) have attracted widespread attention due to their fast response and high precision.

[0003] However, IMU data has problems such as noise and drift, and it is often difficult to achieve ideal results by directly using IMU data for motion compensation. At the same time, traditional motion compensation algorithms have difficulty accurately capturing the six-degree-of-freedom motion of the camera when dealing with complex motion scenes, resulting in poor compensation results. In addition, existing compensation methods often ignore the semantic information of the image content, making it difficult to retain important foreground information while maintaining image stability. Summary of the invention

[0004] The present invention provides an IMU-based camera motion compensation method, device and storage medium. The present invention can effectively fuse IMU data and image information, accurately estimate camera motion parameters, and dynamically adjust the compensation strategy according to scene characteristics to achieve camera motion compensation.

[0005] In a first aspect, the present invention provides a camera motion compensation method based on an IMU, and the camera motion compensation method based on an IMU comprises: The raw data collected by multiple IMU sensors are processed by low-pass filtering and adaptive Kalman filtering to obtain six-degree-of-freedom IMU fusion data; Inputting the six-degree-of-freedom IMU fusion data into a linear convolution aliasing model for processing to obtain simulated motion data, and extracting the corresponding high-dimensional motion feature vector; According to the high-dimensional motion feature vector and the original image sequence, a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model are used for fusion to obtain a camera motion parameter prediction value; Building a look-ahead network compensation model based on the camera motion parameters, and dynamically adjusting the compensation model parameter matrix according to the jitter intensity evaluation results of the IMU data variance and peak analysis; Performing geometric transformation on the original image data based on the compensation model parameter matrix, and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and a foreground mask; Content-aware filling and time-domain filtering are performed on the initial stabilized image sequence and the foreground mask, and target stabilized image sequences in multiple compensation modes are output.

[0006] In a second aspect, the present invention provides an IMU-based camera motion compensation device, the IMU-based camera motion compensation device comprising: The acquisition module is used to perform low-pass filtering and adaptive Kalman filtering on the raw data collected by multiple IMU sensors to obtain six-degree-of-freedom IMU fusion data; An extraction module is used to input the six-degree-of-freedom IMU fusion data into a linear convolution aliasing model for processing to obtain simulated motion data and extract the corresponding high-dimensional motion feature vector; A fusion module is used to fuse the high-dimensional motion feature vector and the original image sequence using a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model to obtain a camera motion parameter prediction value; An evaluation module, used to construct an advance network compensation model based on the camera motion parameters, and dynamically adjust the compensation model parameter matrix according to the jitter intensity evaluation results of the IMU data variance and peak analysis; A transformation module, used for geometrically transforming the original image data based on the compensation model parameter matrix, and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and a foreground mask; The output module is used to perform content-aware filling and time-domain filtering on the initial stabilized image sequence and the foreground mask, and output a target stabilized image sequence in multiple compensation modes.

[0007] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned IMU-based camera motion compensation method.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned IMU-based camera motion compensation method.

[0009] In the technical solution provided by the present invention, by performing low-pass filtering and adaptive Kalman filtering on multiple IMU sensor data, the noise and drift of IMU data are effectively reduced, and the accuracy and reliability of motion data are improved. The linear convolution aliasing model is used to process IMU fusion data, which can better capture complex motion patterns, extract high-dimensional motion feature vectors, and provide richer information for subsequent motion parameter prediction. Combined with the six-degree-of-freedom motion model based on sine wave combination and the BP neural network prediction model, accurate prediction of camera motion parameters is achieved, effectively solving the problem that traditional methods are difficult to accurately describe complex motion. The advanced network compensation model and dynamic adjustment mechanism are introduced to adjust the compensation strategy in real time according to the jitter intensity evaluation results of IMU data, thereby improving the flexibility and adaptability of compensation. Combined with dense optical flow field optimization and time-weighted background pixel pool modeling, while performing image stabilization, foreground information is effectively retained, and the visual quality of the stabilization effect is improved. Through content-aware filling and time-domain filtering processing, the edge blanks and time-domain incoherence that may occur during image stabilization are solved, further improving the quality of the stabilized image sequence. It provides multiple compensation modes (global compensation, local compensation, and adaptive compensation), and dynamically selects the optimal mode through jitter evaluation to meet the stability requirements in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0011] Figure 1 Schematic diagram of the steps of the IMU-based camera motion compensation method in an embodiment of the present invention; Figure 2 Schematic diagram of the structure of an IMU-based camera motion compensation device in an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] Embodiments of the present invention provide a method, device and storage medium for camera motion compensation based on IMU. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0013] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the camera motion compensation method based on IMU in an embodiment of the present invention includes: Step S1, low-pass filtering and adaptive Kalman filtering are performed on the raw data collected by multiple IMU sensors to obtain six-degree-of-freedom IMU fusion data; It is understandable that the execution subject of the present invention may be an IMU-based camera motion compensation device, or a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as an execution subject as an example.

[0014] Specifically, the acceleration and angular velocity raw data collected by each IMU sensor are processed by digital low-pass filtering, and the high-frequency noise components are removed by filtering, and smoother motion data is extracted to obtain the first IMU data. The first IMU data is subjected to zero bias correction and temperature compensation. Zero bias correction can eliminate the error caused by the internal bias of the sensor, while temperature compensation is used to correct the measurement error caused by the drift of the sensor under different temperature environments, so as to obtain more accurate second IMU data. The second IMU data is input into an adaptive Kalman filter. The adaptive Kalman filter is a dynamic filter that can adjust the process noise covariance matrix in real time according to the current motion state to adaptively optimize the filtering result. In this way, the filter can effectively cope with the noise changes under different motion modes, improve the accuracy of the filtered data, and obtain the third IMU data. The third IMU data is subjected to Allan variance analysis, and the noise characteristic parameters of each IMU sensor are calculated, including indicators such as random walk and offset instability, and a noise parameter matrix is ​​constructed through these indicators. Based on the noise parameter matrix, the third IMU data of multiple IMU sensors are weighted fused. The weighted fusion process assigns weights according to the noise characteristics of each IMU sensor, giving priority to sensor data with lower noise, improving the credibility and accuracy of the overall data, and obtaining the fused fourth IMU data. The attitude of the fourth IMU data is solved, and the rotation matrix of the camera is calculated by the quaternion algorithm. Quaternions can avoid the singularity that occurs in the traditional Euler angle representation (such as the gimbal lock problem), and the numerical stability of quaternions also helps to maintain the continuity of the attitude and more accurately describe the attitude change of the camera. The camera attitude data is combined with the fourth IMU data and fused through the complementary filtering algorithm to enhance the stability and accuracy of the attitude solution. Complementary filtering is a lightweight fusion method that combines high-frequency response acceleration data and low-frequency response gyroscope data to effectively filter out long-term drift errors in attitude changes while ensuring dynamic response, and obtain corrected six-degree-of-freedom motion data. The corrected six-degree-of-freedom motion data is time synchronized and interpolated to keep it consistent with the camera frame rate. The data acquisition frequency of the IMU sensor is usually much higher than the frame rate of the camera, so time synchronization is required to ensure that the motion data corresponds to the image sequence collected by the camera. During the time synchronization process, interpolation processing is used to fill the difference between different sampling frequencies, so that accurate motion state information can be obtained at the camera sampling moment. After processing, the final result is the six-degree-of-freedom IMU fusion data aligned with the camera frame rate.

[0015] Step S2: input the six-degree-of-freedom IMU fusion data into the linear convolution aliasing model for processing to obtain simulated motion data, and extract the corresponding high-dimensional motion feature vector; Specifically, the linear convolutional aliasing model consists of three one-dimensional convolutional layers, two maximum pooling layers, and two fully connected layers. Among them, the first convolutional layer uses 32 one-dimensional convolutional kernels of size 5 with a step size of 1 to capture the motion features in the input data in a short time; the second convolutional layer uses 64 one-dimensional convolutional kernels of size 3 with a step size of 1 to extract deeper motion patterns and detailed features; the third convolutional layer uses 128 one-dimensional convolutional kernels of size 3 with a step size of 1 to enrich the feature representation of the data in a higher dimension. In order to adapt the six-degree-of-freedom IMU fusion data to the processing of the convolutional network, the data is divided into time windows to obtain multiple sub-signal sequences. The sub-signal sequence is input into the first one-dimensional convolutional layer of the linear convolutional aliasing model for one-dimensional convolution operation. The one-dimensional convolution captures the local dependency and feature pattern of the input signal in the time dimension by sliding the convolution kernel, and uses the ReLU activation function to obtain 32 first feature maps, each with a size of 96. The 32 first feature maps are subjected to the maximum pooling operation, with a pooling window size of 2 and a step size of 2. The maximum pooling can reduce the size of the feature map, reduce the dimension of the data and the computational complexity, and also has a certain anti-noise ability. Through the pooling operation, 32 downsampled second feature maps are obtained, and the size of each feature map is 48. The 32 downsampled second feature maps are input into the second one-dimensional convolution layer for convolution operation, and 64 third feature maps are obtained through the ReLU activation function, and the size of each feature map is 46. The 64 third feature maps are subjected to the maximum pooling operation again, with a pooling window size of 2 and a step size of 2. This step reduces the resolution and data volume of the feature map, and 64 downsampled fourth feature maps are obtained, and the size of each feature map is 23. The 64 downsampled fourth feature maps are input into the third one-dimensional convolution layer for one-dimensional convolution operation, and 128 fifth feature maps are obtained through the ReLU activation function, and the size of each feature map is 21. After convolution and pooling, the 128 fifth feature maps are flattened to facilitate input into the fully connected layer. The feature vector dimension obtained after flattening is 2688. The feature vector is input into the first fully connected layer, which contains 1024 neurons. The activation function uses ReLU to perform nonlinear mapping on the input features, extract more expressive features, and obtain a 1024-dimensional feature vector. The 1024-dimensional feature vector is batch normalized to make the distribution of features more uniform, avoid the gradient vanishing or gradient explosion problems during model training due to the large data range, and improve the convergence speed and stability of the model. In order to extract the core features of camera motion, the normalized feature vector is input into the autoencoder network. The autoencoder network consists of an encoder and a decoder. The encoder is used to reduce the dimensionality of the features, while the decoder is used to remap the compressed features back to the high-dimensional space.The encoder contains three fully connected layers, with the number of neurons being 512, 256, and 128, respectively, which gradually reduce the dimension of the feature to capture the key information in the feature and remove redundancy. After processing by the encoder, a 128-dimensional compressed feature is obtained, which contains the most important motion information in the original high-dimensional feature. Using the 128-dimensional compressed feature as input, it is reconstructed through the decoder. The decoder also consists of three fully connected layers, with the number of neurons being 256, 512, and 1024, respectively. Through the processing of these layers, the compressed features are restored to the original high-dimensional space to obtain a high-dimensional motion feature vector.

[0016] Step S3, according to the high-dimensional motion feature vector and the original image sequence, a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model are used for fusion to obtain a camera motion parameter prediction value; Specifically, feature point extraction and feature point matching are performed on adjacent frames in the original image sequence. By using feature point detection methods such as SIFT and SURF, significant feature points in the image are identified and matched to obtain a feature point correspondence matrix. Based on the feature point correspondence matrix, abnormal matching points are eliminated, and feature points with incorrect matching are eliminated to improve the accuracy of subsequent motion estimation. By calculating the basic matrix, the relative rotation matrix R and translation vector t of the camera are solved to obtain the initial estimated value of the image motion. The high-dimensional motion feature vector is input into the BP neural network prediction model. The output layer of the BP neural network is set to six neurons, corresponding to the six-degree-of-freedom motion parameters, including the three-dimensional translation and three-dimensional rotation of the camera. In the process of forward propagation, through a series of nonlinear transformations, the BP neural network uses the mapping relationship between the learned motion features and the motion state to calculate the initial motion parameter prediction value based on the IMU data. These prediction values ​​reflect the motion information provided by the IMU sensor, and combined with the image motion features, they can provide a reliable basis for motion compensation. The initial estimated value of image motion and the predicted value of initial motion parameters based on IMU data are fused. The weighted average method is used to combine the motion estimates from two different sources, image and IMU, to obtain the fused camera motion parameters. The weights are dynamically adjusted according to the uncertainty of IMU and image motion estimation, and can adapt to the changes in motion state in different motion scenes. The fused camera motion parameters are substituted into the six-degree-of-freedom motion model based on sine wave combination. This model is used to describe the motion more smoothly and periodically by representing the three-dimensional translation and three-dimensional rotation of the camera as the superposition of six sine wave functions, especially for small-scale, periodic jitter. Through the combined calculation of sine waves, the position and attitude of the camera at the current moment are obtained, which reflects the spatial position of the camera and contains specific information about its orientation. The position and attitude of the camera at the current moment are processed by Kalman filtering to improve the smoothness and accuracy of the prediction results. The state vector of the Kalman filter contains position, velocity and acceleration, and the measurement vector is the fused camera motion parameters. Through the prediction-update cycle of the Kalman filter, the state estimate is continuously updated by combining the prediction model with the actual measurement value to obtain the smoothed camera motion parameters. Kalman filtering can optimally estimate the state of a dynamic system and is suitable for processing linear dynamic systems with Gaussian noise. Based on the smoothed camera motion parameters, the current frame in the original image sequence is back-projected, and the Euclidean distance between the projection point and the actual feature point is calculated to obtain the reprojection error. The reprojection error is used to evaluate the accuracy of the predicted camera motion parameters. If the difference between the projection result and the actual feature point position is small, it means that the predicted value of the motion parameter has a high accuracy.When the reprojection error is less than the preset threshold, the current smoothed camera motion parameters are output as the predicted values ​​of the camera motion parameters; if the reprojection error exceeds the preset threshold, the weight of data fusion is adjusted, and data fusion and processing are performed again. This process will be iterated continuously until the accuracy requirements are met or the maximum number of iterations is reached.

[0017] Step S4, constructing an advance network compensation model based on the camera motion parameters, and dynamically adjusting the compensation model parameter matrix according to the jitter intensity evaluation results of the IMU data variance and peak analysis; Specifically, the camera motion parameters are analyzed in time series, and the predicted sequence of the camera motion parameters is obtained by predicting the motion parameters. The predicted camera motion parameter sequence is input into the advance network compensation model. The advance network compensation model consists of three fully connected layers, the first layer contains 64 neurons, the second layer contains 32 neurons, and the third layer contains 16 neurons. The ReLU activation function is used between each layer to introduce nonlinearity. Through forward propagation calculation, the advance network compensation model can generate a 3×3 initial compensation model parameter matrix, which is used to describe the geometric transformation relationship of the camera during the compensation process. Since the model has a multi-layer fully connected structure, it can better capture the complex dynamic characteristics of camera motion. In order to dynamically adjust the parameter matrix of the compensation model, the jitter characteristics of the IMU data are analyzed. The variance of the six-degree-of-freedom IMU fusion data in the sliding time window is calculated to obtain the variance sequence of the IMU data. The variance calculation can reflect the degree of change of the IMU data in different time periods. The variance sequence of the IMU data is analyzed by fast Fourier transform to obtain the spectrum characteristics of the IMU data. The spectrum feature contains the amplitude information of motion data at different frequencies, which is helpful for identifying the frequency components and intensity of motion jitter. Based on the spectrum features of IMU data, the peak amplitude and corresponding frequency are calculated to obtain the jitter intensity feature vector that describes the motion jitter characteristics, which contains the main jitter frequencies and their corresponding intensity information in the motion process. This information can effectively characterize the instability of the camera during motion. The jitter intensity feature vector is input into the jitter intensity evaluation model. The jitter intensity evaluation model adopts the support vector regression (SVR) algorithm. Through the regression analysis of the SVR model, the jitter intensity evaluation score is calculated. SVR has good generalization ability and can accurately evaluate the jitter intensity when the sample size is limited. Based on the jitter intensity evaluation score, the compensation intensity coefficient is calculated using the preset adaptive weight function. The adaptive weight function flexibly adjusts the compensation intensity according to the different jitter intensity conditions, ensuring that stronger compensation is provided when the jitter is strong, and reducing the compensation intensity when the jitter is weak, so as to avoid introducing artifacts caused by over-compensation. The calculated compensation intensity coefficient is multiplied by each element in the initial compensation model parameter matrix to achieve dynamic adjustment of the compensation model parameter matrix. This adjustment method can optimize the compensation parameters in real time according to the actual motion state, so that the compensation effect is more in line with the needs of the camera during actual motion, ensuring the accuracy and stability of the compensation.

[0018] Step S5, performing geometric transformation on the original image data based on the compensation model parameter matrix, and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and foreground mask; Specifically, the compensation model parameter matrix is ​​applied to the original image data to perform an affine transformation, thereby adjusting the geometric shape of the image to obtain a first image sequence. The affine transformation rotates, scales, and translates the image based on the compensation model parameter matrix to eliminate irregular changes caused by camera motion and preliminarily align the image sequence. A dense optical flow field is calculated between adjacent frames in the first image sequence to obtain a motion vector field at each pixel position. The dense optical flow field calculation can capture the displacement changes of all pixels in the image. Based on the motion vector field obtained from the dense optical flow field, the first image sequence is subjected to local motion compensation and pixel resampling to obtain a second image sequence. Local motion compensation uses the information of the optical flow field to eliminate local subtle motion by accurately relocating pixels, thereby significantly improving the stability of the image sequence. On this basis, a time-weighted background pixel pool is constructed for the second image sequence to capture a stable background in a dynamic environment. A fixed-size pixel value queue is maintained for each pixel position. Newly collected pixel values ​​are queued in a time-decayed weighted manner, that is, a higher weight is given to new pixels so that they have a greater impact on the current background state, while the old pixel values ​​gradually decay. When the queue reaches its maximum capacity, the oldest pixel value will be removed from the queue. The dynamic update mechanism enables the background model to adapt to dynamic situations such as lighting changes and slowly moving objects in the scene, thereby improving the reliability of background modeling. Based on the dynamically updated background model, foreground detection is performed on each frame in the second image sequence to separate the moving objects from the background. By comparing the difference between the current frame and the background model, it is determined which pixels belong to the foreground, and a foreground mask is generated accordingly. The foreground mask is a binary image in which the foreground part is marked as 1 and the background part is marked as 0, which is used to represent the moving objects in the image. The foreground mask is subjected to a pixel-by-pixel AND operation with the second image sequence to extract the foreground area and obtain the foreground image sequence. At the same time, in order to enhance the image stabilization effect, the foreground image sequence and the second image sequence are subjected to frame difference, and the change area in the image caused by camera motion is identified by calculating the absolute difference between adjacent frames. The adaptive threshold segmentation method is applied to segment the calculated absolute difference to distinguish the static and dynamic parts in the image. The use of adaptive threshold can dynamically adjust the threshold according to the grayscale characteristics of the image in different scenes, so as to adapt to different lighting conditions and motion intensity and obtain the foreground and background separation results. Through the above steps, the initial stable image sequence is finally obtained.

[0019] Step S6: performing content-aware filling and time-domain filtering processing on the initial stabilized image sequence and the foreground mask, and outputting target stabilized image sequences in multiple compensation modes.

[0020] Specifically, the edge region of each frame in the initial stable image sequence is detected and edge extended, and the image filling area in the subsequent operation is effectively increased by edge extension to obtain an extended image sequence. The blank area in the image is filled with content awareness according to the extended image sequence and the foreground mask. The content-aware filling process uses the pixel values ​​around the image to reasonably fill the blank area, ensuring that the filled image sequence has a natural transition effect visually and avoiding obvious artifacts caused by compensation and transformation operations. The filled image sequence is processed by time domain Kalman filtering, and the state vector of the Kalman filter is the pixel value of each pixel position, and the observation vector is the pixel value of the corresponding position of the adjacent frame. Through the prediction and update process of the Kalman filter, the time domain changes in the image are smoothed to reduce the image quality degradation caused by random jitter, and the image sequence after time domain filtering is obtained. Kalman filtering can effectively smooth the noise in the image sequence, thereby improving the temporal continuity of the image, making it more stable and smooth in motion scenes. Multi-scale image processing is performed based on the image sequence after time domain filtering, and each frame of the image is decomposed into multiple scales to obtain an initial multi-scale image sequence. The purpose of multi-scale processing is to separate the details and global information in the image so as to perform more accurate operations on features of different scales. Bilateral filtering is applied to each scale of the initial multi-scale image sequence. Bilateral filtering is a nonlinear filtering method that can remove noise while retaining edges, ensuring that the image has clear edges and low noise at all scales, and obtaining a filtered multi-scale image sequence. The filtered multi-scale image sequence is reconstructed to restore the original resolution to obtain a reconstructed image sequence. The reconstruction process ensures that the final image retains the clarity of the details and has overall smoothness by fusing the image information of each scale, thereby improving the visual quality. Three different compensation modes are applied to the reconstructed image sequence to improve the stability and adaptability of the image sequence. The first compensation mode is the global compensation mode, in which the image sequence is processed using a global affine transformation. By performing global rotation, scaling and translation operations on the entire image, it is suitable for relatively simple global motion compensation. The second compensation mode is the local compensation mode, which uses a grid deformation algorithm to divide the image into multiple grids and deform each grid independently to adapt to different local motion characteristics. The local compensation method can handle the non-uniform motion in the image, making the compensation effect more delicate. The third compensation mode is the adaptive compensation mode, which dynamically selects the compensation strategy according to the image content, combines the global and local motion characteristics, and compensates the image in the most suitable way for the current scene. The adaptive compensation mode is suitable for complex scenes and diverse motions. After applying the three different compensation modes, the image sequence of the global compensation mode, the image sequence of the local compensation mode, and the image sequence of the adaptive compensation mode are obtained.In order to select the best compensation result, the image sequences of the three compensation modes are evaluated for jitter, and the root mean square error (RMS error) of the inter-frame difference is calculated. This error is used to measure the stability between adjacent frames. The smaller the error, the more stable the image sequence. The compensation mode with the smallest error is selected as the target stable image sequence for output to ensure that the obtained image sequence has the best stabilization effect and visual quality in terms of motion compensation.

[0021] In the embodiment of the present invention, by performing low-pass filtering and adaptive Kalman filtering on multiple IMU sensor data, the noise and drift of the IMU data are effectively reduced, and the accuracy and reliability of the motion data are improved. The linear convolution aliasing model is used to process the IMU fusion data, which can better capture complex motion patterns, extract high-dimensional motion feature vectors, and provide richer information for subsequent motion parameter prediction. Combined with the six-degree-of-freedom motion model based on sine wave combination and the BP neural network prediction model, accurate prediction of camera motion parameters is achieved, effectively solving the problem that traditional methods are difficult to accurately describe complex motion. The advanced network compensation model and dynamic adjustment mechanism are introduced to adjust the compensation strategy in real time according to the jitter intensity evaluation results of the IMU data, thereby improving the flexibility and adaptability of the compensation. Combined with dense optical flow field optimization and time-weighted background pixel pool modeling, while performing image stabilization, the foreground information is effectively retained, and the visual quality of the stabilization effect is improved. Through content-aware filling and time-domain filtering processing, the edge blanks and time-domain incoherence that may occur during image stabilization are solved, and the quality of the stabilized image sequence is further improved. It provides multiple compensation modes (global compensation, local compensation, and adaptive compensation), and dynamically selects the optimal mode through jitter evaluation to meet the stability requirements in different scenarios.

[0022] In a specific embodiment, the process of executing step S1 may specifically include the following steps: (1) Performing digital low-pass filtering on the raw data of acceleration and angular velocity collected by each IMU sensor to obtain first IMU data, and performing zero bias correction and temperature compensation on the first IMU data to obtain second IMU data; (2) Input the second IMU data into the adaptive Kalman filter, dynamically adjust the process noise covariance matrix according to the current motion state, obtain the third IMU data, perform Allan variance analysis on the third IMU data, calculate the noise characteristic parameters of each IMU sensor, and obtain the noise parameter matrix; (3) Based on the noise parameter matrix, the third IMU data of multiple IMU sensors are weightedly fused to obtain the fourth IMU data, and the attitude of the fourth IMU data is solved. The rotation matrix of the camera is calculated using the quaternion algorithm to obtain the camera attitude data; (4) The camera attitude data is combined with the fourth IMU data and fused through a complementary filtering algorithm to obtain corrected six-degree-of-freedom motion data. The corrected six-degree-of-freedom motion data is then time synchronized and interpolated to obtain six-degree-of-freedom IMU fusion data aligned with the camera frame rate.

[0023] Specifically, the acceleration and angular velocity raw data collected by each IMU sensor are digitally low-pass filtered to eliminate high-frequency noise components in the data, making the sensor data smoother and more suitable for subsequent processing. Assume that the raw acceleration data is , through a digital low-pass filter , the filtered data is expressed as: ; in, is the frequency response function of the filter, which can suppress the signal part above a certain cutoff frequency to obtain the first IMU data. The first IMU data is subjected to zero bias correction and temperature compensation. The IMU sensor is affected by bias and temperature changes, which can cause the measured value to deviate from the true value. In order to correct these effects, zero bias correction is performed, and the correction process is expressed as: ; in, is the bias value of the IMU in a static state, which is measured experimentally and used for subsequent compensation. Temperature compensation is achieved by modeling the change of sensor data with temperature. The temperature model is expressed as ,in is the temperature, and the compensated IMU data is the second IMU data. The second IMU data is input into the adaptive Kalman filter for processing. The Kalman filter is an optimal estimator that estimates the state of the system by combining the prediction model and the measurement data. In order to adapt to the dynamically changing motion state, the adaptive Kalman filter dynamically adjusts the process noise covariance matrix according to the current motion state. Assume the system state is , the process model is: ; in, is the state transition matrix, is the process noise, and its covariance is The adaptive Kalman filter is adjusted according to the motion state. , to improve the accuracy of filtering. After Kalman filtering, the third IMU data is obtained. In order to analyze the noise characteristics of the IMU sensor, AIlan variance analysis is performed on the third IMU data. Allan variance analysis is a tool for evaluating noise characteristics. By calculating the variance of the data at different time intervals, different noise sources in the sensor can be identified. Through Allan variance analysis of the third IMU data, the noise characteristic parameters of each IMU sensor are obtained, such as random walk, bias instability, etc., and a noise parameter matrix is ​​constructed based on these parameters for weighted processing of subsequent data fusion. Based on the noise parameter matrix, the third IMU data of multiple IMU sensors are weightedly fused. According to the noise characteristics of each IMU sensor, the data of each sensor are weighted and summed to improve the accuracy of the overall data. Assume For the The data of IMU sensors, weighted by , then the fourth IMU data after weighted fusion Expressed as ; in, is the number of IMU sensors, weight According to the characteristic parameters in the noise parameter matrix, the sensor data with less noise is given a greater weight. The attitude of the fourth IMU data is solved, and the rotation matrix of the camera is calculated using the quaternion algorithm. Quaternion representation has an important application in attitude solution, especially in avoiding singularity problems in Euler angle representation (i.e. gimbal lock). Assume that the current quaternion is , the camera's rotation matrix Calculated by quaternion: ; The camera's attitude data is obtained through quaternion calculation to describe the camera's rotation. The camera's attitude data is combined with the fourth IMU data and fused through a complementary filtering algorithm to obtain the corrected six-degree-of-freedom motion data. Complementary filtering is a method of combining low-frequency and high-frequency data. IMU data provides short-term high-frequency motion information, while attitude data provides long-term low-frequency stability information. Through fusion, more accurate six-degree-of-freedom motion data is obtained. The corrected six-degree-of-freedom motion data is time-synchronized and interpolated to align it with the camera's frame rate. The data acquisition frequency of the IMU sensor is usually higher than the camera's frame rate. The IMU data is downsampled or interpolated to ensure that the motion data and the image sequence match in time. The interpolation process is implemented through linear interpolation, spline interpolation and other methods, so that accurate six-degree-of-freedom motion data can be obtained at each time point of the camera acquisition, and finally the six-degree-of-freedom IMU fusion data aligned with the camera frame rate is obtained for camera motion compensation and image stabilization.

[0024] In a specific embodiment, the process of executing step S2 may specifically include the following steps: (1) The linear convolutional aliasing model includes three one-dimensional convolutional layers, two maximum pooling layers, and two fully connected layers. The first convolutional layer uses 32 one-dimensional convolutional kernels of size 5 with a step size of 1, the second convolutional layer uses 64 one-dimensional convolutional kernels of size 3 with a step size of 1, and the third convolutional layer uses 128 one-dimensional convolutional kernels of size 3 with a step size of 1. (2) The six-degree-of-freedom IMU fusion data is divided into time windows to obtain multiple sub-signal sequences, and the multiple sub-signal sequences are input into the first one-dimensional convolution layer of the linear convolution aliasing model to perform a one-dimensional convolution operation. The ReLU activation function is used to obtain 32 first feature maps, each with a size of 96; (3) Perform a maximum pooling operation on the 32 first feature maps, with a pooling window size of 2 and a step size of 2, to obtain 32 downsampled second feature maps with a size of 48. The 32 downsampled second feature maps are input into the second one-dimensional convolution layer for a one-dimensional convolution operation. The ReLU activation function is used to obtain 64 third feature maps with a size of 46. (4) Perform a maximum pooling operation on the 64 third feature maps, with a pooling window size of 2 and a step size of 2, to obtain 64 downsampled fourth feature maps with a size of 23. The 64 downsampled fourth feature maps are input into the third one-dimensional convolution layer for a one-dimensional convolution operation. The ReLU activation function is used to obtain 128 fifth feature maps with a size of 21 each. (5) Flatten the 128 fifth feature maps to obtain a 2688-dimensional feature vector, and input it into the first fully connected layer. The number of neurons in the fully connected layer is 1024, and the activation function is ReLU. A 1024-dimensional feature vector is obtained, and the 1024-dimensional feature vector is batch normalized to obtain a normalized feature vector. (6) The normalized feature vector is input into the autoencoder network. The autoencoder network includes an encoder and a decoder. The encoder consists of three fully connected layers with 512, 256 and 128 neurons respectively. The decoder consists of three fully connected layers with 256, 512 and 1024 neurons respectively. The encoder of the autoencoder network performs dimensionality reduction processing on the normalized feature vector to obtain a 128-dimensional compressed feature. The 128-dimensional compressed feature is used as input and reconstructed by the decoder to obtain a high-dimensional motion feature vector.

[0025] Specifically, the linear convolutional aliasing model contains three one-dimensional convolutional layers, two maximum pooling layers, and two fully connected layers. The first convolutional layer uses 32 one-dimensional convolutional kernels of size 5 and a step size of 1 to capture the short-term time-related features in the original input data. Assume that the input six-degree-of-freedom IMU fusion data is , its feature representation after convolution operation is: ; in, represents the weight of the first layer of convolution kernel, Represents the bias term, and 32 first feature maps are obtained through convolution operation, and the size of each feature map is 96. Apply the ReLU activation function to the feature map , to introduce nonlinearity, helping the network to have stronger expression ability when learning complex motion features. The 32 first feature maps are subjected to the maximum pooling operation, with a pooling window size of 2 and a step size of 2. The purpose of the maximum pooling operation is to downsample the feature map, reduce the dimension of the feature and the complexity of the calculation, and increase the invariance of the feature. Through the maximum pooling process, the feature map size is reduced from 96 to 48, and 32 downsampled second feature maps are obtained. The downsampled feature map is input into the second convolution layer. The second convolution layer uses 64 one-dimensional convolution kernels of size 3 and a step size of 1 to continue to extract and refine the features. After the convolution operation, the ReLU activation function is used to obtain 64 third feature maps, each with a size of 46. The 64 third feature maps are subjected to the maximum pooling operation again, with a pooling window size of 2 and a step size of 2. This operation reduces the size of the feature map to 23, and 64 downsampled fourth feature maps are obtained. The 64 downsampled fourth feature maps are input to the third convolutional layer. The third convolutional layer uses 128 one-dimensional convolution kernels of size 3 and a step size of 1. After convolution operation and using the ReLU activation function, 128 fifth feature maps are obtained, and the size of each fifth feature map is 21. After the convolution and pooling operations are completed, the 128 fifth feature maps are flattened into a vector with a total dimension of 128×21=2688. The flattened feature vector contains rich motion feature information extracted by the convolutional neural network. The flattened feature vector is input to the first fully connected layer, which contains 1024 neurons and uses the ReLU activation function to perform nonlinear mapping on the input features to obtain a 1024-dimensional feature vector. In order to enhance the stability of the features and enable the network to converge faster during training, the 1024-dimensional feature vector is batch normalized to obtain a normalized feature vector. Batch normalization adjusts the mean and variance of the features to make them obey the standard normal distribution, thereby improving the training efficiency and performance of the model. In order to reduce the dimension of the feature and extract the most representative motion information from it, the normalized feature vector is input into the autoencoder network. The autoencoder network consists of an encoder and a decoder. The encoder part includes three fully connected layers with 512, 256 and 128 neurons respectively. The main function of the encoder is to gradually compress the input high-dimensional features to extract a more compact feature representation. Assume that the feature representation after the encoder is , then the calculation process is: ; in, Represent the weight matrices of the three fully connected layers respectively, represents the bias term, Represents the activation function (such as ReLU), and finally obtains 128-dimensional compressed features. These compressed features contain the core motion information in the original high-dimensional features, while eliminating redundancy and noise. The 128-dimensional compressed features are input into the decoder, which consists of three fully connected layers with 256, 512 and 1024 neurons respectively. Through the layer-by-layer reverse mapping of the decoder, the compressed features are restored to the high-dimensional space to obtain the motion feature vector of the same dimension as the original high-dimensional feature. Through the training of the autoencoder, the network learns how to effectively represent and reconstruct motion features to ensure that key information is not lost during the feature compression process.

[0026] In a specific embodiment, the process of executing step S3 may specifically include the following steps: A1: Extract and match feature points of adjacent frames in the original image sequence to obtain the feature point correspondence matrix. Based on the feature point correspondence matrix, remove abnormal matching points and calculate the basic matrix. Then, calculate the relative rotation matrix R and translation vector t of the camera to obtain the initial estimation value of the image motion. A2: Input the high-dimensional motion feature vector into the BP neural network prediction model. The output layer of the BP neural network prediction model corresponds to the motion parameters of six degrees of freedom. The initial motion parameter prediction value based on the IMU data is obtained through forward propagation calculation; A3: Perform data fusion on the initial estimated value of image motion and the initial motion parameter prediction value based on IMU data to obtain the fused camera motion parameters; A4: Substitute the fused camera motion parameters into the six-degree-of-freedom motion model based on sine wave combination. The six-degree-of-freedom motion model based on sine wave combination represents the three-dimensional translation and three-dimensional rotation of the camera as the superposition of six sine wave functions, and calculates the position and posture of the camera at the current moment; A5: Perform Kalman filtering on the camera's current position and attitude. The filter's state vector contains position, velocity, and acceleration. The measurement vector is the fused camera motion parameter. The smoothed camera motion parameters are obtained through a prediction-update cycle. A6: Based on the smoothed camera motion parameters, the current frame in the original image sequence is back-projected, and the Euclidean distance between the projected point and the actual feature point is calculated to obtain the reprojection error; A7: When the reprojection error is less than the preset threshold, the current smoothed camera motion parameters are output as the camera motion parameter prediction values; if it is greater than the preset threshold, return to step A3, adjust the weight of data fusion, and re-perform data fusion and processing until the accuracy requirement is met or the maximum number of iterations is reached.

[0027] Specifically, feature point extraction and feature point matching are performed on adjacent frames in the original image sequence. The SIFT or ORB feature extraction algorithm is used to extract unique feature points from the image. These feature points are consistent between different frames and are used to estimate camera motion. After extracting the feature points, a feature point correspondence matrix between adjacent frames is established through a matching algorithm (such as nearest neighbor matching). The accuracy of the matching is improved by removing abnormal matching points. The RANSAC (random sampling consistency) algorithm is used to remove abnormal matching points to ensure that the final matched feature point pairs are reliable. Based on the feature point matching relationship, the basic matrix F is calculated. The basic matrix defines the geometric relationship between the two images. The camera's relative rotation matrix R and translation vector t are solved through the basic matrix. These parameters describe the relative motion of the camera between two adjacent frames, that is, the initial estimate of the image motion. The high-dimensional motion feature vector is input into the BP neural network prediction model. The BP neural network is a multi-layer perceptron with nonlinear mapping capabilities. By learning from training data, it can predict the motion state of the camera. The output layer of the BP neural network corresponds to the motion parameters of six degrees of freedom, which are three-dimensional translation, and 3D rotation In the process of forward propagation, the BP neural network uses the high-dimensional features of the input, and after step-by-step calculation of the hidden layer, outputs the predicted six-degree-of-freedom motion parameters, which are expressed as: ; in, is the input high-dimensional motion feature vector, is the weight matrix of each layer, is the bias, Represents the activation function (such as ReLU). Through forward propagation, the initial motion parameter prediction value based on IMU data is obtained. The initial image motion estimation value and the initial motion parameter prediction value based on IMU data are fused to obtain more accurate camera motion parameters. Data fusion is achieved by weighted averaging, where the weights of image motion estimation and IMU prediction value are assigned according to their respective uncertainties. Assume that the image estimation value is , the IMU prediction value is , then the fused camera motion parameters It is expressed as: ; in, and Represent the weights of image and IMU data respectively, and satisfy . By reasonably allocating weights, the advantages and disadvantages of image and IMU data are balanced to improve the accuracy of motion estimation. The fused camera motion parameters are substituted into the six-degree-of-freedom motion model based on sine wave combination, which is used to model the camera motion more smoothly. Assuming that the three-dimensional translation and three-dimensional rotation of the camera are respectively expressed as the superposition of six sine wave functions, it can be expressed as: ; in, are the amplitudes of the sine waves, is the angular frequency, is the initial phase. The periodic motion characteristics of the camera are better described by the superposition of sine functions, so that the position and attitude of the camera at the current moment can be calculated based on the fused motion parameters. The camera position and attitude at the current moment are processed by Kalman filtering. Kalman filter is an optimal state estimation method suitable for state estimation of linear dynamic systems. The state vector of the filter contains position, velocity and acceleration, which is recorded as , the measurement vector is the fused camera motion parameter, and the smoothed camera motion parameter is obtained through the prediction-update cycle of the Kalman filter. The prediction equation of the Kalman filter is expressed as: ; in, is the state transfer matrix, is the control input matrix, To control the input vector, the motion trajectory is smoothed through prediction and update steps to reduce the impact of noise. Based on the smoothed camera motion parameters, the current frame in the original image sequence is back-projected to calculate the Euclidean distance between the projected point and the actual feature point. Assume that the coordinates of the projected point are , the actual feature point coordinates are , then the Euclidean distance It is expressed as: ; By calculating this distance, the reprojection error is obtained to evaluate the accuracy of the motion parameters. When the reprojection error is less than the preset threshold, the current smoothed camera motion parameters are output as the predicted values ​​of the camera motion parameters. If the reprojection error is greater than the preset threshold, return to step A3, adjust the weight of data fusion, and re-perform data fusion and processing until the accuracy requirement is met or the maximum number of iterations is reached.

[0028] In a specific embodiment, the process of executing step S4 may specifically include the following steps: (1) Perform time series analysis on the camera motion parameters to obtain a predicted camera motion parameter sequence, and input the predicted camera motion parameter sequence into the advance network compensation model. The initial compensation model parameter matrix is ​​obtained by forward propagation calculation. The advance network compensation model contains three fully connected layers, with 64, 32, and 16 neurons in each layer, respectively. The ReLU activation function is used, and the output layer corresponds to a 3×3 compensation model parameter matrix. (2) Calculate the variance of the six-degree-of-freedom IMU fusion data within the sliding time window to obtain the IMU data variance sequence, and perform fast Fourier transform on the IMU data variance sequence to obtain the spectral characteristics of the IMU data; (3) Based on the frequency spectrum characteristics of the IMU data, the peak amplitude and frequency are calculated to obtain the jitter intensity feature vector, and the jitter intensity feature vector is input into the jitter intensity assessment model. The jitter intensity assessment score is calculated by the jitter intensity assessment model. The jitter intensity assessment model adopts the support vector regression algorithm. (4) According to the jitter intensity evaluation score, the compensation intensity coefficient is calculated using a preset adaptive weight function, and the compensation intensity coefficient is multiplied by each element of the initial compensation model parameter matrix to dynamically adjust the compensation model parameter matrix.

[0029] Specifically, the camera motion parameters are analyzed in time series to predict the camera motion state at future moments. Assume that the camera motion parameters are , predict the future motion parameter sequence through a time series analysis model (such as an autoregressive model or a long short-term memory network LSTM), expressed as Time series analysis can capture the trend and periodicity of camera motion. The predicted motion parameter sequence is input into the advance network compensation model, which consists of three fully connected layers, each with 64, 32 and 16 neurons, respectively. The ReLU activation function is used to introduce nonlinearity, thereby enhancing the network's expressive power. Through forward propagation calculation, the 3×3 compensation model parameter matrix of the output layer is obtained. . Assume that the input vector is , after each layer of transformation, is expressed as: ; in, is the weight matrix, is the bias, ReLU function ReLU Used to introduce nonlinearity. The output of the forward propagation process is a 3×3 initial compensation model parameter matrix, which is used to describe the preliminary geometric transformation relationship of camera motion compensation. In order to optimize the parameters of the compensation model, the jitter characteristics in the IMU data are analyzed. The variance of the six-degree-of-freedom IMU fusion data is calculated in the sliding time window to obtain the variance sequence of the IMU data. The variance sequence can reflect the degree of fluctuation of the IMU data at different times. Assume that the IMU data is , then in the sliding time window The variance within is calculated as: ; in, is the mean of the data in the window, For in time The variance at the moment. Perform a fast Fourier transform on this variance sequence to obtain the spectral characteristics of the IMU data. The spectral characteristics describe the energy distribution of the data at different frequency components, which can help identify periodic jitter in camera motion. Based on the spectrum analysis results, calculate the peak amplitude and corresponding frequency in the spectrum to obtain the jitter intensity feature vector ,in Indicates The peak value, Represents the corresponding frequency. These features reflect the impact of different frequency components on the camera during movement. The input is then fed into the jitter intensity assessment model, which uses the support vector regression (SVR) algorithm to calculate the jitter intensity assessment score by learning the relationship between the spectrum features and the jitter intensity. The goal of support vector regression is to find a function that can accurately describe the relationship between input and output so as to minimize the error. The jitter intensity evaluation score is used to indicate the degree of jitter in the current camera motion. , using the preset adaptive weight function to calculate the compensation intensity coefficient The adaptive weight function is obtained based on experience or through training to ensure that appropriate compensation can be provided under different jitter intensities. For example, the compensation intensity coefficient It is expressed as: ; in, To adjust the parameters, is the threshold value, and the function form is similar to the Sigmoid function, which is used to map the jitter assessment score to a reasonable compensation intensity range. When the jitter intensity is large, the compensation intensity coefficient is large, and vice versa. Multiply each element of the initial compensation model parameter matrix to achieve dynamic adjustment of the compensation model parameter matrix. The adjusted compensation model parameter matrix It is expressed as: ; The dynamic adjustment mechanism enables the compensation model to adapt to different motion environments. When the jitter is large, the compensation strength is increased to reduce the unstable effect caused by motion, and when the jitter is small, the compensation is reduced to avoid unnatural pictures caused by over-correction.

[0030] In a specific embodiment, the process of executing step S5 may specifically include the following steps: (1) applying the compensation model parameter matrix to the original image data, performing affine transformation to obtain a first image sequence, and performing dense optical flow field calculation between adjacent frames in the first image sequence to obtain a motion vector field; (2) Based on the motion vector field, local motion compensation and pixel resampling are performed on the first image sequence to obtain a second image sequence, and a time-weighted background pixel pool is constructed for the second image sequence. A fixed-size pixel value queue is maintained for each pixel position. New pixel values ​​are queued with a time-attenuated weight. When the queue is full, the oldest pixel value is removed from the queue to obtain a dynamically updated background model. (3) Based on the dynamically updated background model, foreground detection is performed on each frame of the second image sequence to obtain a foreground mask; (4) Performing a pixel-by-pixel AND operation on the foreground mask and the second image sequence to extract the foreground area and obtain a foreground image sequence. Performing frame difference on the foreground image sequence and the second image sequence to calculate the absolute difference and performing adaptive threshold segmentation to obtain an initial stable image sequence.

[0031] Specifically, the compensation model parameter matrix is ​​applied to the original image data for affine transformation. Affine transformation is a two-dimensional spatial transformation that performs operations such as rotation, scaling, and translation on the image to correct the irregular transformation caused by camera motion. Assume that the compensation model parameter matrix is , the coordinates of the original image are , the new coordinates after affine transformation are , the formula of affine transformation is expressed as: ; in, are the scaling and rotation parameters, is the translation parameter. Through affine transformation, the first image sequence is obtained, in which the original image data is compensated so that the influence of camera motion on the image is preliminarily corrected. The dense optical flow field is calculated between adjacent frames in the first image sequence to obtain the motion vector field of each pixel position. The dense optical flow field is used to describe the pixel motion between adjacent frames. Assume that the pixel point in the first frame The corresponding position in the next frame is ,in Respectively expressed in and The purpose of optical flow field calculation is to find the corresponding position of each pixel in the next frame and obtain the motion vector field. The calculation of optical flow field is realized by the optical flow constraint equation: ; in, Respectively represent the images in and The gradient in direction, Represents the gradient of the image over time. By solving this equation, we can get the motion vector of each pixel. . Based on the obtained motion vector field, local motion compensation and pixel resampling are performed on the first image sequence to obtain the second image sequence. The purpose of local motion compensation is to eliminate subtle motions in local areas of the image so that objects in the image remain aligned between adjacent frames and improve image stability. According to the motion vector of each pixel position, the pixel is repositioned and the pixel value is interpolated to ensure that the compensated image is visually continuous and consistent. A time-weighted background pixel pool is constructed for the second image sequence. For each pixel position, a fixed-size pixel value queue is maintained, and new pixel values ​​are queued with time-attenuated weights. When the queue is full, the oldest pixel value is dequeued to achieve dynamic updating of the background model. Assume that at time The pixel value of a certain pixel position at the moment is , its weight when joining the queue is ,in is the time decay coefficient, then the pixel values ​​in the background pixel pool gradually reduce their influence over time, and the new pixel values ​​occupy a larger proportion. This approach ensures that the background model adapts to changes in the scene, such as gradual changes in illumination and slowly moving objects in the background. Based on the dynamically updated background model, foreground detection is performed on each frame in the second image sequence to obtain a foreground mask. The goal of foreground detection is to separate moving objects in the scene from the static background, which is usually achieved by comparing the current frame with the background model. If the difference between the position of a pixel and the background model exceeds a preset threshold, the pixel is considered to belong to the foreground, otherwise it belongs to the background, thereby generating a foreground mask, which is a binary image used to identify moving objects in the image. The foreground mask is pixel-by-pixel ANDed with the second image sequence to extract the foreground area and obtain a foreground image sequence. The pixels marked as foreground in the foreground mask are multiplied with the corresponding pixels of the original image, and the pixel values ​​of the remaining positions are set to 0 to obtain a foreground image containing only moving objects. The foreground image sequence and the second image sequence are inter-frame differentiated, and the changing area in the scene can be identified by calculating the absolute difference between adjacent frames. Assume that the two frames of images in the second image sequence are and , then the inter-frame difference is expressed as: ; in, Represents the absolute difference image between frames, reflecting the pixel difference between the two frames. , an adaptive threshold segmentation method is applied to determine which differences belong to valid foreground motion and which belong to noise. Adaptive threshold segmentation dynamically selects the threshold according to the grayscale histogram of the image, thereby adapting to the lighting and motion conditions of different scenes, and finally obtaining an initial stable image sequence.

[0032] In a specific embodiment, the process of executing step S6 may specifically include the following steps: (1) Detect and extend the edge region of each frame in the initial stable image sequence to obtain an extended image sequence, and fill the blank region according to the extended image sequence and the foreground mask to obtain a filled image sequence; (2) Perform time-domain Kalman filtering on the padded image sequence, with the state vector being the pixel value and the observation vector being the pixel value at the corresponding position of the adjacent frame, to obtain the image sequence after time-domain filtering; (3) Performing multi-scale image processing based on the image sequence after time domain filtering to obtain an initial multi-scale image sequence, and performing bilateral filtering on each scale of the initial multi-scale image sequence to obtain a filtered multi-scale image sequence; (4) Reconstructing the filtered multi-scale image sequence to the original resolution to obtain a reconstructed image sequence; (5) Applying three different compensation modes to the reconstructed image sequence: global compensation mode, local compensation mode and adaptive compensation mode. The global compensation mode uses global affine transformation, the local compensation mode uses a grid deformation algorithm, and the adaptive compensation mode dynamically selects a compensation strategy according to the image content, thereby obtaining an image sequence in the global compensation mode, an image sequence in the local compensation mode and an image sequence in the adaptive compensation mode; (6) Perform jitter evaluation on the image sequence of the global compensation mode, the image sequence of the local compensation mode, and the image sequence of the adaptive compensation mode, calculate the root mean square error of the inter-frame difference, and select the compensation mode with the smallest error as the target stable image sequence output.

[0033] Specifically, the edge region of each frame in the initial stable image sequence is detected and extended. The edge detection adopts the classic Canny edge detection method, which can identify the significant edges in the image. Assume that the image is , the edge image is obtained after edge detection , and then expand the edge area to obtain the extended image sequence The blank area is filled based on the extended image and the foreground mask. The blank area is filled using a content-aware filling method based on the extended area, assuming that the pixel value of the blank area is The filling process interpolates or copies the pixel value after filling from the adjacent known area. To ensure that the blank part of the image is naturally integrated into the overall content, forming a filled image sequence . The padded image sequence is subjected to time-domain Kalman filtering to smooth the time-domain noise in the image sequence. Kalman filtering is an optimal state estimation method suitable for linear dynamic systems. The state vector is defined as the pixel value, and the observation vector is the pixel value at the corresponding position of the adjacent frame. Assume that the state vector is , the observation vector is , the prediction equation of Kalman filter is: ; in, is the state transfer matrix, is the control matrix, is the control input. Through the prediction and update steps, Kalman filtering is performed on each pixel value to obtain the image sequence after time domain filtering. This process effectively reduces short-term jitter in the image sequence, making the image more stable. Multi-scale image processing is performed based on the image sequence after time domain filtering, and the image is decomposed into multiple scales to process different levels of detail respectively. Each frame of the image is decomposed into multiple scales through the Gaussian pyramid to obtain the initial multi-scale image sequence , each scale contains images of different resolutions. The lower scale is used to capture the global features of the image, and the higher scale is used to process the details. At each scale, bilateral filtering is applied to the image. Bilateral filtering is an edge-preserving filtering method that can remove noise while keeping edges clear. The formula for bilateral filtering is: ; in, and are Gaussian functions of spatial distance and pixel intensity difference, respectively. is the normalization coefficient. The filtered multi-scale image sequence is obtained by bilateral filtering The filtered multi-scale image sequence is reconstructed to the original resolution to form a reconstructed image sequence The reconstruction process is to perform step-by-step upsampling and additive fusion of the multi-scale pyramid to restore the original resolution, ensuring that the image retains details while maintaining global consistency and improving visual quality. Three different compensation modes are applied to the reconstructed image sequence, namely global compensation mode, local compensation mode and adaptive compensation mode. In the global compensation mode, the image is processed using the overall affine transformation, and the affine transformation formula is as follows: ; in is the affine transformation matrix, including rotation, scaling and translation parameters. Global compensation is suitable for situations where the overall camera motion is relatively consistent. In the local compensation mode, the image is processed using a grid deformation algorithm, and the image is divided into multiple grids. Each grid is deformed independently to adapt to the local motion. This method can compensate for local complex motion more finely. In the adaptive compensation mode, the compensation strategy is dynamically selected according to the image content. If global motion is detected, affine transformation is used for compensation. If strong local motion is detected, the grid deformation method is used. The compensation strategy is automatically adjusted according to the image content to adapt to complex scenes. After applying the three compensation modes, the image sequence of the global compensation mode is obtained. , image sequence of local compensation mode and image sequences in adaptive compensation mode In order to select the best compensation result, the jitter evaluation of the three image sequences is performed, and the root mean square error (RMS error) of the inter-frame difference is calculated to measure the smoothness of the compensated image sequence. Assume that the two frames of images are and , then the root mean square error of the inter-frame difference is calculated as: ; in, is the total number of pixels in the image, and Respectively represent pixels in time and By calculating the RMS error of the image sequences under the three compensation modes, the compensation mode with the smallest error is selected as the target stable image sequence output, which means that this compensation mode performs best in smoothing the image sequence and reducing jitter.

[0034] The above describes the camera motion compensation method based on IMU in the embodiment of the present invention. The following describes the camera motion compensation device based on IMU in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, an IMU-based camera motion compensation device includes: The acquisition module is used to perform low-pass filtering and adaptive Kalman filtering on the raw data collected by multiple IMU sensors to obtain six-degree-of-freedom IMU fusion data; The extraction module is used to input the six-degree-of-freedom IMU fusion data into the linear convolution aliasing model for processing, obtain simulated motion data, and extract the corresponding high-dimensional motion feature vector; A fusion module is used to fuse the high-dimensional motion feature vector and the original image sequence using a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model to obtain a camera motion parameter prediction value; An evaluation module is used to build a look-ahead network compensation model based on camera motion parameters and dynamically adjust the compensation model parameter matrix according to the jitter intensity evaluation results of IMU data variance and peak analysis; The transformation module is used to geometrically transform the original image data based on the compensation model parameter matrix, and combine dense optical flow field optimization and time-weighted background pixel pool modeling to obtain the initial stable image sequence and foreground mask; The output module is used to perform content-aware filling and time-domain filtering on the initial stabilized image sequence and the foreground mask, and output a target stabilized image sequence with multiple compensation modes.

[0035] Through the cooperation of the above components, the noise and drift of IMU data are effectively reduced by low-pass filtering and adaptive Kalman filtering of multiple IMU sensor data, and the accuracy and reliability of motion data are improved. The linear convolution aliasing model is used to process IMU fusion data, which can better capture complex motion patterns and extract high-dimensional motion feature vectors, providing richer information for subsequent motion parameter prediction. Combining the six-degree-of-freedom motion model based on sine wave combination and the BP neural network prediction model, the accurate prediction of camera motion parameters is achieved, effectively solving the problem that traditional methods are difficult to accurately describe complex motion. The introduction of the advanced network compensation model and dynamic adjustment mechanism adjusts the compensation strategy in real time according to the jitter intensity evaluation results of the IMU data, which improves the flexibility and adaptability of the compensation. Combined with dense optical flow field optimization and time-weighted background pixel pool modeling, the foreground information is effectively retained while image stabilization is performed, and the visual quality of the stabilization effect is improved. Through content-aware filling and time-domain filtering processing, the edge blanks and time-domain incoherence that may occur during image stabilization are solved, further improving the quality of the stabilized image sequence. It provides multiple compensation modes (global compensation, local compensation, and adaptive compensation), and dynamically selects the optimal mode through jitter evaluation to meet the stability requirements in different scenarios.

[0036] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0037] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0038] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0039] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0040] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0041] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0042] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camera motion compensation method based on IMU, characterized in that: The method comprises: The raw data collected by multiple IMU sensors are processed by low-pass filtering and adaptive Kalman filtering to obtain six-degree-of-freedom IMU fusion data; Inputting the six-degree-of-freedom IMU fusion data into a linear convolution aliasing model for processing to obtain simulated motion data, and extracting the corresponding high-dimensional motion feature vector; According to the high-dimensional motion feature vector and the original image sequence, a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model are used for fusion to obtain a camera motion parameter prediction value; Building a look-ahead network compensation model based on the camera motion parameters, and dynamically adjusting the compensation model parameter matrix according to the jitter intensity evaluation results of the IMU data variance and peak analysis; Performing geometric transformation on the original image data based on the compensation model parameter matrix, and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and a foreground mask; Content-aware filling and time-domain filtering are performed on the initial stabilized image sequence and the foreground mask, and target stabilized image sequences in multiple compensation modes are output.

2. The camera motion compensation method based on IMU according to claim 1, characterized in that: The raw data collected by multiple IMU sensors are processed by low-pass filtering and adaptive Kalman filtering to obtain six-degree-of-freedom IMU fusion data, including: Performing digital low-pass filtering on the acceleration and angular velocity raw data collected by each IMU sensor to obtain first IMU data, and performing zero bias correction and temperature compensation on the first IMU data to obtain second IMU data; Input the second IMU data into an adaptive Kalman filter, dynamically adjust the process noise covariance matrix according to the current motion state, obtain the third IMU data, perform Allan variance analysis on the third IMU data, calculate the noise characteristic parameters of each IMU sensor, and obtain a noise parameter matrix; Based on the noise parameter matrix, weighted fusion is performed on the third IMU data of multiple IMU sensors to obtain fourth IMU data, and attitude calculation is performed on the fourth IMU data, and a rotation matrix of the camera is calculated using a quaternion algorithm to obtain camera attitude data; The camera posture data is combined with the fourth IMU data, and fused through a complementary filtering algorithm to obtain corrected six-degree-of-freedom motion data, and the corrected six-degree-of-freedom motion data is time synchronized and interpolated to obtain six-degree-of-freedom IMU fusion data aligned with the camera frame rate.

3. The camera motion compensation method based on IMU according to claim 2, characterized in that: The six-degree-of-freedom IMU fusion data is input into a linear convolution aliasing model for processing to obtain simulated motion data, and the corresponding high-dimensional motion feature vector is extracted, including: The linear convolution aliasing model includes three one-dimensional convolution layers, two maximum pooling layers and two fully connected layers, wherein the first convolution layer uses 32 one-dimensional convolution kernels of size 5 with a step size of 1, the second convolution layer uses 64 one-dimensional convolution kernels of size 3 with a step size of 1, and the third convolution layer uses 128 one-dimensional convolution kernels of size 3 with a step size of 1; Performing time window segmentation on the six-degree-of-freedom IMU fusion data to obtain multiple sub-signal sequences, and inputting the multiple sub-signal sequences into the first one-dimensional convolution layer of the linear convolution aliasing model to perform a one-dimensional convolution operation, using a ReLU activation function to obtain 32 first feature maps, each of which has a size of 96; Performing a maximum pooling operation on the 32 first feature maps, with a pooling window size of 2 and a step size of 2, to obtain 32 downsampled second feature maps, the size of the second feature map is 48, and inputting the 32 downsampled second feature maps into the second one-dimensional convolution layer, performing a one-dimensional convolution operation, using a ReLU activation function, to obtain 64 third feature maps, the size of each third feature map is 46; Performing a maximum pooling operation on the 64 third feature maps, with a pooling window size of 2 and a step size of 2, to obtain 64 downsampled fourth feature maps, the size of the fourth feature map is 23, and inputting the 64 downsampled fourth feature maps into the third one-dimensional convolution layer, performing a one-dimensional convolution operation, using a ReLU activation function, to obtain 128 fifth feature maps, the size of each fifth feature map is 21; Flatten the 128 fifth feature maps to obtain a 2688-dimensional feature vector, and input it into the first fully connected layer. The number of neurons in the fully connected layer is 1024, and the activation function adopts ReLU to obtain a 1024-dimensional feature vector. The 1024-dimensional feature vector is batch normalized to obtain a normalized feature vector. The normalized feature vector is input into an autoencoder network, which includes an encoder and a decoder. The encoder consists of three fully connected layers, and the number of neurons is 512, 256 and 128 respectively. The decoder consists of three fully connected layers, and the number of neurons is 256, 512 and 1024 respectively. The encoder of the autoencoder network performs dimensionality reduction processing on the normalized feature vector to obtain 128-dimensional compressed features. The 128-dimensional compressed features are used as input and reconstructed by the decoder to obtain a high-dimensional motion feature vector.

4. The IMU-based camera motion compensation method according to claim 3, characterized in that: The method of obtaining the camera motion parameter prediction value by fusing the high-dimensional motion feature vector and the original image sequence with a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model includes: A1: Extract and match feature points of adjacent frames in the original image sequence to obtain a feature point correspondence matrix, remove abnormal matching points based on the feature point correspondence matrix, calculate the basic matrix, and then solve the relative rotation matrix R and translation vector t of the camera to obtain the initial estimation value of the image motion; A2: Input the high-dimensional motion feature vector into a BP neural network prediction model, the output layer of which corresponds to the motion parameters of six degrees of freedom, and obtain the initial motion parameter prediction value based on the IMU data through forward propagation calculation; A3: performing data fusion on the image motion initial estimation value and the initial motion parameter prediction value based on the IMU data to obtain fused camera motion parameters; A4: Substituting the fused camera motion parameters into a six-degree-of-freedom motion model based on a sine wave combination, the six-degree-of-freedom motion model based on a sine wave combination represents the three-dimensional translation and three-dimensional rotation of the camera as a superposition of six sine wave functions, and calculating the position and posture of the camera at the current moment; A5: Perform Kalman filtering on the position and posture of the camera at the current moment. The state vector of the filter includes position, velocity and acceleration. The measurement vector is the fused camera motion parameter. The smoothed camera motion parameter is obtained through a prediction-update cycle. A6: Based on the smoothed camera motion parameters, reversely project the current frame in the original image sequence, calculate the Euclidean distance between the projected point and the actual feature point, and obtain a reprojection error; A7: When the reprojection error is less than a preset threshold, the current smoothed camera motion parameter is output as a camera motion parameter prediction value; if it is greater than the preset threshold, return to step A3, adjust the weight of data fusion, and re-perform data fusion and processing until the accuracy requirement is met or the maximum number of iterations is reached.

5. The IMU-based camera motion compensation method according to claim 4, characterized in that: The advanced network compensation model is constructed based on the camera motion parameters, and the compensation model parameter matrix is ​​dynamically adjusted according to the jitter intensity evaluation results of the IMU data variance and peak analysis, including: Performing time series analysis on the camera motion parameters to obtain a predicted camera motion parameter sequence, and inputting the predicted camera motion parameter sequence into the advance network compensation model, and obtaining an initial compensation model parameter matrix by forward propagation calculation, wherein the advance network compensation model comprises three fully connected layers, each with 64, 32 and 16 neurons, respectively, and using a ReLU activation function, and the output layer corresponds to a 3×3 compensation model parameter matrix; Calculating the variance of the six-degree-of-freedom IMU fusion data in a sliding time window to obtain an IMU data variance sequence, and performing a fast Fourier transform on the IMU data variance sequence to obtain a frequency spectrum feature of the IMU data; Based on the frequency spectrum characteristics of the IMU data, the amplitude and frequency of the peak are calculated to obtain a jitter intensity feature vector, and the jitter intensity feature vector is input into a jitter intensity assessment model, and a jitter intensity assessment score is obtained by calculating the jitter intensity assessment model, wherein the jitter intensity assessment model adopts a support vector regression algorithm; According to the jitter intensity evaluation score, a compensation intensity coefficient is calculated using a preset adaptive weight function, and the compensation intensity coefficient is multiplied by each element of the initial compensation model parameter matrix to dynamically adjust the compensation model parameter matrix.

6. The IMU-based camera motion compensation method according to claim 5, characterized in that: The method of geometrically transforming the original image data based on the compensation model parameter matrix and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and a foreground mask includes: Applying the compensation model parameter matrix to the original image data to perform affine transformation to obtain a first image sequence, and performing dense optical flow field calculation between adjacent frames in the first image sequence to obtain a motion vector field; Based on the motion vector field, local motion compensation and pixel resampling are performed on the first image sequence to obtain a second image sequence, and a time-weighted background pixel pool is constructed for the second image sequence, a pixel value queue of a fixed size is maintained for each pixel position, new pixel values ​​are queued according to time decay weights, and the oldest pixel value is dequeued when the queue is full, so as to obtain a dynamically updated background model; Based on the dynamically updated background model, performing foreground detection on each frame of the second image sequence to obtain a foreground mask; The foreground mask and the second image sequence are subjected to a pixel-by-pixel AND operation to extract the foreground area to obtain a foreground image sequence, and inter-frame difference is performed on the foreground image sequence and the second image sequence to calculate the absolute difference, and adaptive threshold segmentation is performed to obtain an initial stable image sequence.

7. The IMU-based camera motion compensation method according to claim 6, characterized in that: The performing content-aware filling and time-domain filtering on the initial stabilized image sequence and the foreground mask to output a target stabilized image sequence in multiple compensation modes includes: Detecting and extending the edge region of each frame of the initial stable image sequence to obtain an extended image sequence, and filling the blank region according to the extended image sequence and the foreground mask to obtain a filled image sequence; Performing time-domain Kalman filtering on the padded image sequence, where the state vector is the pixel value and the observation vector is the pixel value at the corresponding position of the adjacent frame, to obtain a time-domain filtered image sequence; Performing multiscale image processing based on the image sequence after the time domain filtering to obtain an initial multiscale image sequence, and performing bilateral filtering on each scale of the initial multiscale image sequence to obtain a filtered multiscale image sequence; Reconstructing the filtered multi-scale image sequence to the original resolution to obtain a reconstructed image sequence; Applying three different compensation modes to the reconstructed image sequence: a global compensation mode, a local compensation mode, and an adaptive compensation mode, wherein the global compensation mode uses an overall affine transformation, the local compensation mode uses a grid deformation algorithm, and the adaptive compensation mode dynamically selects a compensation strategy according to image content, thereby obtaining an image sequence in the global compensation mode, an image sequence in the local compensation mode, and an image sequence in the adaptive compensation mode; The image sequence of the global compensation mode, the image sequence of the local compensation mode and the image sequence of the adaptive compensation mode are subjected to jitter evaluation, the root mean square error of the inter-frame difference is calculated, and the compensation mode with the smallest error is selected as the target stable image sequence output.

8. A camera motion compensation device based on IMU, characterized in that: For executing the IMU-based camera motion compensation method according to any one of claims 1 to 7, the device comprises: The acquisition module is used to perform low-pass filtering and adaptive Kalman filtering on the raw data collected by multiple IMU sensors to obtain six-degree-of-freedom IMU fusion data; An extraction module is used to input the six-degree-of-freedom IMU fusion data into a linear convolution aliasing model for processing to obtain simulated motion data and extract the corresponding high-dimensional motion feature vector; A fusion module is used to fuse the high-dimensional motion feature vector and the original image sequence using a six-degree-of-freedom motion model based on a sine wave combination and a BP neural network prediction model to obtain a camera motion parameter prediction value; An evaluation module, used to construct an advance network compensation model based on the camera motion parameters, and dynamically adjust the compensation model parameter matrix according to the jitter intensity evaluation results of the IMU data variance and peak analysis; A transformation module, used for geometrically transforming the original image data based on the compensation model parameter matrix, and combining dense optical flow field optimization and time-weighted background pixel pool modeling to obtain an initial stable image sequence and a foreground mask; The output module is used to perform content-aware filling and time-domain filtering on the initial stabilized image sequence and the foreground mask, and output a target stabilized image sequence in multiple compensation modes.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and is characterized in that when the processor executes the computer program, the IMU-based camera motion compensation method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is enabled to perform the IMU-based camera motion compensation method according to any one of claims 1 to 7.

Images