An image stabilization method for unmanned surface vessels based on electromechanical integration technology
By combining mechanical and electronic image stabilization technologies, using mechanical image stabilization with IMU, LSTM neural network and fuzzy PID controller, and electronic image stabilization methods with SIFT, improved RANSAC and adaptive Gaussian filtering, the image jitter problem of unmanned surface vessels in complex sea conditions is solved, and efficient video image stabilization is achieved.
Patent Information
- Application Number
- CN202310701838.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-14
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-06-14
AI Technical Summary
The image jitter problem of unmanned surface vessels in complex sea conditions is a challenge. Existing mechanical image stabilization technology cannot effectively remove high-frequency, low-amplitude vibrations, while electronic image stabilization algorithms are complex and time-consuming, affecting environmental perception capabilities.
Combining mechanical and electronic image stabilization technologies, attitude angles are predicted using an inertial measurement unit (IMU) and a long short-term memory (LSTM) neural network. Mechanical image stabilization is performed using a fuzzy PID controller and a spherical gimbal, while electronic image stabilization is achieved through the SIFT algorithm, an improved RANSAC algorithm, and adaptive Gaussian filtering.
It achieves image stabilization under complex sea conditions, improves video image quality, and balances stability and processing efficiency, meeting the environmental perception needs of unmanned surface vessels.
Smart Images

Figure CN116723399B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image stabilization method, specifically an image stabilization method for unmanned surface vessels based on electromechanical technology. Background Technology
[0002] Video images acquired by unmanned surface vessels (USVs) in complex sea conditions are affected by both low-frequency, high-amplitude jitter from ocean waves and high-frequency, low-amplitude vibrations from their internal mechanical structures. To avoid impacting the USV's environmental perception capabilities, image stabilization preprocessing is necessary. Mechanical image stabilization often achieves good results and is widely used in shipborne, airborne, and vehicle-mounted applications. However, due to limitations in technical precision and frictional resistance between mechanical structures, it is generally only used to stabilize low-frequency, high-amplitude jitter. Electronic image stabilization, primarily achieved through algorithms, offers high flexibility and low cost, and can be used to remove high-frequency, low-amplitude vibrations. However, the stabilization result is related to the quality of the image itself, and the motion estimation algorithms are relatively complex and time-consuming. Summary of the Invention
[0003] To address the problems existing in the background technology, this invention provides an image stabilization method for unmanned surface vessels (USVs) based on electromechanical combined image stabilization technology. To address the image jitter problem of USVs in complex sea conditions, mechanical and electronic image stabilization technologies can be combined to achieve video stabilization optimization. This invention processes video from two aspects: image stabilization gimbal control and image de-shaking algorithms, thereby improving video image quality. Mechanical image stabilization is achieved by controlling the movement of the image stabilization gimbal. Then, an image stabilization algorithm based on an improved RANSAC algorithm and adaptive Gaussian filtering is proposed to further de-shake the image sequence obtained by the camera, achieving electronic image stabilization.
[0004] The technical solution used in this invention is:
[0005] This invention relates to an image stabilization method for unmanned surface vessels based on electromechanical integration technology:
[0006] The method includes the following steps:
[0007] Step 1: When the unmanned surface vessel equipped with a spherical gimbal is navigating, mechanical image stabilization is performed on the spherical gimbal.
[0008] Step 2: Capture real-time video of the unmanned surface vessel (USV) while it is sailing using a spherical gimbal. Then, use electronic image stabilization to process the real-time video captured by the spherical gimbal to obtain a stabilized video, thus stabilizing the images captured by the USV.
[0009] In step one, mechanical image stabilization is used to stabilize the spherical gimbal, as detailed below:
[0010] 1.1) The three-axis attitude angle data of the unmanned surface vessel is collected by the inertial measurement unit (IMU). The three-axis attitude angle data is preprocessed by normalization, and then filtered by a Gaussian filter to obtain filtered three-axis attitude angle data. The filtered three-axis attitude angle data is input into the long short-term memory (LSTM) neural network for training to obtain the optimal LSTM neural network. Normalization preprocessing of the input data can eliminate the influence of the dimensions of the input attitude angles and speed up the training of the neural network.
[0011] 1.2) When the unmanned surface vessel is navigating, the three-axis attitude angle data of the unmanned surface vessel are collected in real time by the inertial measurement unit (IMU) and input into the optimal long short-term memory (LSTM) neural network. The optimal LSTM neural network outputs the predicted three-axis attitude angle data of the unmanned surface vessel.
[0012] 1.3) The predicted three-axis attitude angle data of the unmanned surface vessel is used to calculate the attitude using the Euler angle method and then input into a constrained fuzzy PID controller. The fuzzy PID controller outputs control signals to control the spherical gimbal. The spherical gimbal detects its own angle in real time and inputs it into the fuzzy PID controller for closed-loop control, thereby realizing the mechanical image stabilization of the spherical gimbal.
[0013] Attitude calculation can also be performed using methods such as direction cosine and quaternions. When using the Euler angle method, quaternions can be used for further optimization. When the stabilized gimbal operates on an unmanned surface vessel (USV), it operates in two coordinate systems. One is the geographic coordinate system Oxyz, denoted as {n}, with the gimbal's center of mass as the origin, east as the x-axis, north as the y-axis, and the z-axis forming a right-handed coordinate system with the x and y axes, pointing vertically upwards. The other is the gimbal's own carrier coordinate system Ox'y'z', denoted as {b}, also with the gimbal's center of mass as the origin, the x'-axis pointing to the right along the camera's vertical line of sight, the y'-axis pointing forward along the camera's line of sight, and the z'-axis forming a right-handed coordinate system with the x and y axes, pointing upwards. To obtain the attitude angles of the gimbal's coordinate system within the geographic coordinate system, corresponding attitude calculations are required. When using the Euler angle method, the gimbal coordinate system can be considered as obtained by rotating the geographic coordinate system around three axes: the angle of rotation around the z-axis is the heading angle α, the angle of rotation around the x-axis is the pitch angle β, and the angle of rotation around the y-axis is the roll angle γ. Assume that the coordinates of a point P in space in the geographic coordinate system are (x... o ,y o ,z o The coordinates of the ), in the gimbal coordinate system are (x' o ,y' o ,z' o Then, we can obtain the coordinate transformation around the x, y, and z axes. The rotation matrix from the geographic coordinate system {n} to the carrier coordinate system {b} where the gimbal is located is obtained. The rotation is performed in the order of the x-axis, y-axis, and z-axis. Different rotation orders will result in different coordinate transformations.
[0014] In step 1.1), the filtered three-axis attitude angle data is input into the Long Short-Term Memory (LSTM) neural network for training to obtain the optimal LSTM neural network. Specifically, the filtered three-axis attitude angle data is used as the training set and input into the LSTM neural network for training. In practice, the filtered three-axis attitude angle data is divided into training set and test set according to a preset ratio of 8:2 and input into the LSTM neural network for training and correction. When the number of training times reaches a preset upper limit or the loss function value of the LSTM neural network is lower than a preset threshold, the optimal weight parameters can be found, the training ends, and the optimal LSTM neural network is obtained.
[0015] Considering both prediction accuracy and training cost, the number of neurons in the hidden layer can be set to 64. During training, the initial values of the weight matrix are set to random samples from a normal distribution, and the initial bias is set to 0. The Adam learner is used for training, with a learning rate of 0.001. Considering the latency of the gimbal receiving commands, a Long Short-Term Memory (LSTM) neural network is used to predict the attitude angle changes approximately 1 second in advance, and a Gaussian filter can be used to slightly smooth the attitude angle curve.
[0016] Because there is a certain delay in the transmission of motor drive signals, a Long Short-Time Memory (LSTM) neural network is used to predict the three-axis attitude angles of the unmanned surface vessel (USV) to achieve real-time control. The LSTM neural network is a special variant of the RNN neural network, which modifies the neuron structure of the RNN and is mainly used to solve the problems of sequence memorization difficulties and gradient vanishing / exploding during long sequence training.
[0017] In step 1.3), the spherical gimbal includes a gimbal motor, a camera, a drive unit, and an angle sensor. The camera is mounted on the hull of the unmanned surface vessel via the gimbal motor, while the drive unit and angle sensor are installed inside the spherical gimbal. The fuzzy PID controller outputs a control signal to control the spherical gimbal. Specifically, the fuzzy PID controller outputs a control signal to the drive unit, which outputs a motor drive signal to control the movement of the gimbal motor, stabilizing the mechanical vibration experienced by the camera during shooting. The spherical gimbal detects its own angle in real time, specifically by using an angle sensor to detect the angle of the spherical gimbal in real time.
[0018] The three-axis attitude angles of the unmanned surface vessel (USV), predicted and smoothed by a Long Short-Term Memory (LSTM) neural network, will be used as control inputs for the image-stabilized gimbal system after attitude calculation. The image-stabilized gimbal system uses a fuzzy PID controller for control. Traditional PID control is simple in principle, easy to implement, and highly reliable, but the three parameters K of the PID controller... p K i K dOnce determined, the parameters of a PID controller cannot be changed during the control process, which greatly limits its application in nonlinear or time-varying systems. Fuzzy PID, based on traditional PID, dynamically adjusts the three parameters of the PID controller by adjusting the input error and the rate of change of the error, solving the problem of its inability to adjust in real time and optimizing the dynamic performance of the controller. When using a fuzzy controller, the input and output quantities must first be fuzzified; the choice of fuzzy set determines the controller's control accuracy. Considering both control accuracy and hardware performance requirements, the fuzzy subset can be divided into {negative large, negative medium, negative small, zero, positive small, positive medium, positive large} = {NB, NM, NS, ZO, PS, PM, PB}.
[0019] In step two, electronic image stabilization is used to process the real-time video captured by the spherical gimbal to obtain stabilized video, as detailed below:
[0020] 2.1) Capture video using a spherical gimbal and extract feature points from the video using the Scale-invariant feature transform (SIFT) algorithm, then perform feature point matching to obtain several sets of matching feature points.
[0021] 2.2) Based on the feature points matched in each group, the optimal motion model is obtained by using the Random Sample Consensus (RANSAC) algorithm or the improved RANSAC algorithm. The preset initial point of the spherical gimbal is input into the optimal motion model, and the optimal motion model outputs the motion trajectory of the spherical gimbal.
[0022] 2.3) The motion trajectory of the spherical gimbal is filtered by adaptive Gaussian filtering to obtain an adaptive smooth path. Motion compensation is then performed on the real-time video based on the adaptive smooth path to obtain a stabilized video, thus realizing electronic image stabilization processing of real-time video.
[0023] For image sequences acquired by unmanned surface vessel (USV) video sensors, an image stabilization algorithm based on adaptive Gaussian filtering is proposed for further stabilization. The SIFT algorithm is used to extract feature points from each image and perform feature point matching. An affine transformation model is selected as the camera's motion model, and the optimal model parameters are obtained using an improved RANSAC algorithm. After obtaining the camera's motion trajectory, the adaptive Gaussian filtering algorithm is used to filter out jitter while preserving the main camera motion, achieving electronic image stabilization.
[0024] In step 2.1), a spherical gimbal captures video and the Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points from the video and perform feature point matching to obtain several sets of matching feature points. Specifically, for each frame of the continuous frame images in the video captured by the spherical gimbal, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points from the image and perform feature point matching to obtain several sets of matching feature points.
[0025] In step 2.2), based on the matched feature points of each group, the optimal motion model is obtained using the Random Sample Consistent RANSAC algorithm or an improved version of the Random Sample Consistent RANSAC algorithm. Specifically, for each frame in the continuous frame images captured by the spherical gimbal, the matched feature points obtained from each image are processed using the Random Sample Consistent RANSAC algorithm. During processing, an affine transformation model is selected as the motion model for iteration. The optimal motion model is the model with the most inliers. If the Random Sample Consistent RANSAC algorithm fails to obtain the optimal motion model after a preset number of iterations N (i.e., the inlier rate is below 50% in each iteration), the iteration stops. The improved Random Sample Consistent RANSAC algorithm is then used to process the matched feature points obtained from each image, and the final motion model obtained is taken as the optimal motion model. The affine transformation model only considers three motion conditions: translation, rotation, and scaling. In this case, the affine transformation has four degrees of freedom, which simplifies the calculation. Improving the RANSAC algorithm's model quality assessment under low inlier rate conditions can lead to finding models that better describe camera motion.
[0026] The process of using the improved Random Sample Consistency (RANSAC) algorithm to process the feature points of each set of matching images involves first establishing a judgment target model. For each set of outliers obtained by iterating the RANSAC algorithm for a preset number of iterations N, the residual of each outlier before and after being input into the motion model obtained in the current iteration is calculated. The residual of each outlier is then input into the judgment target model, which outputs a judgment value. The motion model obtained when the judgment value is the smallest is selected as the optimal motion model.
[0027] The improved RANSAC algorithm can remove erroneous feature point pairs obtained by the SIFT algorithm. This can eliminate the possibility of incorrect matching when using the SIFT algorithm to match feature points between adjacent image frames, which can affect the parameter estimation of subsequent motion models, cause camera path calculation errors, and result in poor image stabilization.
[0028] When the inlier rate of the samples is high, the conventional RANSAC algorithm can be used to obtain a satisfactory motion model with fewer iterations. When the inlier rate of the samples is low, the improved RANSAC algorithm can eliminate the impact of the exponential increase in the required number of iterations as the inlier rate decreases, which reduces computational efficiency. The improved RANSAC algorithm can measure the quality of the motion model by counting the number of inliers when the inlier rate is low. However, this may result in more outliers deviating from the model, which may lead to the model failing to accurately describe the camera's motion state.
[0029] The established target judgment model is as follows:
[0030]
[0031]
[0032] r i >3d, (i=1,2,...,M)
[0033] Where, r i o′ represents the residual of the i-th outlier in the input, and M represents the total number of outliers in the input; ix and o′ iy Let x and y represent the x and y coordinates of the outgoing point in the image coordinate system obtained by substituting the i-th feature point in the current frame image into the motion model obtained in the current iteration. ix and o iy represents the x and y coordinates in the image coordinate system of the external point in the next frame that matches the i-th feature point in the current frame, respectively; d represents the Euclidean distance between the i-th feature point in the current frame and its matching feature point in the next frame.
[0034] The above formula can eliminate outliers that significantly deviate from the model. The smaller the sum of squares of the residuals of the outliers that meet the requirements, the closer these outliers are to the motion model. The motion model with the smallest sum of squares is selected as the optimal motion model.
[0035] In step 2.3), an adaptive smooth path is obtained by filtering the motion trajectory of the spherical gimbal using an adaptive Gaussian filter. Specifically, the standard deviation of the Gaussian distribution is first obtained based on the motion trajectory of the spherical gimbal. Then, the Gaussian template for the adaptive Gaussian filter is obtained based on the standard deviation of the Gaussian distribution and the camera's frame rate. Finally, an adaptive Gaussian filter is established. The specific adaptive Gaussian filter is as follows:
[0036]
[0037] Where G() represents the adaptive Gaussian filter; x represents the value of the Gaussian template, which is a one-dimensional array, and x takes integers between [-ff]; f represents the frame rate of the camera; σ represents the standard deviation of the Gaussian distribution, and σ represents the degree of dispersion of the data.
[0038] If σ is large, the difference between the center coefficient and the surrounding coefficients of the generated template is small, which can achieve a better smoothing effect on the image. However, when the camera motion vibration amplitude is large, a Gaussian template with a large σ value may lead to a reduction in image pixels. Conversely, when σ is small, the center coefficient of the Gaussian template is large and the surrounding coefficients are small, which can better preserve the main motion of the camera, but it is also prone to overfitting the path, resulting in poor image stabilization. The camera motion path is a set of one-dimensional arrays. When using the Gaussian function to filter a one-dimensional array, a one-dimensional Gaussian template must first be generated.
[0039] Finally, an adaptive Gaussian filtering algorithm is used to filter the motion trajectory of the spherical gimbal to obtain an adaptive smooth path. After applying the adaptive Gaussian filtering algorithm to filter the video path, image cropping, scaling, and other operations are then applied to obtain the stabilized video path.
[0040] Adaptive Gaussian filtering can remove oscillations from the path, retaining only the desired main motion to obtain stable video. Simultaneously, adaptive Gaussian filtering can selectively avoid smoothing some violent movements considered as the camera's main motion, preventing the reduction of pixel count and insufficient effective information that might result from smoothing violent movements. A Gaussian filter is a linear low-pass filter that attenuates high-frequency noise in the signal. Adaptive Gaussian filtering can adaptively apply the Gaussian filter, assigning different σ values to different levels of motion shakiness to achieve better image stabilization.
[0041] The process first obtains the standard deviation of the Gaussian distribution based on the motion trajectory of the spherical gimbal, and then obtains the Gaussian template for adaptive Gaussian filtering based on the standard deviation of the Gaussian distribution and the camera's frame rate. The specific standard deviation σ of the Gaussian distribution is as follows:
[0042] σ=K*(1-CV)+σ min
[0043]
[0044] X = {x1, x2, ..., x} f-1 ,x f ,x f+1 ,...,x Size}
[0045] Size = 2f + 1
[0046] Where K represents the scaling factor; CV represents the coefficient of variation, CV takes the minimum value in the range (CV,1), CV=min(CV,1); σ min S represents the minimum value of the standard deviation σ of the Gaussian distribution; S() represents the standard deviation; X represents the moving point x in the trajectory of the spherical gimbal at time t. f The position sequence one second before and after it, x1, x2, ..., x f-1 Let x represent the moving point in the trajectory of the spherical gimbal. f The positions of the 1st, 2nd, ..., f-1st elements in the previous second, x f+1 ,...,x Size Let x represent the moving point in the trajectory of the spherical gimbal. f The positions of the (f+1), ..., Size elements in the previous second; The point x in the trajectory of the spherical gimbal at time t represents the point of motion. f The mean of the position sequence one second before and after it.
[0047] If an image point exists within one second before or after a certain moment where it has not yet started moving or has stopped moving, then the position information at these moments can be filled in. Considering that zero-padding might cause the subsequent standard deviation calculation to be too large, affecting the selection of σ, the mean coordinates of all moving points along the motion path during the motion process are chosen as the filling content.
[0048] The coefficient of variation (CV) reflects the dispersion of data. The larger the coefficient, the greater the dispersion of the data. Normally, the CV value is between 0 and 1. However, in special cases, such as when the mean of the data is close to zero, the CV value can diverge outwards, and even small perturbations can have a significant impact on the CV. Limiting the value of the CV can eliminate this effect.
[0049] Since CV is positively correlated with the degree of data jitter, and σ should be smaller when jitter is severe to better preserve the subject's motion, σ and CV should be negatively correlated. K specifies the range of σ values, typically set to 10. Furthermore, to prevent σ from approaching 0 in extreme cases, a minimum value σ is added to each σ. min Generally, σ is taken. min =0.5. After calculating σ for each point of the trajectory, a Gaussian filter with a specific σ value is selected empirically to slightly smooth the trajectory, avoiding abrupt changes in the value of σ and improving the smoothing effect.
[0050] Filtering the camera position at each moment needs to take into account the positions of the camera before and after the moment. The size of the Gaussian template is limited by the camera's frame rate f to avoid the Gaussian template being too large, which would also increase the computational load.
[0051] The final Gaussian template is determined based on the size of the Gaussian template and the standard deviation σ of the Gaussian distribution.
[0052] The beneficial effects of this invention are:
[0053] This invention improves video image quality by processing the video from two aspects: image stabilization gimbal control and image stabilization algorithm. Mechanical image stabilization is achieved by controlling the movement of the image stabilization gimbal. Then, an image stabilization algorithm based on an improved RANSAC algorithm and adaptive Gaussian filtering is proposed to further stabilize the image sequence obtained by the camera, achieving electronic image stabilization. Attached Figure Description
[0054] Figure 1 This is a block diagram of the electromechanical integrated image stabilization system.
[0055] Figure 2 Diagram of LSTM neuron structure;
[0056] Figure 3 Here is a block diagram of the fuzzy PID control algorithm;
[0057] Figure 4 Flowchart of the improved RANSAC algorithm;
[0058] Figure 5 This is a comparison chart of the filtering results of the present invention, wherein, Figure 5 (a) is a comparison diagram of the vertical direction filtering results of the present invention. Figure 5 (b) is a comparison diagram of the horizontal filtering results of the present invention. Detailed Implementation
[0059] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0060] The image stabilization method for unmanned surface vessels based on electromechanical integration technology of the present invention includes the following steps:
[0061] Step 1: When the unmanned surface vessel equipped with a spherical gimbal is navigating, mechanical image stabilization is performed on the spherical gimbal.
[0062] In step one, mechanical image stabilization is used to stabilize the spherical gimbal, as detailed below:
[0063] 1.1) The three-axis attitude angle data of the unmanned surface vessel is collected by the inertial measurement unit (IMU). The three-axis attitude angle data is preprocessed by normalization, and then filtered by a Gaussian filter to obtain filtered three-axis attitude angle data. The filtered three-axis attitude angle data is input into the long short-term memory (LSTM) neural network for training to obtain the optimal LSTM neural network. Normalization preprocessing of the input data can eliminate the influence of the dimensions of the input attitude angles and speed up the training of the neural network.
[0064] In step 1.1), the filtered three-axis attitude angle data is input into the Long Short-Term Memory (LSTM) neural network for training to obtain the optimal LSTM neural network. Specifically, the filtered three-axis attitude angle data is used as the training set and input into the LSTM neural network for training. In practice, the filtered three-axis attitude angle data is divided into training set and test set according to a preset ratio of 8:2 and input into the LSTM neural network for training and correction. When the number of training times reaches a preset upper limit or the loss function value of the LSTM neural network is lower than a preset threshold, the optimal weight parameters can be found, the training ends, and the optimal LSTM neural network is obtained.
[0065] Considering both prediction accuracy and training cost, the number of neurons in the hidden layer can be set to 64. During training, the initial values of the weight matrix are set to random samples from a normal distribution, and the initial bias is set to 0. The Adam learner is used for training, with a learning rate of 0.001. Considering the latency of the gimbal receiving commands, a Long Short-Term Memory (LSTM) neural network is used to predict the attitude angle changes approximately 1 second in advance, and a Gaussian filter can be used to slightly smooth the attitude angle curve.
[0066] Because there is a certain delay in the transmission of motor drive signals, a Long Short-Time Memory (LSTM) neural network is used to predict the three-axis attitude angles of the unmanned surface vessel (USV) to achieve real-time control. The LSTM neural network is a special variant of the RNN neural network, which modifies the neuron structure of the RNN and is mainly used to solve the problems of sequence memorization difficulties and gradient vanishing / exploding during long sequence training.
[0067] 1.2) When the unmanned surface vessel is navigating, the three-axis attitude angle data of the unmanned surface vessel are collected in real time by the inertial measurement unit (IMU) and input into the optimal long short-term memory (LSTM) neural network. The optimal LSTM neural network outputs the predicted three-axis attitude angle data of the unmanned surface vessel.
[0068] 1.3) The predicted three-axis attitude angle data of the unmanned surface vessel is used to calculate the attitude using the Euler angle method and then input into a constrained fuzzy PID controller. The fuzzy PID controller outputs control signals to control the spherical gimbal. The spherical gimbal detects its own angle in real time and inputs it into the fuzzy PID controller for closed-loop control, thereby realizing the mechanical image stabilization of the spherical gimbal.
[0069] Attitude calculation can also be performed using methods such as direction cosine and quaternions. When using the Euler angle method, quaternions can be used for further optimization. When the stabilized gimbal operates on an unmanned surface vessel (USV), it operates in two coordinate systems. One is the geographic coordinate system Oxyz, denoted as {n}, with the gimbal's center of mass as the origin, east as the x-axis, north as the y-axis, and the z-axis forming a right-handed coordinate system with the x and y axes, pointing vertically upwards. The other is the gimbal's own carrier coordinate system Ox'y'z', denoted as {b}, also with the gimbal's center of mass as the origin, the x'-axis pointing to the right along the camera's vertical line of sight, the y'-axis pointing forward along the camera's line of sight, and the z'-axis forming a right-handed coordinate system with the x and y axes, pointing upwards. To obtain the attitude angles of the gimbal's coordinate system within the geographic coordinate system, corresponding attitude calculations are required. When using the Euler angle method, the gimbal coordinate system can be considered as obtained by rotating the geographic coordinate system around three axes: the angle of rotation around the z-axis is the heading angle α, the angle of rotation around the x-axis is the pitch angle β, and the angle of rotation around the y-axis is the roll angle γ. Assume that the coordinates of a point P in space in the geographic coordinate system are (xo, yo, z). o The coordinates of the ), in the gimbal coordinate system are (x' o ,y' o ,z' o Then, we can obtain the coordinate transformation around the x, y, and z axes. The rotation matrix from the geographic coordinate system {n} to the carrier coordinate system {b} where the gimbal is located is obtained. The rotation is performed in the order of the x-axis, y-axis, and z-axis. Different rotation orders will result in different coordinate transformations.
[0070] In step 1.3), the spherical gimbal includes a gimbal motor, a camera, a drive unit, and an angle sensor. The camera is mounted on the hull of the unmanned surface vessel via the gimbal motor, while the drive unit and angle sensor are installed inside the spherical gimbal. The fuzzy PID controller outputs a control signal to control the spherical gimbal. Specifically, the fuzzy PID controller outputs a control signal to the drive unit, which outputs a motor drive signal to control the movement of the gimbal motor, stabilizing the mechanical vibration experienced by the camera during shooting. The spherical gimbal detects its own angle in real time, specifically through the angle sensor.
[0071] The three-axis attitude angles of the unmanned surface vessel (USV), predicted and smoothed by a Long Short-Term Memory (LSTM) neural network, will be used as control inputs for the image-stabilized gimbal system after attitude calculation. The image-stabilized gimbal system uses a fuzzy PID controller for control. Traditional PID control is simple in principle, easy to implement, and highly reliable, but the three parameters K of the PID controller... p K i K dOnce determined, the parameters of a PID controller cannot be changed during the control process, which greatly limits its application in nonlinear or time-varying systems. Fuzzy PID, based on traditional PID, dynamically adjusts the three parameters of the PID controller by adjusting the input error and the rate of change of the error, solving the problem of its inability to adjust in real time and optimizing the dynamic performance of the controller. When using a fuzzy controller, the input and output quantities must first be fuzzified; the choice of fuzzy set determines the controller's control accuracy. Considering both control accuracy and hardware performance requirements, the fuzzy subset can be divided into {negative large, negative medium, negative small, zero, positive small, positive medium, positive large} = {NB, NM, NS, ZO, PS, PM, PB}.
[0072] Step 2: Capture real-time video of the unmanned surface vessel (USV) while it is sailing using a spherical gimbal. Then, use electronic image stabilization to process the real-time video captured by the spherical gimbal to obtain a stabilized video, thus stabilizing the images captured by the USV.
[0073] In step two, electronic image stabilization is used to process the real-time video captured by the spherical gimbal to obtain stabilized video, as detailed below:
[0074] 2.1) Capture video using a spherical gimbal and extract feature points from the video using the Scale-invariant feature transform (SIFT) algorithm, then perform feature point matching to obtain several sets of matching feature points.
[0075] In step 2.1), video is captured using a spherical gimbal, and feature points in the video are extracted using the Scale Invariant Feature Transform (SIFT) algorithm. Feature point matching is then performed to obtain several sets of matching feature points. Specifically, for each frame of the continuous frame images in the video captured by the spherical gimbal, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points in the image and feature point matching is performed to obtain several sets of matching feature points.
[0076] In step 2.2), based on the matched feature points of each group, the optimal motion model is obtained using the Random Sample Consistent RANSAC algorithm or an improved version of the Random Sample Consistent RANSAC algorithm. Specifically, for each frame in the continuous frame images captured by the spherical gimbal, the matched feature points obtained from each image are processed using the Random Sample Consistent RANSAC algorithm. During processing, an affine transformation model is selected as the motion model for iteration. The optimal motion model is the model with the most inliers. If the Random Sample Consistent RANSAC algorithm fails to obtain the optimal motion model after a preset number of iterations N (i.e., the inlier rate is below 50% in each iteration), the iteration stops. The improved Random Sample Consistent RANSAC algorithm is then used to process the matched feature points obtained from each image, and the final motion model obtained is taken as the optimal motion model. The affine transformation model only considers three motion conditions: translation, rotation, and scaling. In this case, the affine transformation has four degrees of freedom, which simplifies the calculation. Improving the RANSAC algorithm's model quality assessment under low inlier rate conditions can lead to finding models that better describe camera motion.
[0077] The improved Random Sample Consistency (RANSAC) algorithm is used to process the feature points of each set of matching obtained from the image. Specifically, a judgment target model is first established. For all outliers obtained by iterating the RANSAC algorithm for a preset number of iterations N, the residual of each outlier before and after being substituted into the motion model obtained in the current iteration is calculated. The residual of each outlier is input into the judgment target model, the judgment target model outputs the judgment value, and the motion model obtained by the iteration with the smallest judgment value is selected as the optimal motion model.
[0078] The improved RANSAC algorithm can remove erroneous feature point pairs obtained by the SIFT algorithm. This can eliminate the possibility of incorrect matching when using the SIFT algorithm to match feature points between adjacent image frames, which can affect the parameter estimation of subsequent motion models, cause camera path calculation errors, and result in poor image stabilization.
[0079] When the inlier rate of the samples is high, the conventional RANSAC algorithm can be used to obtain a satisfactory motion model with fewer iterations. When the inlier rate of the samples is low, the improved RANSAC algorithm can eliminate the impact of the exponential increase in the required number of iterations as the inlier rate decreases, which reduces computational efficiency. The improved RANSAC algorithm can measure the quality of the motion model by counting the number of inliers when the inlier rate is low. However, this may result in more outliers deviating from the model, which may lead to the model failing to accurately describe the camera's motion state.
[0080] The established judgment target model is as follows:
[0081]
[0082]
[0083] r i >3d, (i=1,2,...,M)
[0084] Where, r i o′ represents the residual of the i-th outlier in the input, and M represents the total number of outliers in the input; ix and o′ iy Let x and y represent the x and y coordinates of the outgoing point in the image coordinate system obtained by substituting the i-th feature point in the current frame image into the motion model obtained in the current iteration. ix and o iy represents the x and y coordinates in the image coordinate system of the external point in the next frame that matches the i-th feature point in the current frame, respectively; d represents the Euclidean distance between the i-th feature point in the current frame and its matching feature point in the next frame.
[0085] The above formula can eliminate outliers that significantly deviate from the model. The smaller the sum of squares of the residuals of the outliers that meet the requirements, the closer these outliers are to the motion model. The motion model with the smallest sum of squares is selected as the optimal motion model.
[0086] 2.2) Based on the feature points matched in each group, the optimal motion model is obtained by using the Random Sample Consensus (RANSAC) algorithm or the improved RANSAC algorithm. The preset initial point of the spherical gimbal is input into the optimal motion model, and the optimal motion model outputs the motion trajectory of the spherical gimbal.
[0087] 2.3) The motion trajectory of the spherical gimbal is filtered by adaptive Gaussian filtering to obtain an adaptive smooth path. Motion compensation is then performed on the real-time video based on the adaptive smooth path to obtain a stabilized video, thus realizing electronic image stabilization processing of real-time video.
[0088] For image sequences acquired by unmanned surface vessel (USV) video sensors, an image stabilization algorithm based on adaptive Gaussian filtering is proposed for further stabilization. The SIFT algorithm is used to extract feature points from each image and perform feature point matching. An affine transformation model is selected as the camera's motion model, and the optimal model parameters are obtained using an improved RANSAC algorithm. After obtaining the camera's motion trajectory, the adaptive Gaussian filtering algorithm is used to filter out jitter while preserving the main camera motion, achieving electronic image stabilization.
[0089] In step 2.3), an adaptive Gaussian filter is used to filter the motion trajectory of the spherical gimbal to obtain an adaptive smooth path. Specifically, the standard deviation of the Gaussian distribution is first obtained based on the motion trajectory of the spherical gimbal. Then, the Gaussian template for the adaptive Gaussian filter is obtained based on the standard deviation of the Gaussian distribution and the camera's frame rate. Finally, an adaptive Gaussian filter is established. The specific adaptive Gaussian filter is as follows:
[0090]
[0091] Where G() represents the adaptive Gaussian filter; x represents the value of the Gaussian template, which is a one-dimensional array, and x takes integers between [-ff]; f represents the frame rate of the camera; σ represents the standard deviation of the Gaussian distribution, and σ represents the degree of dispersion of the data.
[0092] If σ is large, the difference between the center coefficient and the surrounding coefficients of the generated template is small, which can achieve a better smoothing effect on the image. However, when the camera motion vibration amplitude is large, a Gaussian template with a large σ value may lead to a reduction in image pixels. Conversely, when σ is small, the center coefficient of the Gaussian template is large and the surrounding coefficients are small, which can better preserve the main motion of the camera, but it is also prone to overfitting the path, resulting in poor image stabilization. The camera motion path is a set of one-dimensional arrays. When using the Gaussian function to filter a one-dimensional array, a one-dimensional Gaussian template must first be generated.
[0093] Finally, an adaptive Gaussian filtering algorithm is used to filter the motion trajectory of the spherical gimbal to obtain an adaptive smooth path. After applying the adaptive Gaussian filtering algorithm to filter the video path, image cropping, scaling, and other operations are then applied to obtain the stabilized video path.
[0094] Adaptive Gaussian filtering can remove oscillations from the path, retaining only the desired main motion to obtain stable video. Simultaneously, adaptive Gaussian filtering can selectively avoid smoothing some violent movements considered as the camera's main motion, preventing the reduction of pixel count and insufficient effective information that might result from smoothing violent movements. A Gaussian filter is a linear low-pass filter that attenuates high-frequency noise in the signal. Adaptive Gaussian filtering can adaptively apply the Gaussian filter, assigning different σ values to different levels of motion shakiness to achieve better image stabilization.
[0095] First, the standard deviation of the Gaussian distribution is obtained based on the motion trajectory of the spherical gimbal. Then, an adaptive Gaussian filter Gaussian template is obtained based on the standard deviation of the Gaussian distribution and the camera's frame rate. The specific standard deviation σ of the Gaussian distribution is as follows:
[0096] σ=K*(1-CV)+σ min
[0097]
[0098] X = {x1, x2, ..., x} f-1 ,x f ,x f+1 ,...,x Size}
[0099] Size = 2f + 1
[0100] Where K represents the scaling factor; CV represents the coefficient of variation, CV takes the minimum value in the range (CV,1), CV=min(CV,1); σ min S represents the minimum value of the standard deviation σ of the Gaussian distribution; S() represents the standard deviation; X represents the moving point x in the trajectory of the spherical gimbal at time t. f The position sequence one second before and after it, x1, x2, ..., x f-1 Let x represent the moving point in the trajectory of the spherical gimbal. f The positions of the 1st, 2nd, ..., f-1st elements in the previous second, x f+1 ,...,x Size Let x represent the moving point in the trajectory of the spherical gimbal. f The positions of the (f+1), ..., Size elements in the previous second; The point x in the trajectory of the spherical gimbal at time t represents the point of motion. f The mean of the position sequence one second before and after it.
[0101] If an image point exists within one second before or after a certain moment where it has not yet started moving or has stopped moving, then the position information at these moments can be filled in. Considering that zero-padding might cause the subsequent standard deviation calculation to be too large, affecting the selection of σ, the mean coordinates of all moving points along the motion path during the motion process are chosen as the filling content.
[0102] The coefficient of variation (CV) reflects the dispersion of data. The larger the coefficient, the greater the dispersion of the data. Normally, the CV value is between 0 and 1. However, in special cases, such as when the mean of the data is close to zero, the CV value can diverge outwards, and even small perturbations can have a significant impact on the CV. Limiting the value of the CV can eliminate this effect.
[0103] Since CV is positively correlated with the degree of data jitter, and σ should be smaller when jitter is severe to better preserve the subject's motion, σ and CV should be negatively correlated. K specifies the range of σ values, typically set to 10. Furthermore, to prevent σ from approaching 0 in extreme cases, a minimum value σ is added to each σ. min Generally, σ is taken. min=0.5. After calculating σ for each point of the trajectory, a Gaussian filter with a specific σ value is selected empirically to slightly smooth the trajectory, avoiding abrupt changes in the value of σ and improving the smoothing effect.
[0104] Filtering the camera position at each moment needs to take into account the positions of the camera before and after the moment. The size of the Gaussian template is limited by the camera's frame rate f to avoid the Gaussian template being too large, which would also increase the computational load.
[0105] The final Gaussian template is determined based on the size of the Gaussian template and the standard deviation σ of the Gaussian distribution.
[0106] like Figure 1 The diagram shows the overall block diagram of the electromechanical image stabilization system of this invention, which consists of two parts: mechanical image stabilization gimbal control and electronic image stabilization. The overall structure of the image stabilization gimbal comprises an inertial measurement unit (IMU), a fuzzy PID controller, and a drive motor. The IMU collects three-axis attitude angle data of the unmanned surface vessel (USV), and after LSTM neural network prediction and attitude calculation, it is input into a constrained fuzzy PID control algorithm. The control result is converted into a motor drive signal to control the movement of the gimbal motor, thereby stabilizing the mechanical shaking experienced by the camera. Simultaneously, the video acquired by the gimbal camera after mechanical stabilization is further subjected to electronic algorithm stabilization, including motion estimation, motion filtering, and motion compensation. The SIFT algorithm is used to extract feature points from each image and perform feature point matching. An affine transformation model is selected as the camera's motion model, and the improved RANSAC algorithm is used to solve for the optimal model parameters. After obtaining the camera's motion trajectory, an adaptive Gaussian filtering algorithm is used to filter out jitter and retain the main motion of the camera. For the blank areas in the filtered image, appropriate frame cropping and image compensation can be performed, such as using bilinear interpolation to compensate for the image, in order to achieve electronic image stabilization.
[0107] like Figure 2 The diagram shown illustrates the structure of an LSTM neuron, including the forgetting gate, input gate, and output gate. The forgetting gate represents the degree of forgetting in short-term memory, taking the input x at time t as an example. t and the hidden state h at time t-1 t-1 The input is fed into the forget gate, mapped to the range (0,1) by the sigmoid function, and then input into cell c. t-1 In this context, the closer the value is to 0, the higher the degree of forgetting; 0 represents complete forgetting, and 1 represents complete memory. The input gate is responsible for determining the weight i of the incoming new memory information. t On the other hand, it is responsible for generating new memory cell information. The output gate is used to output the new hidden state of the LSTM neuron.
[0108] like Figure 3The diagram shown is a block diagram of a fuzzy PID controller. The fuzzy controller uses a two-input, three-output logic design, with the rates of change of error e and error ec as inputs and the increments ΔK of the three PID controller parameters as inputs. p ΔK i ΔK d As the output of the fuzzy controller, the control input r(t) of the image-stabilized gimbal control system is the desired two-axis attitude angle of the gimbal, and the control output y(t) is the actual output attitude angle, where e is the angle error and ec is the rate of change of the angle error. During each adjustment, the angles at the current and previous moments are collected by sensors, and the different values of e and ec are input into the fuzzy PID controller. Based on fuzzy rules, the compensation values of the three PID parameters are adjusted in real time, and the output of the PID controller is converted into drive signals for the servo motor. These signals are then used to control the motor's rotation via a bus, adjusting the gimbal's attitude angle and achieving mechanical image stabilization.
[0109] like Figure 4 The diagram shows the flowchart of the improved RANSAC algorithm. The specific implementation steps of the RANSAC algorithm are as follows: 1) First, select different motion models according to different application scenarios. For example, to achieve line fitting, two pairs of feature points are needed. For camera motion, we choose an affine motion model for fitting. To simplify the calculation, only three motion conditions are considered: translation, rotation, and scaling. At this time, the affine transformation has four degrees of freedom. Extract four sets of feature points as initial values and substitute them into the model to obtain the initial motion parameters. 2) Apply the obtained affine model to all feature point pairs and calculate the Euclidean distance between the feature point pairs before and after substitution. When the Euclidean distance is less than the set threshold, the feature point is considered an interior point; otherwise, it is an exterior point. Calculate the interior point rate of the feature points. 3) When the interior point rate is greater than 50%, use the conventional model discrimination method to optimize the motion model, that is, obtain a better motion model by comparing the number of interior points that conform to different models. Conversely, when the interior point rate is less than 50%, use the improved model discrimination method to optimize the motion model by minimizing the sum of squares of the residuals of the exterior points that meet the requirements. 4) When the number of iterations exceeds the set value N or the current model meets the requirements, output the current optimal model as the final motion model; otherwise, repeat steps 1) to 3) until the loop is exited.
[0110] like Figure 5 (a) and Figure 5 As shown in (b), this image compares the vertical and horizontal motion paths of a video segment with their filtering results. It can be seen from the image that our proposed algorithm effectively filters out jitter in the path while preserving the camera's main motion, thus achieving a better image stabilization effect.
[0111] This invention's electromechanical image stabilization technology consists of two parts: image stabilization gimbal control and electronic image stabilization. The overall structure of the image stabilization gimbal includes an inertial measurement unit (IMU), a fuzzy PID controller, and a drive motor. The IMU collects three-axis attitude angle data of the unmanned surface vessel (USV), which is then predicted and calculated using an LSTM neural network and input into a constrained fuzzy PID control algorithm. The control result is converted into a motor drive signal to control the movement of the gimbal motor, thereby stabilizing the mechanical shaking experienced by the camera. Simultaneously, the video captured by the gimbal camera after mechanical image stabilization is further subjected to electronic image stabilization, including motion estimation, motion filtering, and motion compensation. The SIFT algorithm is used to extract feature points from each image and perform feature point matching. An affine transformation model is selected as the camera's motion model, and an improved RANSAC algorithm is used to solve for the optimal model parameters. After obtaining the camera's motion trajectory, an adaptive Gaussian filtering algorithm is used to filter out the shaking while retaining the main motion of the camera, thus achieving electronic image stabilization.
Claims
1. An image stabilization method for unmanned surface vessels based on electromechanical integration technology, characterized in that: The method includes the following steps: Step 1: When the unmanned surface vessel equipped with a spherical gimbal is navigating, mechanical image stabilization is performed on the spherical gimbal using the mechanical image stabilization method; Step 2: Capture real-time video of the unmanned surface vessel (USV) during its navigation using a spherical gimbal. Then, apply electronic image stabilization to the real-time video captured by the spherical gimbal to obtain stabilized video, thus stabilizing the images captured by the USV. ; In step two, electronic image stabilization is used to process the real-time video captured by the spherical gimbal to obtain stabilized video, as detailed below: 2.1) Capture video using a spherical gimbal and extract feature points from the video using the Scale Invariant Feature Transform (SIFT) algorithm, then perform feature point matching to obtain several sets of matched feature points; 2.2) Based on the feature points matched in each group, the optimal motion model is obtained by using the Random Sample Consensus (RANSAC) algorithm or the improved RANSAC algorithm. The preset initial point of the spherical gimbal is input into the optimal motion model, and the optimal motion model outputs the motion trajectory of the spherical gimbal. 2.3) The motion trajectory of the spherical gimbal is filtered by adaptive Gaussian filtering to obtain an adaptive smooth path. Motion compensation is then performed on the real-time video based on the adaptive smooth path to obtain a stabilized video, thus realizing electronic image stabilization processing of real-time video. In step 2.1), a spherical gimbal is used to capture video, and the scale-invariant feature transform (SIFT) algorithm is used to extract feature points from the video and perform feature point matching to obtain several sets of matching feature points. Specifically, for each frame of the continuous frame images in the video captured by the spherical gimbal, the scale-invariant feature transform (SIFT) algorithm is used to extract feature points from the image and perform feature point matching to obtain several sets of matching feature points. In step 2.2), based on the matched feature points of each group, the optimal motion model is obtained by using the Random Sample Consistent RANSAC algorithm or an improved version of the Random Sample Consistent RANSAC algorithm. Specifically, for each frame in the continuous frame images captured by the spherical gimbal, the matched feature points obtained from each image are processed using the Random Sample Consistent RANSAC algorithm. During processing, an affine transformation model is selected as the motion model for iterative processing. The optimal motion model is obtained using the Random Sample Consistent RANSAC algorithm. If the optimal motion model is not obtained after a preset number of iterations N (i.e., the inlier rate is less than 50% in each iteration), the iteration stops. The improved version of the Random Sample Consistent RANSAC algorithm is then used to process the matched feature points obtained from each image, and the final motion model obtained is taken as the optimal motion model. ; The process of using the improved Random Sample Consistency (RANSAC) algorithm to process the feature points of each set of matching images involves first establishing a judgment target model. For each set of outliers obtained by iterating the RANSAC algorithm for a preset number of iterations N, the residual of each outlier before and after being input into the motion model obtained in the current iteration is calculated. The residual of each outlier is then input into the judgment target model, which outputs a judgment value. The motion model obtained when the judgment value is the smallest is selected as the optimal motion model.
2. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 1, characterized in that: In step one, mechanical image stabilization is used to stabilize the spherical gimbal, as detailed below: 1.1) The three-axis attitude angle data of the unmanned surface vessel are collected by the inertial measurement unit (IMU), the three-axis attitude angle data are normalized and preprocessed, and then filtered by a Gaussian filter to obtain filtered three-axis attitude angle data. The filtered three-axis attitude angle data are input into the long short-term memory (LSTM) neural network for training to obtain the optimal LSTM neural network. 1.2) When the unmanned surface vessel is navigating, the three-axis attitude angle data of the unmanned surface vessel are collected in real time by the inertial measurement unit (IMU) and input into the optimal long short-term memory (LSTM) neural network. The optimal long short-term memory (LSTM) neural network outputs the predicted three-axis attitude angle data of the unmanned surface vessel. 1.3) The predicted three-axis attitude angle data of the unmanned surface vessel is calculated using the Euler angle method and then input into the fuzzy PID controller. The fuzzy PID controller outputs control signals to control the spherical gimbal. The spherical gimbal detects its own angle in real time and inputs it into the fuzzy PID controller for closed-loop control, thereby realizing the mechanical image stabilization of the spherical gimbal.
3. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 2, characterized in that: In step 1.1), the filtered three-axis attitude angle data is input into the Long Short-Term Memory (LSTM) neural network for training to obtain the optimal LSTM neural network. Specifically, the filtered three-axis attitude angle data is used as the training set to input into the LSTM neural network for training. When the number of training iterations reaches a preset upper limit or the loss function value of the LSTM neural network is lower than a preset threshold, the training ends and the optimal LSTM neural network is obtained.
4. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 2, characterized in that: In step 1.3), the spherical gimbal includes a gimbal motor, a camera, a drive unit, and an angle sensor. The camera is mounted on the hull of the unmanned surface vessel via the gimbal motor, while the drive unit and angle sensor are installed inside the spherical gimbal. The fuzzy PID controller outputs a control signal to control the spherical gimbal, specifically by outputting a control signal to the drive unit, which then outputs a motor drive signal to control the movement of the gimbal motor. The spherical gimbal detects its own angle in real time, specifically by using an angle sensor to detect its own angle in real time.
5. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 1, characterized in that: The established target judgment model is as follows: in, Let M represent the residual of the i-th outlier point in the input, and M represent the total number of outliers in the input. and Let x and y represent the x and y coordinates of the outgoing point in the image coordinate system obtained by substituting the i-th feature point in the current frame image into the motion model obtained in the current iteration. and represents the x and y coordinates in the image coordinate system of the external point in the next frame that matches the i-th feature point in the current frame, respectively; d represents the Euclidean distance between the i-th feature point in the current frame and its matching feature point in the next frame.
6. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 1, characterized in that: In step 2.3), an adaptive smooth path is obtained by filtering the motion trajectory of the spherical gimbal using an adaptive Gaussian filter. Specifically, the standard deviation of the Gaussian distribution is first obtained based on the motion trajectory of the spherical gimbal. Then, the Gaussian template for the adaptive Gaussian filter is obtained based on the standard deviation of the Gaussian distribution and the camera's frame rate. Finally, an adaptive Gaussian filter is established. The specific adaptive Gaussian filter is as follows: in, σ represents an adaptive Gaussian filter; x represents the value of the Gaussian template, which is a one-dimensional array, and x takes integer values between [-ff]; f represents the camera's frame rate; σ represents the standard deviation of the Gaussian distribution. Finally, an adaptive smooth path is obtained by filtering the motion trajectory of the spherical gimbal using an adaptive Gaussian filtering algorithm.
7. The image stabilization method for unmanned surface vessels based on electromechanical integration technology according to claim 6, characterized in that: The process first obtains the standard deviation of the Gaussian distribution based on the motion trajectory of the spherical gimbal, and then obtains the Gaussian template for adaptive Gaussian filtering based on the standard deviation of the Gaussian distribution and the camera's frame rate. The specific standard deviation σ of the Gaussian distribution is as follows: Where K represents the scaling factor; CV represents the coefficient of variation, and CV takes the value of K. The minimum value within the range; The minimum value of the standard deviation σ of the Gaussian distribution is represented. X represents the standard deviation; X represents the point of motion in the trajectory of the spherical gimbal at time t. The position sequence one second before and after it, These represent the points of motion in the trajectory of the spherical gimbal. The positions of the 1st, 2nd, ..., f-1st elements in the previous second. These represent the points of motion in the trajectory of the spherical gimbal. The positions of the (f+1), ..., Size elements in the previous second; This represents the point of motion in the trajectory of the spherical gimbal at time t. The mean of the position sequence one second before and after it; The final Gaussian template is determined based on the size of the Gaussian template and the standard deviation σ of the Gaussian distribution.
