An unmanned ship real-time video stabilization method based on XFeat

Abstract

The application discloses an unmanned ship real-time video stabilization method based on XFeat, which comprises the following steps: segmenting a dynamic water surface area from an image and generating a water surface mask; extracting feature points from a non-water surface area outside the image water surface mask by using an XFeat method based on deep learning; re-distributing the extracted feature points in space by using a square continuous clustering algorithm; estimating the motion of a camera between continuous video frames by using the homogenized feature points and constructing a global motion trajectory; filtering out high-frequency jitter in the estimated camera motion trajectory and retaining effective motion of the camera; and applying the smoothed motion trajectory to an original video sequence to generate a stable video frame sequence. The invalid feature points of the dynamic water surface area are removed by the multi-feature fusion mask module, the stability of feature point extraction and tracking is improved, and the failure problem of traditional methods in a weak texture and a dynamic background is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) video stabilization technology, and more specifically, to a real-time video stabilization method for USVs based on XFeat. Background Technology

[0002] In recent years, unmanned surface vessels (USVs) have been increasingly used in tasks such as ranging, positioning, and maritime rescue. Their onboard cameras, as core sensors for acquiring environmental information, are playing an increasingly important role. However, USV videos often experience severe shaking due to external disturbances during navigation, such as wave impact, propeller vibration, and strong winds. This not only leads to an unpleasant visual experience but also seriously affects the stability and accuracy of subsequent tasks such as visual ranging, target recognition, and tracking. Therefore, video stabilization, as a key visual preprocessing task, is crucial for improving the overall reliability of unmanned surface vessel vision systems.

[0003] To address the issues of unstable feature extraction in traditional methods and high computational complexity in deep learning methods, it is of great significance to propose a video stabilization method that integrates traditional methods and deep learning. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a real-time video stabilization method for unmanned vessels based on XFeat.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A real-time video stabilization method for unmanned surface vessels based on XFeat includes the following steps:

[0007] Step S1: Segment the dynamic water surface region from the images of the water surface scene video taken by the unmanned vessel and generate a water surface mask;

[0008] Step S2: The XFeat method based on deep learning is used to extract feature points from the non-water surface region outside the water surface mask in the image.

[0009] Step S3: The extracted feature points are spatially redistributed using a square continuous clustering algorithm to simultaneously ensure both the quality of the feature points and the balance of their spatial distribution.

[0010] Step S4: Use the homogenized feature points to estimate the camera's motion between consecutive video frames and construct the global motion trajectory.

[0011] Step S5: Filter out high-frequency jitter in the estimated camera motion trajectory and retain the effective motion actively performed by the camera;

[0012] Step S6: Apply the smoothed motion trajectory to the original video sequence to generate a stable video frame sequence.

[0013] Furthermore, step S1 includes the following steps:

[0014] Step S11: Convert the input RGB image to the HSV color space and perform preliminary segmentation using the specific color distribution of the water surface area in the HSV color space.

[0015] Step S12: Divide the image into a grid and mark high-probability water surface areas based on the characteristics of unmanned surface vessel photography;

[0016] Step S13: Construct a water likelihood probability model based on HSV channel values, calculate the probability that each pixel belongs to the water surface, and generate a color mask.

[0017] Step S14: Calculate the gradient magnitude using the Sobel operator on the grayscale image, and set a low gradient threshold to generate a low gradient mask.

[0018] Step S15: The color mask and gradient mask are fused to obtain a combined mask, and morphological closing and opening operations are used for optimization to obtain a water surface mask.

[0019] Furthermore, in step S13, the generation of the color mask includes the following steps:

[0020] Step S131: Calculate the HSV mean for each grid using the following formula:

[0021]

[0022] In the formula, m and n are the row and column indices of the grid, respectively. The location of the grid cell; , and They represent the locations respectively The average hue, saturation, and highlight values ​​of the grid; For pixels within the grid; and These are the grid height and width, respectively.

[0023] Step S132: Based on the physical likelihood of water surface pixels within the same water area, construct a water likelihood probability model:

[0024]

[0025] In the formula, It is the sigmoid function; , , and All are fitted parameters; Represents all pixels The probability of water surface;

[0026] Step S133: Obtain the upper and lower bound thresholds of HSV for water surface pixels by combining the water likelihood probability with thresholding, and then define the water surface pixel set C:

[0027]

[0028] In the formula, , and Representing pixels Saturation, brightness, and hue components in the HSV color space; and These are the upper and lower bound thresholds for saturation, respectively. and These are the upper and lower bound thresholds for brightness, respectively. and These are the upper and lower thresholds for the hue, respectively;

[0029] Step S134: Generate a color mask :

[0030] .

[0031] Furthermore, in step S14, the low gradient mask Generate using the following formula:

[0032]

[0033] In the formula, Indicates at pixel point Sobel gradient vector at point; This represents the L2 norm, which is the gradient magnitude. This is a low gradient threshold.

[0034] Furthermore, in step S15, the mask is assembled. Generate using the following formula:

[0035]

[0036] Water surface mask Generate using the following formula:

[0037]

[0038] In the formula, For morphological closing operations; This is an opening operation.

[0039] Furthermore, step S3 includes the following steps:

[0040] Step S31: Sort all candidate feature points according to their response level;

[0041] Step S32: Use a dynamically resizable square window to iteratively select the feature points with the strongest response within the window, and use binary search to dynamically determine the appropriate window size;

[0042] Step S33 is repeated until the number of selected feature points reaches the preset target range.

[0043] Furthermore, step S4 includes the following steps:

[0044] Step S41: The LK pyramid optical flow method is used to perform forward tracking and reverse optical flow verification on the feature points in order to determine the consistency of the feature point matching.

[0045] Step S42: Use the RANSAC algorithm to estimate the fundamental matrix of the feature points that have passed the consistency verification, and retain the feature point pairs with high confidence.

[0046] Step S43: Fit the affine transformation matrix using high-confidence feature points, and construct the camera's global motion trajectory by accumulating the results frame by frame.

[0047] Furthermore, step S43 includes the following steps:

[0048] Step S431: Utilize feature points that match the previous frame and the current frame. and The position information is used to perform affine matrix fitting on the feature points using the following formula to estimate the translation and rotation transformations between consecutive frames;

[0049]

[0050] In the formula, H is the affine transformation matrix, defined as follows:

[0051]

[0052] In the formula, , This is the translation amount; The rotation angle;

[0053] Step S432, using the inter-frame affine transformation matrix Multiply by the product to construct the global motion trajectory of the camera:

[0054]

[0055] In the formula, Let be the affine transformation matrix from frame n to frame n+1; This is the cumulative motion transformation matrix from frame 1 to frame n.

[0056] Furthermore, step S5 includes the following steps:

[0057] Step S51: A covariance model of state transition and observation error is established using Kalman filtering, and the global motion trajectory of the camera is optimized through the prediction and update stages.

[0058] Step S52: After Kalman filtering, a one-dimensional time-domain Gaussian filter is applied to perform weighted convolution on each dimension of the trajectory to enhance local smoothness.

[0059] Furthermore, step S6 includes the following steps:

[0060] Step S61: Calculate the smoothed affine transformation matrix corresponding to each frame based on the trajectory smoothing result, and apply the smoothed affine transformation matrix to the original video frame image to perform image remapping and registration.

[0061] Step S62: Crop the image edges to remove the black borders caused by the transformation, and generate a stable video without black borders.

[0062] The beneficial effects of this invention are:

[0063] 1. This invention removes invalid feature points in dynamic water surface areas by using a multi-feature fusion mask module, thereby improving the stability of feature point extraction and tracking and effectively solving the failure problem of traditional methods in weak textures and dynamic backgrounds.

[0064] 2. This invention uses the XFeat neural network combined with the pyramid Lucas-Kanade optical flow to achieve robust feature extraction and efficient tracking, which ensures estimation accuracy and reduces computational complexity, thus providing good real-time performance.

[0065] 3. This invention integrates Kalman filtering and Gaussian filtering for trajectory smoothing, balancing stability with adaptability to sudden disturbances, and effectively suppressing video jitter caused by water waves, wind, and ship vibration.

[0066] 4. This invention does not rely on large-scale supervised data, and combines the robustness of deep learning with the controllability and low overhead of traditional methods. It can efficiently output stable video in complex marine environments and has strong engineering practicality and promotion value. Attached Figure Description

[0067] Figure 1 This is a flowchart of a real-time video stabilization method for unmanned surface vessels based on XFeat in this embodiment;

[0068] Figure 2 This is a flowchart of dynamic water surface segmentation in this embodiment;

[0069] Figure 3 This is a flowchart of XFeat feature point extraction in this embodiment;

[0070] Figure 4a This is a schematic diagram of a feature point before the feature points are uniform in this embodiment;

[0071] Figure 4b This is a schematic diagram of a feature point after the feature points are uniformly distributed in this embodiment;

[0072] Figure 5 This is a schematic diagram of one structure of the LK pyramid in this embodiment;

[0073] Figure 6a This is a schematic diagram of an optical flow tracking result without forward verification in this embodiment;

[0074] Figure 6b This is a schematic diagram of an optical flow tracking result after forward verification in this embodiment;

[0075] Figure 7a This is a comparison chart showing the smoothing of the camera motion trajectory x-axis using the original, Kalman, Gaussian, and combined Kalman and Gaussian methods in this embodiment.

[0076] Figure 7b This is a comparison chart showing the smoothing of the camera motion trajectory along the y-axis using the original, Kalman, Gaussian, and combined Kalman and Gaussian methods in this embodiment.

[0077] Figure 7c This is a comparison chart showing the smoothing of camera motion trajectory angles using the original, Kalman, Gaussian, and combined Kalman and Gaussian methods in this embodiment.

[0078] Figure 8 This is a comparison image before and after the black border was cropped in this embodiment. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Example: A real-time video stabilization method for unmanned surface vessels based on XFeat, such as... Figure 1 As shown, it includes the following steps:

[0081] Step S1, Dynamic water surface segmentation: Segment the dynamic water surface region from the image of the water surface scene video taken by the unmanned vessel and generate a water surface mask.

[0082] In aerial videos of water surfaces captured by unmanned surface vessels (USVs), dynamic water surface areas such as waves and ripples generally exhibit high-frequency, non-rigid motion characteristics. These areas can significantly interfere with camera motion estimation based on feature point matching. Therefore, accurately identifying and masking dynamic water surface areas is crucial for improving video stability. Regarding the selection of segmentation methods, although deep learning-based methods are more robust under extreme lighting changes or complex water surface reflections, these methods typically rely on large labeled datasets for training, and the computational complexity of the inference process is high, making it difficult to meet the real-time requirements of USV platforms. To address this, this embodiment proposes an efficient water surface segmentation method to achieve accurate detection and segmentation of dynamic water surface areas.

[0083] As shown in equation (1), each frame of the unstable video... It is a reliable area and unreliable areas It is composed of jitter factors such as translation, rotation, and scaling. It is generated by action. Therefore, if the jitter factor can be accurately estimated... This allows for the correction of temporal frames through motion compensation, resulting in more stable videos with better visual effects.

[0084] (1)

[0085] Since unreliable water surface wave regions contain a large number of unreliable features, directly using them for motion estimation will introduce estimation errors and ultimately affect the video stabilization effect. Therefore, we first identify and segment unreliable water surface regions to exclude them from subsequent motion trajectory estimation. Specifically, we perform feature detection on the reliable regions of the current frame and match them with the features of the reliable regions of the previous frame to obtain the reliable regions of the current frame. Inner reliable region of the previous frame Matching feature points As shown in equation (2). Similarly, the matching feature points from the previous frame to the current frame can be obtained. .

[0086] (2)

[0087] In the formula, This is a region feature extraction function; This is for feature point matching operations.

[0088] Ideally, the matching feature points between two frames should satisfy the following relationship:

[0089] (3)

[0090] The core of this method lies in accurately estimating the turbulence factor by eliminating dynamic water surface regions. In unmanned surface aerial photography, water surfaces are typically located in the lower middle of each frame. Therefore, water surface detection focuses primarily on these areas, aiming to identify and remove dynamic water surface pixels while retaining water bodies with stable reflective characteristics, such as calm water surfaces containing reflections, to ensure that subsequent feature extraction and motion estimation are performed only in reliable regions. Dynamic water surface detection and segmentation is a multi-step optimization process, the overall workflow of which is as follows: Figure 2 As shown, it includes the following steps:

[0091] Step S11: Convert the input RGB image to the HSV color space and perform preliminary segmentation using the specific color distribution of the water surface area in the HSV color space.

[0092] Because water surfaces typically exhibit specific color distribution characteristics in the HSV color space, such as low saturation (S), medium to high brightness (V), and specific hue (H), preliminary segmentation is performed using prior knowledge of the water surface spectrum.

[0093] Step S12: Based on the probability of water surface appearance, the image is divided into a grid, and combined with the characteristics of unmanned surface vessel photography, the grid is further divided. Figure 2 The red grid marks indicate high-probability water surface areas. .

[0094] Step S13: Construct a water likelihood probability model based on HSV channel values, calculate the probability that each pixel belongs to the water surface, and generate a color mask; specifically:

[0095] Step S131: Calculate the mean HSV value of each grid using equation (4), which will be used for subsequent water area identification and segmentation.

[0096] (4)

[0097] In the formula, m and n are the row and column indices of the grid, respectively. The location of the grid cell; , and They represent the locations respectively The average hue, saturation, and highlight values ​​of the grid; For pixels within the grid; and These are the grid height and width, respectively.

[0098] Step S132: Based on the physical likelihood of water surface pixels within the same water area, construct a water likelihood probability model:

[0099] (5)

[0100] In the formula, It is the sigmoid function; , , and These are all fitting parameters, used to characterize the relationship between each channel and the probability of water surface; Represents all pixels The probability of water surface.

[0101] Step S133: Obtain the upper and lower bound thresholds of HSV for water surface pixels by combining the water likelihood probability with thresholding, and then define the water surface pixel set C:

[0102] (6)

[0103] In the formula, , and Representing pixels Saturation, brightness, and hue components in the HSV color space; and These are the upper and lower bound thresholds for saturation, respectively. and These are the upper and lower bound thresholds for brightness, respectively. and These are the upper and lower thresholds for the hue, respectively.

[0104] Step S134: Generate a color mask :

[0105] (7)

[0106] Step S14: Calculate the gradient magnitude using the Sobel operator on the grayscale image and set a low gradient threshold to generate a low gradient mask.

[0107] Although color masks can identify candidate water surface regions that match color characteristics, they may still mistakenly include non-water surface regions with similar colors but rich textures, such as vegetation and shadows. Considering that the gradient of a real dynamic water surface is low in local regions, the Sobel operator is further used on the grayscale image to calculate the gradient magnitude, and a low gradient threshold is set. Generate a low gradient mask :

[0108] (8)

[0109] In the formula, Indicates at pixel point Sobel gradient vector at point; This represents the L2 norm, i.e., the gradient magnitude.

[0110] Step S15, apply the color mask and gradient mask Fusion is performed to obtain a combined mask. :

[0111] (9)

[0112] However, the fused mask may exhibit issues such as isolated points, holes, or irregular edges. To further improve the structural coherence and integrity of the mask, morphological closing operations are employed. Fill in the holes and gaps inside the mask and connect adjacent areas. Finally, perform an opening operation. After removing noise points, the optimized water surface mask is obtained. :

[0113] (10)

[0114] Step 2, Feature point extraction: The XFeat method based on deep learning is used to extract feature points in the non-water surface region outside the water surface mask in the image.

[0115] Under the constraint of a dynamic water surface mask, the shipborne vision system obtains a stable non-water surface region, providing a reliable foundation for the accurate extraction and tracking of subsequent feature points. The XFeat method is a lightweight and efficient deep feature extraction method. Its network structure significantly reduces computational overhead while maintaining excellent matching performance and can run efficiently on CPU or low-power GPU platforms, making it particularly suitable for video stabilization tasks with high timeliness requirements. Compared with traditional methods, XFeat exhibits stronger robustness in handling complex scenes such as lighting changes and water surface reflections. Compared with other deep models, XFeat achieves a trade-off between feature detection quality and real-time performance with a lighter model and higher operating efficiency.

[0116] The XFeat feature point extraction network structure is as follows: Figure 3 As shown, unlike traditional methods, XFeat represents the L×W×1 input image as a 2D grid consisting of 8×8 pixels in each cell, reducing the resolution to 1 / 8 of the original image. Each cell is then reshaped into an L / 8×W / 8×64 feature tensor. After four convolutional layers, a keypoint embedding L / 8×W / 8×(64+1) feature tensor is obtained. Finally, the L×W×1 image is normalized and reconstructed to obtain the feature point detection results.

[0117] During feature point extraction, the image with the water surface mask applied is input into the XFeat neural network. The XFeat neural network is an end-to-end structure that can jointly optimize feature point detection and descriptor extraction. That is, the feature point extraction operation is only performed in a reliable area outside the water surface mask. This ensures that the extracted feature points come from stable backgrounds such as the sky, shore, and static buildings, avoiding interference from dynamic water surfaces. These feature points will be used as input for step 3 (homogenization processing) to improve their spatial distribution.

[0118] Step S3, homogenization: The extracted feature points are spatially redistributed using a square continuous clustering algorithm to simultaneously consider the quality of the feature points and the balance of their spatial distribution.

[0119] Although Xfeat can extract high-quality feature points, these feature points are often concentrated in areas with rich texture or high contrast, leading to sparse or even missing features in some non-water surface areas. This uneven spatial distribution negatively impacts optical flow estimation and geometric transformation fitting, easily causing local motion estimation failures, and consequently resulting in image distortion and decreased stability. To improve the balance of feature point spatial distribution, a feature point homogenization strategy is introduced. Considering robustness, flexibility, and computational efficiency, this method employs the Square Continuous Clustering (SSC) algorithm instead of the conventional grid partitioning method. SSC can adaptively adjust its spatial distribution while maintaining the response intensity of feature points, effectively avoiding the problem of feature points clustering in local areas or missing edge information. Specifically:

[0120] Step S31: Sort all candidate feature points according to their response level;

[0121] Step S32: Use a dynamically resizable square window to iteratively select the feature points with the strongest response within the window, and use binary search to dynamically determine the appropriate window size;

[0122] Step S33 is repeated until the number of selected feature points reaches the preset target range.

[0123] The square continuous clustering algorithm is used to spatially redistribute the original feature points extracted in step 2, achieving a trade-off between feature point quality and spatial distribution. This avoids feature points clustering in local high-texture regions, thus providing a more spatially uniform and representative set of feature points for subsequent motion estimation. A comparison of feature point distribution before and after the square continuous clustering algorithm is shown below. Figure 4a , Figure 4b As shown.

[0124] Step S4, Motion Estimation: Use the homogenized feature points to estimate the camera's motion between consecutive video frames and construct the global motion trajectory.

[0125] The LK pyramid optical flow method is an improved algorithm that combines the Lucas-Kanade optical flow method (LK method) with the pyramid layering strategy. It achieves a good balance between computational efficiency and matching accuracy, and shows higher applicability and stability in real-time systems.

[0126] like Figure 5 As shown, the left image illustrates the feature point tracking path within the LK pyramid structure. From the low-resolution top layer to the bottom layer at the original image resolution, the displacement estimation of the feature points is progressively refined. The green lines in the image indicate the tracking trajectory of the feature points from the top layer (layer 1) to the bottom layer (layer 4). Figure 5 The right figure further illustrates the process of updating the offset between layers. The optical flow offset h1 initially calculated at the top layer (layer 1) is passed to the next layer as an initial conjecture. Then, in layer 2, it is refined into a more accurate offset h2 by combining local image information, and so on down to the bottom layer (layer 4). Finally, the feature point tracking result is obtained on the original resolution image.

[0127] To further improve the robustness and accuracy of camera motion estimation, this method introduces a consistency verification mechanism based on traditional unidirectional optical flow tracking. It uses bidirectional optical flow estimation to determine the consistency of feature point matching, thereby effectively eliminating mismatches caused by factors such as occlusion, reflection, or local perturbations. Specifically, for example... Figure 6a and Figure 6b As shown, it includes the following steps:

[0128] Step S41: The LK pyramid optical flow method is used to perform forward tracking and reverse optical flow verification of feature points to determine the consistency of feature point matching. That is, based on the feature points extracted in the previous frame, the LK pyramid optical flow method is used for forward tracking to predict their position in the current frame. Then, the reverse optical flow is estimated based on the predicted position to map it back to the previous frame image and calculate the pixel error with the original position to determine the effectiveness of the matching.

[0129] Step S42: For feature points that pass the consistency verification, the RANSAC algorithm is used to estimate the fundamental matrix, thereby filtering out geometrically inconsistent abnormal matches and retaining feature point pairs with high confidence.

[0130] Step S43: Fit an affine transformation matrix using high-confidence feature points, and construct the camera's global motion trajectory by accumulating the results frame by frame; specifically:

[0131] Step S431: Utilize feature points that match the previous frame and the current frame. and The position information is used to fit the feature points with an affine matrix using Equation (11) to estimate the translation and rotation transformations between consecutive frames;

[0132] (11)

[0133] In the formula, H is the affine transformation matrix, defined as follows:

[0134] (12)

[0135] In the formula, , This is the translation amount; The rotation angle;

[0136] Step S432, using the inter-frame affine transformation matrix Multiply by the product to construct the global motion trajectory of the camera:

[0137] (13)

[0138] In the formula, Let be the affine transformation matrix from frame n to frame n+1; This is the cumulative motion transformation matrix from frame 1 to frame n.

[0139] Step S5, Real-time trajectory smoothing: Filter out high-frequency jitter in the estimated camera motion trajectory and retain the effective motion actively performed by the camera.

[0140] After acquiring the camera's motion trajectory, directly using the original trajectory for image registration and inter-frame transformation often introduces high-frequency disturbances such as platform vibration and water surface ripples into the image stabilization result. This leads to jitter, jumps, or even local stabilization failures in the output image, resulting in unpleasant video stabilization outcomes. To effectively filter out such unwanted motion while retaining the camera's effective motion information during navigation, such as translation and viewpoint switching, a real-time trajectory smoothing method is needed. Specifically:

[0141] Step S51: A covariance model of state transition and observation error is established using Kalman filtering, and the global motion trajectory of the camera is optimized through the prediction and update stages.

[0142] Kalman filtering is an algorithm for performing optimal state estimation, capable of dynamically modeling and predicting system states. This algorithm predicts and updates trajectories by establishing a covariance model of state transitions and observation errors. Its optimization process consists of two stages: prediction and update. In the prediction step, Kalman filtering predicts the state of the current frame based on the optimal estimate from the previous frame, specifically defined as follows:

[0143] (14)

[0144] In the formula, It is the state prediction quantity corresponding to the nth frame; It is the optimal estimate corresponding to the state of the (n-1)th frame; It is the state covariance matrix corresponding to the nth frame; It is the optimal covariance estimate corresponding to the (n-1)th frame; It is the system's transfer matrix; It is the covariance of the predicted noise; T is the matrix transpose operation.

[0145] During the update phase, the Kalman filter corrects the predicted state based on the observation results. The update process is as follows:

[0146] (15)

[0147] In the formula, Kalman gain; Here is the state covariance matrix; Mean square error; To observe the noise covariance matrix; It is the observation matrix obtained from the trajectory; It is an identity matrix.

[0148] Step S52: After Kalman filtering, a one-dimensional time-domain Gaussian filter is applied to perform weighted convolution on each dimension of the trajectory to enhance local smoothness.

[0149] To further enhance the continuity and smoothness of the local trajectory, especially in response to short-term severe disturbances, a one-dimensional time-domain Gaussian filter is introduced for secondary smoothing based on the Kalman filter. As shown in Equation (16), for each dimension of the trajectory sequence, such as the translation amount... , and rotation angle We use a one-dimensional Gaussian kernel for weighted convolution.

[0150] (16)

[0151] In the formula, Indicates the index of the current frame; This is the smoothed trajectory value for the nth frame; This represents the original trajectory value of the ni-th frame; The normalized Gaussian weights corresponding to the i-th frame in history satisfy the following conditions: The definition is as follows:

[0152] (17)

[0153] In the formula, The standard deviation of the Gaussian kernel controls the decay rate of the Gaussian weights; The maximum number of frames to backtrack, with a window length of [value missing]. ; As a normalization factor, it ensures that the sum of all weights is 1, maintaining consistent energy before and after smoothing.

[0154] Considering that trajectory signals contain both systematic errors and may be affected by local disturbances, a single filtering strategy cannot simultaneously achieve both global stability and local smoothness. Therefore, a joint Kalman and Gaussian filtering framework is designed. Figures 7a-7c As shown, this joint filter effectively preserves the true motion trend while significantly suppressing the interference of platform vibration and water surface ripples on the camera trajectory. Compared with traditional single filtering methods, this joint filter achieves a better balance between global stability and local smoothness, thereby improving the stability of inter-frame geometric transformations.

[0155] Step 6, Stabilized Frame Generation: After smoothing the camera motion trajectory, the smoothed trajectory is applied to the original video sequence to generate a stable video frame sequence. Specifically:

[0156] Step S61: Calculate the smoothed affine transformation matrix corresponding to each frame based on the trajectory smoothing result, and then apply the smoothed affine transformation matrix to the original video frame image to perform image remapping and registration.

[0157] Through geometric transformation, the images are registered and aligned frame by frame with the original video frame images to eliminate high-frequency motion components caused by platform jitter and water surface disturbance.

[0158] Step S62: Crop the image edges to remove the black borders caused by the transformation, and generate a stable video without black borders.

[0159] Black borders often appear during image transformation. If left unaddressed, they can significantly impact the final visual effect. To address this issue, a cropping strategy is employed to remove the black borders, thereby ensuring a stable overall video appearance.

[0160] To verify the effectiveness of this method, five videos were selected from the dataset for experimentation, and the final results are as follows. Figure 8 As shown. According to Figure 8 It can be seen that the stabilized result without cropping exhibits obvious black borders. Furthermore, the video stabilization effect was evaluated using four metrics: cropping, distortion, stability, and pixel fidelity. A value closer to 1 for each metric indicates better performance, as detailed in Table 1. Experimental results show that this method performs well in terms of cropping, distortion, stability, and pixel fidelity, and no obvious black borders were observed in the stabilized video results.

[0161] Table 1 Evaluation Indicators and Results

[0162]

[0163] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A real-time video stabilization method for unmanned surface vessels based on XFeat, characterized in that, Includes the following steps: Step S1: Segment the dynamic water surface region from the images of the water surface scene video taken by the unmanned vessel and generate a water surface mask; Step S2: The XFeat method based on deep learning is used to extract feature points from the non-water surface region outside the water surface mask in the image. Step S3: The extracted feature points are spatially redistributed using a square continuous clustering algorithm to simultaneously ensure both the quality of the feature points and the balance of their spatial distribution. Step S4: Use the homogenized feature points to estimate the camera's motion between consecutive video frames and construct the global motion trajectory. Step S5: Filter out high-frequency jitter in the estimated camera motion trajectory and retain the effective motion actively performed by the camera; Step S6: Apply the smoothed motion trajectory to the original video sequence to generate a stable video frame sequence; Step S1 includes the following steps: Step S11: Convert the input RGB image to the HSV color space and perform preliminary segmentation using the specific color distribution of the water surface area in the HSV color space. Step S12: Divide the image into a grid and mark high-probability water surface areas based on the characteristics of unmanned surface vessel photography; Step S13: Construct a water likelihood probability model based on HSV channel values, calculate the probability that each pixel belongs to the water surface, and generate a color mask. Step S14: Calculate the gradient magnitude using the Sobel operator on the grayscale image, and set a low gradient threshold to generate a low gradient mask. Step S15: The color mask and gradient mask are fused to obtain a combined mask, and morphological closing and opening operations are used for optimization to obtain a water surface mask.

2. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 1, characterized in that, In step S13, the generation of the color mask includes the following steps: Step S131: Calculate the HSV mean for each grid using the following formula: In the formula, m and n are the row and column indices of the grid, respectively. The location of the grid cell; , and They represent the locations respectively The average hue, saturation, and highlight values ​​of the grid; For pixels within the grid; and These are the grid height and width, respectively. Step S132: Based on the physical likelihood of water surface pixels within the same water area, construct a water likelihood probability model: In the formula, It is the sigmoid function; , , and All are fitted parameters; Represents all pixels The probability of water surface; Step S133: Obtain the upper and lower bound thresholds of HSV for water surface pixels by combining the water likelihood probability with thresholding, and then define the water surface pixel set C: In the formula, , and Representing pixels Saturation, brightness, and hue components in the HSV color space; and These are the upper and lower bound thresholds for saturation, respectively. and These are the upper and lower bound thresholds for brightness, respectively; and These are the upper and lower thresholds for the hue, respectively; Step S134: Generate a color mask : 。 3. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 2, characterized in that, In step S14, the low gradient mask Generate using the following formula: In the formula, Indicates at pixel point Sobel gradient vector at point; This represents the L2 norm, which is the gradient magnitude. This is a low gradient threshold.

4. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 3, characterized in that, In step S15, the combined mask is... Generate using the following formula: Water surface mask Generate using the following formula: In the formula, For morphological closing operations; This is an opening operation.

5. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Sort all candidate feature points according to their response level; Step S32: Use a dynamically resizable square window to iteratively select the feature points with the strongest response within the window, and use binary search to dynamically determine the appropriate window size; Step S33 is repeated until the number of selected feature points reaches the preset target range.

6. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: The LK pyramid optical flow method is used to perform forward tracking and reverse optical flow verification on the feature points in order to determine the consistency of the feature point matching. Step S42: Use the RANSAC algorithm to estimate the fundamental matrix of the feature points that have passed the consistency verification, and retain the feature point pairs with high confidence. Step S43: Fit the affine transformation matrix using high-confidence feature points, and construct the camera's global motion trajectory by accumulating the results frame by frame.

7. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 6, characterized in that, Step S43 includes the following steps: Step S431: Utilize feature points that match the previous frame and the current frame. and The position information is used to perform affine matrix fitting on the feature points using the following formula to estimate the translation and rotation transformations between consecutive frames; In the formula, H is the affine transformation matrix, defined as follows: In the formula, , This is the translation amount; The rotation angle; Step S432, using the inter-frame affine transformation matrix Multiply by the product to construct the global motion trajectory of the camera: In the formula, Let be the affine transformation matrix from frame n to frame n+1; This is the cumulative motion transformation matrix from frame 1 to frame n.

8. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: A covariance model of state transition and observation error is established using Kalman filtering, and the global motion trajectory of the camera is optimized through the prediction and update stages. Step S52: After Kalman filtering, a one-dimensional time-domain Gaussian filter is applied to perform weighted convolution on each dimension of the trajectory to enhance local smoothness.

9. The method for real-time video stabilization of unmanned surface vessels based on XFeat according to claim 1, characterized in that, Step S6 includes the following steps: Step S61: Calculate the smoothed affine transformation matrix corresponding to each frame based on the trajectory smoothing result, and apply the smoothed affine transformation matrix to the original video frame image to perform image remapping and registration. Step S62: Crop the image edges to remove the black borders caused by the transformation, and generate a stable video without black borders.

Images