A spatio-temporal context fusion unmanned surface vehicle electronic image stabilization and target detection method

Abstract

This invention discloses a spatiotemporal context-based electronic image stabilization and target detection method for unmanned surface vessels (USVs). Addressing the problem of slow detection speed and low accuracy caused by shaking and swaying during USV operation, this invention proposes an electronic image stabilization and target detection method incorporating spatiotemporal context. The method includes: designing and training a deep learning-based target detection neural network model; using deep learning methods to identify the location of detection boxes and the category of detected targets; employing a context fusion strategy combining the location of detection boxes and the relationship between adjacent image frames; using a feature point detection and matching algorithm to perform motion estimation on the image; and using an adaptive Kalman filter algorithm to perform motion compensation on the image. This invention achieves real-time electronic image stabilization and surface target detection for USV vision, improving the system's adaptability and robustness.

Description

Technical Field

[0001] This invention relates to the field of surface target detection, specifically to a spatiotemporal context fusion-based electronic image stabilization and target detection method for unmanned surface vessels. Background Technology

[0002] The 21st century is the century of the ocean. With the depletion of land resources, people need to seek development resources and space from the ocean. In recent years, with the development of artificial intelligence technology, the global marine industry will also undergo transformation. The emergence of unmanned maritime systems has given wings to the development of the marine industry, and the era of intelligent and unmanned maritime operations has arrived.

[0003] Unmanned surface vessels (USVs) are an emerging type of ship that fully utilizes artificial intelligence technology in their operation, enabling unmanned surface operations. Target recognition technology based on USV visual sensing has enormous development potential. Its applications are rich and wide-ranging, playing a significant role in areas such as intelligent video surveillance, robot navigation, autonomous driving, inland waterway and maritime activity monitoring, and port vessel scheduling.

[0004] As deep learning is increasingly applied in machine vision, it has achieved mature engineering applications in autonomous driving and facial recognition. However, many problems remain to be solved in its application to water surface target recognition. The vibration and swaying of unmanned surface vessels (USVs) while navigating water cause vibrations in the images captured by cameras, significantly impacting subsequent image processing. Electronic image stabilization can address this issue. However, due to the high likelihood of mismatches in large, repetitive areas of water and sky, feature point matching methods are difficult to apply to electronic image stabilization for USV vision.

[0005] The YOLOv5 neural network has a complex structure and many parameters, making it unsuitable for embedded systems on unmanned surface vessels. Furthermore, this algorithm cannot maintain continuous attention on detected targets and can only perform calculations for each frame of the image individually. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a spatiotemporal context fusion-based electronic image stabilization and target detection method for unmanned surface vessels. The purpose is to solve the problem of low target detection accuracy caused by jitter in practical applications, and to accelerate the speed and accuracy of target detection while maintaining continuous focus on the same target.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A spatiotemporal context fusion method for electronic image stabilization and target detection of unmanned surface vessels, comprising the following steps:

[0009] Step S1: Read real-time image frames from the camera;

[0010] Step S2 uses an improved YOLOv5 neural network to detect all targets;

[0011] Step S3 uses a spatiotemporal context fusion strategy to record the category of the water surface target and the position of the target box in the current image frame, continues to read the next frame image from the camera, and narrows the range of the two frames to within the target box;

[0012] Step S4 uses the Scale Invariant Feature Transform (SIFT) algorithm and a nearest neighbor to second nearest neighbor feature point matching strategy to perform feature point detection and matching on the two frames of images after the range was reduced in step S3.

[0013] Step S5 uses an image transformation matrix filter based on adaptive Kalman filtering to perform motion compensation on the image read in step S3, and then jumps to S2.

[0014] The improved YOLOv5 neural network in step S2 above includes the following improvements:

[0015] (1) The K-Means++ algorithm is used to replace the anchor box generation algorithm used by the original YOLOv5 neural network. The K-Means++ algorithm is an algorithm for selecting initial values ​​for the k-means algorithm. Although the initial selection in this algorithm takes extra time, the k-means part itself will converge quickly after the initial selection is completed. Therefore, this algorithm actually reduces the computation time and can effectively alleviate the problem that the convergence of the original algorithm is heavily dependent on the initialization status of the cluster center, thereby improving the detection accuracy and effect to a certain extent.

[0016] The specific steps of the K-Means++ algorithm are as follows:

[0017] ① Randomly select a center u1 among the data points, where u1 refers to the first center point;

[0018] ② For each data point x that has not yet been selected, calculate Where i is the label of the center point; j is the number of center points; and distance is the distance; that is, calculating the distance between each data point x and the center point that is closest to that data point.

[0019] ③ Use a weighted probability distribution to randomly select a new data point as the new center, where the probability of the selected data point x is proportional to the probability of the selected data point x. Proportional;

[0020] ④ Repeat steps ② and ③ until k center points are selected, where k is the number of categories;

[0021] ⑤ Using the selected k center points as initial center points, perform the k-means algorithm;

[0022] (2) The C3 structure backbone of the YOLOv5 neural network is replaced with the EfficientFormer network, which reduces the number of parameters of the neural network and increases the detection speed, enabling its application on real unmanned surface vessels. At the same time, this method can overcome the defect that the YOLOv5 neural network backbone is a fully convolutional network.

[0023] The EfficientFormer network consists of patch embeddings and a meta-transformer block stack, where patch embeddings are represented by PatchEmbed and meta-transformer block stacks are represented by MB, as shown below:

[0024]

[0025] In the formula, X0 is the input image, B is the batch size, [C,D] is the size of the feature map, y is the expected output, m is the total number of blocks, and MB consists of an unspecified token mixer and MLP blocks, where the token mixer is represented by TokenMixer and the MLP block is represented by MLP, and the composition relationship is as follows:

[0026] X l+1 =MB l (X l =MLP(TokenMixer(X) l (2)

[0027] Among them, X l It is the feature map of the l-th MB, and the stage is further defined as a stack of multiple meta-blocks with features of the same spatial size.

[0028] The spatiotemporal context fusion strategy in step S3 mentioned above specifically refers to:

[0029] A spatiotemporal context fusion strategy is incorporated into object detection and electronic image stabilization. The temporal context is reflected in the feature point detection and matching process during electronic image stabilization, which is accomplished by comparing the motion relationship between the current frame and the previous frame. The spatial context is reflected in the recording of the detection box position during object detection. Feature point detection and matching are performed on images within the same target box, narrowing the image range and avoiding mismatches that easily occur with large areas of overlapping water and sky. Object detection obtains the category and location information of the measured object, which is then applied to electronic image stabilization. Electronic image stabilization, in turn, serves object detection, eliminating jitter caused by the unmanned surface vessel's movement. The stabilized image is then used for object detection, improving its accuracy. Furthermore, because the detection box position is recorded, if the detection box position overlaps with the recorded position by more than 70% when detecting in the next frame, it is determined that the two detection boxes are detecting the same target, thus achieving continuous focus on the same target.

[0030] The specific process of the SIFT feature point detection algorithm in step S4 above is as follows:

[0031] ① A Gaussian difference pyramid with different scales and parameters is constructed using Gaussian filtering to search for image locations at all scales. The extreme points are identified by simulating the Gaussian differential function using the Gaussian difference pyramid. The Gaussian scale space formula for image I(a,b) is as follows:

[0032]

[0033] Where L is the image representation at different resolutions; G is the Gaussian function; I is the image size; (a,b) is the pixel position of the image; σ represents the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the scale. Large scale corresponds to the overall features of the image, and small scale corresponds to the detailed features of the image. Represents convolution operation;

[0034] ② After obtaining the feature points at different scales, a Gaussian difference scale space is generated from the Gaussian scale space. The Gaussian difference scale space can be represented by the following formula:

[0035]

[0036] Where o is the proportionality coefficient;

[0037] ③ Threshold the image to remove noise, retaining regions that satisfy the following formula:

[0038]

[0039] Where T is set to 0.04, n is the number of images to be extracted, and abs(val) is the pixel value of the image;

[0040] ④ Find the extreme point a0 in the Gaussian difference pyramid;

[0041] ⑤ At the detected extreme points, the position of the feature points is corrected by performing a ternary second-order Taylor expansion using the following formula, so that the feature points can appear at sub-pixel positions:

[0042]

[0043] Where f(a) is the ternary second-order Taylor expansion at the extreme point; for transpose; f T It is the transpose of f(a); The sign of the partial derivative;

[0044] ⑥ Points that satisfy the following formula are defined as points with low contrast and are removed;

[0045]

[0046] ⑦ Since the extracted target feature points are corner points rather than edges, and the aforementioned measures can only guarantee the capture of points with drastic grayscale value changes, while edge points also meet this characteristic, we will remove points that satisfy the matrix determinant Det(M) < 0 and those whose matrix determinant and trace do not satisfy this condition. The feature points are used to remove edge points, where M is the Hessian matrix, Tr(M) is the trace of M, Det(M) is the determinant of M, and γ0 is a meaningful empirical value, which is set to 10 here.

[0047] In step S4 above, the feature point matching adopts the nearest neighbor to second nearest neighbor feature point matching strategy, as shown in the following formula:

[0048]

[0049] The ratio is the ratio obtained by dividing the nearest distance by the second nearest distance.

[0050] The image transformation matrix filtering based on adaptive Kalman filtering in step S5 above is specifically as follows:

[0051] ① Observation noise and prediction noise are denoted as and The state equations and observation equations are as follows:

[0052]

[0053]

[0054] Where t represents time; S t Φ is the state vector. t,t-1 Z is the state transition matrix; t H is the observation vector; t W is the observation matrix. t-1 For system noise; V t To observe noise; This represents the mean of the observed noise. Let the mean of the predicted noise be denoted as .

[0055] ② Calculate the predicted state And noise covariance matrix P t,t-1 ;

[0056]

[0057]

[0058] Where Φ t,t-1 TFor Φ t,t-1 Transpose of; Here is the system noise covariance matrix;

[0059] ③ Update filter gain K t ;

[0060]

[0061] Where H t T For H t Transpose of; The covariance matrix of the observation noise;

[0062] ④ Calculate the residual ε t ;

[0063]

[0064] ⑤ Update the state vector And noise covariance matrix P t ;

[0065]

[0066] P t =(IK t H t )P t,t-1 (16)

[0067] ⑥ Calculate the weighting coefficient d t In the formula, c is the relaxation factor;

[0068]

[0069] ⑦ Update Considering that measurement accuracy is positively correlated with the logarithm of matching points, update

[0070]

[0071]

[0072]

[0073]

[0074] In the formula, g is the number of matching point pairs e t The larger the value of g, the greater the impact of the number of points on measurement accuracy; ε t T For ε t The transpose of .

[0075] Compared with the prior art, the beneficial effects of the present invention are:

[0076] This invention employs a spatiotemporal context fusion strategy to narrow the image range for feature point detection and matching to the vicinity of the detected object, thereby avoiding mismatches that are prone to occur with large areas of repeated water and sky. Furthermore, by utilizing the relationship between consecutive frames, motion compensation is applied to the image, addressing the problem that vibrations and swaying of unmanned surface vessels (USVs) on water can cause image vibrations in camera-captured images, significantly impacting subsequent image processing. Simultaneously, because this strategy records the category and location of the detected object, the algorithm can maintain continuous focus on the same target, facilitating the collection and processing of subsequent information.

[0077] This invention significantly reduces the number of neural network parameters and accelerates the target detection speed of the neural network by improving the YOLOv5 neural network, making the network more suitable for practical applications of unmanned surface vessels. Attached Figure Description

[0078] Figure 1 This is a flowchart of the algorithm of the present invention;

[0079] Figure 2 This is a diagram of the Efficientformer backbone network structure of the replaced YOLOv5;

[0080] Figure 3 It is MB in Efficientformer 4D Detailed structural diagram;

[0081] Figure 4 It is MB in Efficientformer 3D The specific structural diagram. Detailed Implementation

[0082] The following description, in conjunction with the accompanying drawings, further illustrates the spatiotemporal context fusion-based electronic image stabilization and target detection method for unmanned surface vessels provided by this invention.

[0083] To address the shortcomings of existing technologies, the present invention aims to provide a spatiotemporal context fusion-based electronic image stabilization and target detection method for unmanned surface vessels, which solves the problem of low target detection accuracy caused by jitter in practical applications, while maintaining continuous focus on the same target.

[0084] The objective of this invention is achieved through the following technical solution:

[0085] like Figure 1As shown in this paper, an electronic image stabilization and target detection method for unmanned surface vessels (USVs) based on spatiotemporal context fusion is proposed. After the method starts running, real-time image data is first read through the USV's onboard camera. Then, an improved YOLOv5 neural network is used for target detection. Next, a spatiotemporal context fusion strategy is used to record the category of the surface target and the position of the target bounding box. Then, the next frame image from the camera is acquired, and electronic image stabilization is performed on this image. The specific electronic image stabilization process involves cropping the current frame image and the recorded image, with the cropped portion being the image within the recorded target bounding box. For the two adjacent cropped frames, a SIFT-based feature point detection algorithm is applied, and a nearest / second nearest neighbor feature point matching strategy is used for feature point matching. Finally, an image transformation matrix filter based on adaptive Kalman filtering is used to perform motion compensation on the new frame image, thereby achieving electronic image stabilization. The stabilized image is then used for target detection. This process is repeated to achieve fast, stable, and accurate target detection.

[0086] To improve the speed and accuracy of object detection, this paper improves the YOLOv5 neural network by replacing the anchor box generation algorithm used in the original YOLOv5 neural network with the K-Means++ algorithm. The K-Means++ algorithm is an algorithm for selecting initial values ​​for the k-means algorithm. Although the initial selection in this algorithm takes additional time, the k-means part itself converges quickly after the initial selection is completed. Therefore, this algorithm actually reduces the computation time and can effectively alleviate the problem that the convergence of the original algorithm is heavily dependent on the initialization of the cluster centers, thereby improving the detection accuracy and effect to a certain extent.

[0087] The specific steps of the K-Means++ algorithm are as follows:

[0088] ② Randomly select a center u1 among the data points, where u1 refers to the first center point;

[0089] ② For each data point x that has not yet been selected, calculate Where i is the label of the center point; j is the number of center points; and distance is the distance; that is, calculating the distance between each data point x and the center point that is closest to that data point.

[0090] ③ Use a weighted probability distribution to randomly select a new data point as the new center, where the probability of the selected data point x is proportional to the probability of the selected data point x. Proportional;

[0091] ④ Repeat steps ② and ③ until k center points are selected, where k is the number of categories;

[0092] ⑤ Using the selected k center points as initial center points, perform the k-means algorithm;

[0093] Another modification to the YOLOv5 neural network in this paper is to replace the C3 structure backbone network of the YOLOv5 neural network with the EfficientFormer network. This reduces the number of parameters in the neural network, increases the detection speed, and enables its application on real-world unmanned surface vessels. At the same time, this method overcomes the limitation of the YOLOv5 neural network's backbone being a fully convolutional network.

[0094] like Figure 2 As shown, the EfficientFormer network consists of patch embedding and metatransformer block stack. Patch embedding is represented as Embedding in the figure, and metatransformer block stack is represented as MB in the figure. CONV stem in the figure is a convolutional layer with a size of 3×3 and a stride of 2. Figure 2 In this model, 4D and 3D are labeled on the MB to distinguish tensors of different dimensions. The network has a dimension-consistent design, dividing the network into a 4D partition with operators implemented in a convolutional network style and a 3D partition. Linear projection and attention mechanisms are performed on the 3D tensor, enjoying the global modeling capabilities of the multi-head attention mechanism (MHSA) without sacrificing efficiency.

[0095] Figure 3 and Figure 4 MB was displayed respectively 4D and MB 3D The specific structure, Figure 3 In the diagram, pooling refers to the pooling layer, CONV:1×1 is a convolutional layer with a size of 1×1, BN refers to batch normalization, and GeLU refers to the GeLU activation function. Figure 4 In this context, LN refers to layer normalization, Linear refers to linear projection learning, and Q, K, and V represent the query, key, and value learned through linear projection.

[0096] The specific composition of the EfficientFormer network is shown below:

[0097]

[0098] In the formula, X0 is the input image, B is the batch size, [C,D] is the size of the feature map, y is the expected output, m is the total number of blocks, and MB consists of an unspecified token mixer and MLP blocks, where the token mixer is represented by TokenMixer and the MLP block is represented by MLP, and the composition relationship is as follows:

[0099] X l+1 =MB l (X l =MLP(TokenMixer(X) l (2)

[0100] Among them, Xl This is the feature map of the l-th MB. Further, a stage is defined as a stack of multiple meta-blocks with features of the same spatial size, denoted as a Stage. For example... Figure 2 As shown, the network consists of four stages, each with an embedding operation to project the embedding dimension and downsampled token length.

[0101] The improved YOLOv5 neural network not only reduces a large number of parameters and size, but also improves accuracy compared to the original YOLOv5 neural network.

[0102] This paper integrates a spatiotemporal context fusion strategy into object detection and electronic image stabilization, combining the two into an inseparable whole. Object detection acquires the category and location information of the object being measured, which is then applied to electronic image stabilization. In turn, electronic image stabilization serves object detection, eliminating the impact of motion-induced shaking caused by the unmanned surface vessel (USV). The temporal context is reflected in the feature point detection and matching process during electronic image stabilization, which is accomplished by comparing the motion relationship between the current frame and the previous frame. The spatial context is reflected in the object detection recording of the bounding box position; feature point detection and matching are performed on images within the same bounding box, narrowing the image range and avoiding mismatches that are prone to occur with large areas of overlapping water and sky.

[0103] Electronic image stabilization consists of two parts: the first part is feature point detection and matching, and the second part is motion compensation.

[0104] The feature point detection algorithm in this paper adopts the SIFT algorithm, and the specific process is as follows:

[0105] ① By constructing Gaussian difference pyramids with different scales and parameters through Gaussian filtering, the image locations at all scales are searched. The extreme points are identified by simulating the Gaussian differential function through the Gaussian difference pyramids. The Gaussian scale space formula for image I(a,b) is as follows:

[0106]

[0107] Where L is the image representation at different resolutions; G is the Gaussian function; I is the image size; (a,b) is the pixel position of the image, where a is the horizontal coordinate and b is the vertical coordinate; σ represents the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the scale. Large scale corresponds to the overall features of the image, and small scale corresponds to the detailed features of the image. Represents convolution operation;

[0108] ② After obtaining the feature points at different scales, a Gaussian difference scale space is generated from the Gaussian scale space. The Gaussian difference scale space can be represented by the following formula:

[0109]

[0110] Where o is the proportionality coefficient;

[0111] ③ Threshold the image to remove noise, retaining regions that satisfy the following formula:

[0112]

[0113] Where T is set to 0.04, n is the number of images to be extracted, and abs(val) is the pixel value of the image;

[0114] ④ Find the extreme point a0 in the Gaussian difference pyramid;

[0115] ⑤ At the detected extreme points, the position of the feature points is corrected by performing a ternary second-order Taylor expansion using the following formula, so that the feature points appear at sub-pixel positions:

[0116]

[0117] Where f(a) is the ternary second-order Taylor expansion at the extreme point; for transpose; f T It is the transpose of f(a); The sign of the partial derivative;

[0118] ⑥ Points that satisfy the following formula are defined as points with low contrast and are removed;

[0119]

[0120] ⑦ Since the extracted target feature points are corner points rather than edges, and the aforementioned measures can only guarantee the capture of points with drastic grayscale value changes, while edge points also meet this characteristic, we will remove points that satisfy the matrix determinant Det(M) < 0 and those whose matrix determinant and trace do not satisfy this condition. The feature points are used to remove edge points, where M is the Hessian matrix, Tr(M) is the trace of M, Det(M) is the determinant of M, and γ0 is a meaningful empirical value, which is set to 10 here.

[0121] After feature point detection, this paper uses a nearest / second nearest neighbor feature point matching strategy to match feature points of two images one by one, as shown in the following formula:

[0122]

[0123] The ratio is the ratio obtained by dividing the nearest distance by the second nearest distance.

[0124] Finally, motion compensation for the entire image is performed using an image transformation matrix filter based on adaptive Kalman filtering. The specific steps are as follows:

[0125] ① Observation noise and prediction noise are denoted as and The state equations and observation equations are as follows:

[0126]

[0127]

[0128] Where S t Φ is the state vector. t,t-1 Z is the state transition matrix; t H is the observation vector; t W is the observation matrix. t-1 For system noise; V t To observe noise; This represents the mean of the observed noise. Let the mean of the predicted noise be denoted as .

[0129] ② Calculate the predicted state And noise covariance matrix P t,t-1 ;

[0130]

[0131]

[0132] Where Φ t,t-1 T For Φ t,t-1 Transpose of; Here is the system noise covariance matrix;

[0133] ③ Update filter gain K t ;

[0134]

[0135] Where H t T For H t Transpose of; The covariance matrix of the observation noise;

[0136] ④ Calculate the residual ε t ;

[0137]

[0138] ⑤ Update the state vector and noise covariance matrix;

[0139]

[0140] P t=(IK t H t )P t,t-1 (16)

[0141] ⑥ Calculate the weighting coefficient d t In the formula, c is the relaxation factor;

[0142]

[0143] ⑦ Update Considering that measurement accuracy is positively correlated with the logarithm of matching points, update

[0144]

[0145]

[0146]

[0147]

[0148] In the formula, g is the number of matching point pairs e t The larger the value of g, the greater the impact of the number of points on measurement accuracy; ε t T For ε t The transpose of .

[0149] In addition to the above embodiments, the present invention may have other implementation methods. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A spatiotemporal context fusion-based electronic image stabilization and target detection method for unmanned surface vessels, comprising the following steps: Step S1: Read real-time image frames from the camera; Step S2 uses an improved YOLOv5 neural network to detect all targets; the improved YOLOv5 neural network includes: (1) Replace the anchor box generation algorithm used by the original YOLOv5 neural network with the K-Means++ algorithm. The K-Means++ algorithm includes the following steps: ① Randomly select a center u1 among the data points, where u1 refers to the first center point; ② For each data point x that has not yet been selected, calculate Where i is the label of the center point; j is the number of center points; and distance is the distance; that is, calculating the distance between each data point x and the center point that is closest to that data point. ③ Use a weighted probability distribution to randomly select a new data point as the new center, where the probability of the selected data point x is proportional to the probability of the selected data point x. Proportional; ④ Repeat steps ② and ③ until k center points are selected, where k is the number of categories; ⑤ Using the selected k center points as initial center points, perform the k-means algorithm; (2) Replace the C3 backbone of the YOLOv5 neural network with the EfficientFormer network: The EfficientFormer network consists of patch embeddings and a meta-transformer block stack, where patch embeddings are represented by PatchEmbed and meta-transformer block stacks are represented by MB, as shown below: In the formula, X0 is the input image, B is the batch size, [C, D] is the size of the feature map, y is the expected output, m is the total number of blocks, and MB consists of an unspecified token mixer and MLP blocks, where the token mixer is represented by TokenMixer and the MLP block is represented by MLP, and the composition relationship is as follows: X l+1 =MB l (X l )=MLP(TokenMixer(X l )) (2) Among them, X l It is the feature map of the l-th MB, and the stage is further defined as a stack of multiple meta-blocks with features of the same spatial size; Step S3 uses a spatiotemporal context fusion strategy to record the category of the water surface target and the position of the target box in the current image frame, continues to read the next frame image from the camera, and narrows the range of the two frames to within the target box; Step S4 uses the scale-invariant feature transformation algorithm and the nearest neighbor to second nearest neighbor feature point matching strategy to perform feature point detection and matching on the two frames of images after the range was reduced in step S3. Step S5 uses an image transformation matrix filter based on adaptive Kalman filtering to perform motion compensation on the image read in step S3, and then jumps to S2.

2. The method for electronic image stabilization and target detection of unmanned surface vessels based on spatiotemporal context fusion according to claim 1, wherein the spatiotemporal context fusion strategy in step S3 includes: A spatiotemporal context fusion strategy is incorporated into object detection and electronic image stabilization. The temporal context is reflected in the feature point detection and matching in the electronic image stabilization process by comparing the motion relationship between the current frame and the previous frame. Spatial context is reflected in the location of the detection box in the target detection record. Feature point detection and matching are performed on images within the same target box, which narrows the image range and avoids mismatches that are very likely to occur with large areas of repeated water surfaces and sky.

3. The method for electronic image stabilization and target detection of unmanned surface vessels based on spatiotemporal context fusion according to claim 1, wherein the scale-invariant feature transformation algorithm in step S4 comprises the following steps: ① By constructing Gaussian difference pyramids with different scales and parameters through Gaussian filtering, the image locations at all scales are searched. The extreme points are identified by simulating the Gaussian differential function through the Gaussian difference pyramids. The Gaussian scale space formula for image I(a,b) is as follows: Where L is the image representation at different resolutions; G is the Gaussian function; I is the image size; (a, b) is the pixel position of the image, where a is the horizontal coordinate and b is the vertical coordinate; σ represents the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the scale. Large scale corresponds to the overall features of the image, and small scale corresponds to the detailed features of the image. Represents convolution operation; ② After obtaining the feature points at different scales, a Gaussian difference scale space is generated from the Gaussian scale space. The Gaussian difference scale space can be represented by the following formula: Where o is the proportionality coefficient; ③ Threshold the image to remove noise, retaining regions that satisfy the following formula: Where T is set to 0.04, n is the number of images to be extracted, and abs(val) is the pixel value of the image; ④ Find the extreme point a0 in the Gaussian difference pyramid; ⑤ At the detected extreme points, the position of the feature points is corrected by performing a ternary second-order Taylor expansion using the following formula, so that the feature points appear at sub-pixel positions: Where f(a) is the ternary second-order Taylor expansion at the extreme point; for transpose; f T It is the transpose of f(a); The sign of the partial derivative; ⑥ Points that satisfy the following formula are defined as points with low contrast and are removed; ⑦ Remove matrices that satisfy the condition Det(M) < 0, or whose determinant and trace do not satisfy this condition. The feature points are used to remove edge points, where M is the Hessian matrix; Tr(M) is the trace of M; Det(M) is the determinant of M; γ0 is a meaningful empirical value, which is set to 10 here.

4. The method for electronic image stabilization and target detection of unmanned surface vessels based on spatiotemporal context fusion according to claim 1, wherein the nearest neighbor to second nearest neighbor feature point matching strategy in step S4 is formulated as follows: The ratio is the ratio obtained by dividing the nearest distance by the second nearest distance.

5. The method for electronic image stabilization and target detection of unmanned surface vessels based on spatiotemporal context fusion according to claim 1, wherein the image transformation matrix filtering based on adaptive Kalman filtering in step S5 includes the following steps: ① Observation noise and prediction noise are denoted as and The state equations and observation equations are as follows: Where S t Φ is the state vector. t，t-1 Z is the state transition matrix; t H is the observation vector; t W is the observation matrix. t-1 For system noise; V t To observe noise; This represents the mean of the observed noise. Let the mean of the predicted noise be denoted as . ② Calculate the predicted state And noise covariance matrix K; Where Φ t，t-1 T For Φ t，t-1 Transpose of; Here is the system noise covariance matrix; ③ Update filter gain K t ; Where H t T For H t Transpose of; The covariance matrix of the observation noise; ④ Calculate the residual ε t ; ⑤ Update the state vector And noise covariance matrix P t ; P t =(I-K t H t )P t,t-1 (16) ⑥ Calculate the weighting coefficient d t In the formula, c is the relaxation factor; ⑦ Update Considering that measurement accuracy is positively correlated with the logarithm of matching points, update In the formula, g is the number of matching point pairs e t The larger the value of g, the greater the impact of the number of points on measurement accuracy; ε t T For ε t The transpose of .

Images