A method for calculating the speed of tidal bore propagation based on a drone and a deep SORT algorithm

Abstract

The application innovatively combines the DeepSORT algorithm with the monocular ranging principle of the unmanned aerial vehicle, and proposes a surge propagation speed calculation method based on the unmanned aerial vehicle and the DeepSORT algorithm. Compared with the traditional surge observation platform, the application takes the unmanned aerial vehicle as the surge observation platform, and has the characteristics of being mobile and flexible and having a wide observation range. For the surge which is powerful and has a strong destructive power, the application can realize the observation of the large-range surge propagation process. Compared with the application of the traditional image processing algorithm in the field of surge observation, the application can realize the continuous identification and tracking of the surge propagation process with the help of the DeepSORT algorithm, and can draw the surge motion trajectory in the image picture in real time and calculate the pixel moving speed of the surge target in the image coordinate system. Combined with the monocular ranging principle of the unmanned aerial vehicle, the pixel moving speed is converted into the actual propagation speed of the surge. The application can provide important data support for the research on the surge propagation mechanism.

Description

Technical Field

[0001] This invention belongs to the field of marine information and relates to a method for observing tidal bores, specifically a method for calculating the propagation speed of tidal bores based on unmanned aerial vehicles and the DeepSORT algorithm. Background Technology

[0002] Tidal bores are a special tidal phenomenon occurring in funnel-shaped estuaries or bays, characterized by increased tidal range. During high tide, the tidal wave enters the estuary or bay, where the water level suddenly narrows, the bottom slope becomes steeper, and a large volume of water enters the narrow channel, concentrating energy and causing a sharp increase in amplitude. Simultaneously, the water near the bottom of the tidal wave moves at a slower speed than the water above, due to bottom friction, thus making the leading edge of the wave crest steeper. This steepening gradually intensifies as the water depth decreases and the river flow pushes the wave forward. After propagating a certain distance, the tidal crest rises and tilts forward, forming a tidal bore, resembling a vertical wall of water advancing with tremendous force; this is called a tidal bore. Approximately 450 estuaries worldwide are affected by tidal bores. The large amount of energy contained in tidal bores during their propagation significantly impacts navigation, water-related structures, and the production and lives of people along the river. Therefore, research on the propagation mechanism and laws of tidal bores is of paramount importance for the protection of tidal bore resources and disaster prevention along riverbanks.

[0003] Tidal bore propagation speed refers to the forward speed of the tidal bore head. It is an important characteristic parameter for studying the propagation mechanism and laws of tidal bores. The propagation speed of tidal bores is related to the water depth and flow velocity before and after the tidal bore. By studying more convenient and efficient methods for calculating the propagation speed of tidal bores, important technical support can be provided for the study of the laws of tidal bore propagation.

[0004] Current research on tidal bores primarily utilizes fixed tidal gauge stations on the shore for continuous tidal level observations. The propagation speed of the tidal bore is calculated based on the continuously observed data. Some scholars have also established methods for calculating the propagation speed of tidal bores through flume model experiments and theoretical analysis. In the patent "A Tidal Bore Propagation Speed ​​Measurement Device Based on Step Jump Recognition" (Publication No.: CN203204010U), a novel tidal bore propagation speed measurement device is disclosed. This invention aims to overcome the problem that the detected flow velocity cannot represent the actual tidal bore propagation speed due to the inconsistency between the local flow velocity and the overall flow velocity when the tidal bore passes through. It provides a tidal bore propagation speed measurement device based on step jump recognition, using a radar current meter to measure the flow velocity and the tidal bore propagation speed, providing the necessary information for tidal bore forecasting. However, this method is limited by the observation points and can only obtain relevant data from specific points. The patent "A method for observing the propagation speed of tidal bore by combining flight path mission and virtual control" (publication number: CN115079716A) discloses a method for observing the propagation speed of tidal bore by combining flight path mission and virtual control. This patent uses a UAV observation platform and image processing technology to realize the dynamic tracking of the tidal bore by the UAV, and then calculates the propagation speed of the tidal bore at different spatial points. However, this patent does not consider the impact of the changes in the morphological characteristics of the tidal bore under different time and space on its tidal bore line detection algorithm, and can only identify and track the tidal bore line of a single tidal bore in the image. Its application scenario is limited to a single tidal bore with a relatively obvious tidal bore.

[0005] To address the aforementioned problems, this invention proposes a method for calculating tidal bore propagation velocity based on UAVs and the DeepSORT algorithm. UAVs, with their maneuverability and wide observation range, can easily observe the large-scale propagation process of powerful and destructive tidal bores. The DeepSORT deep learning algorithm enables continuous identification and tracking of the tidal bore propagation process, and the propagation velocity is calculated using the UAV's monocular ranging principle, providing crucial data support for the study of tidal bore propagation mechanisms. This invention combines a UAV observation platform with deep learning algorithms and applies them to the field of tidal bore observation to measure tidal bore propagation velocity. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing methods for calculating tidal bore propagation speed. By combining a UAV observation platform and the DeepSORT deep learning algorithm, a method for calculating tidal bore propagation speed based on UAV and DeepSORT algorithm is proposed. This method has the characteristics of high security, convenient operation, wide observation range and high measurement accuracy, and can quickly and efficiently complete the measurement of tidal bore propagation speed.

[0007] A method for calculating tidal bore propagation velocity based on UAVs and the DeepSORT algorithm, the process is as follows: Figure 1As shown, it includes the following steps:

[0008] Step 1: Use the remote controller to control the drone to fly above the river and hover, and use the gimbal camera on the drone to acquire real-time images, and send the acquired real-time images to the PC.

[0009] Step 2: Using the tidal bore target detection model trained in the detector of the DeepSORT algorithm, target detection is performed on the input real-time image to identify the tidal bore targets in the image;

[0010] Step 3: Based on the target detection results of consecutive frames, use the target tracking model in the tracker of the DeepSORT algorithm to lock the tidal bore target that needs to be tracked in the target detection results, and assign a target ID to it to realize tidal bore target tracking;

[0011] Step 4: Based on the target tracking results, record the image coordinates of the tidal bore targets in each frame of the image in real time, and draw the motion trajectory of each tidal bore target in the image.

[0012] Step 5: Calculate the propagation speed of each tidal bore target;

[0013] Step 6: Complete the task of measuring the propagation speed of the tidal bore in the designated river section, save the results in the database, and return the drone to the starting point.

[0014] The construction steps of the tidal surge target detection model in step 2 are as follows:

[0015] Step 2.1: Collect the dataset. In order to make the target detection model more robust and representative, the data collection process should fully consider the temporal and spatial variation characteristics of the tidal bore. In terms of time, the tidal bore variation pattern is different in different tide periods. The data collection should cover the tidal bore pattern data under different tidal bore intensities, such as spring tide, mid-tidal, and neap tide. In terms of space, the tidal bore pattern formed by different river sections due to geographical differences is different. The data collection should cover the tidal bore pattern data of different typical river sections.

[0016] Step 2.2: Training dataset. The collected tidal bore data is labeled using labelImg software. The label format is txt file. The labeled file corresponds one-to-one with the image file. Each labeled file corresponds to the target information in one image. After labeling, the image files and labeled files are divided into training set, validation set and test set in a ratio of 4:1:1. The training set and validation set are used to train the neural network, and the test set is used to test the accuracy of the model generated after training.

[0017] The tidal bore target tracking in step 3 includes the following steps:

[0018] Step 3.1: After performing target detection on the input frame, for each tidal surge target, the feature extraction network extracts features from its target region and transforms them into high-dimensional feature vectors. These feature vectors can describe the appearance, motion, and semantic information of the target object.

[0019] Step 3.2: In each frame, target association is performed using the appearance and motion features between targets. The similarity between targets is evaluated by calculating the similarity measure between feature vectors, thereby determining the correspondence of the same target in different frames.

[0020] Step 3.3: Use Kalman filtering to estimate and predict the state of the tidal bore target, including its position, velocity, and acceleration. Kalman filtering can fuse prior information and observation information by predicting and updating the state, thereby achieving the estimation and prediction of the target state. It can effectively estimate the state of the target and provide the ability to predict the future state even in the presence of measurement noise and system model errors, enabling the target tracking algorithm to continuously track the target.

[0021] Step 3.4: Use the Hungarian algorithm to associate the target detection results in the current frame with the previously tracked targets, assign a new ID to the newly detected targets, and retain the existing IDs of the tracked targets to ensure that each detected tidal target has a unique ID and to continue tracking.

[0022] Furthermore, the similarity metric in step 3.2 is obtained by linearly weighting both Mahalanobis distance and cosine distance. The Mahalanobis distance calculation formula is as follows:

[0023]

[0024] Where, d j y represents the position of the j-th detection box. i S represents the predicted position of the target by the i-th tracker. i This represents the covariance matrix between the detection box and the predicted box;

[0025] Cosine distance calculation formula:

[0026]

[0027] Where, r j This represents the feature vector extracted from the j-th detection box. This represents the k-th vector of the most recent matching result in the i-th predicted trajectory;

[0028] The final similarity measurement formula is:

[0029] c i,j=λd1(i,j)+(1-λ)d2(i,j)

[0030] Where λ is the weighting coefficient, d1(i,j) is the Mahalanobis distance between the i-th tracker and the j-th detector, and d2(i,j) is the minimum cosine distance between all associated feature vectors of the i-th tracker and the feature vector of the j-th detection result.

[0031] Furthermore, in step 3.3, the Kalman filter estimates and predicts the target state through a prediction phase and an update phase. The prediction phase includes:

[0032]

[0033] P k - =FP k-1 F T +Q

[0034] Where F is the state transition matrix, This is the posterior estimate from the previous time step. P is the prior estimate of k at the current time. k-1 Let P be the posterior mean square error matrix of the previous time step. k - Let be the mean squared error matrix of the prior estimate at time k, and Q be the noise matrix;

[0035] Update phase:

[0036] K = P k - H T HP k - H T +R) -1

[0037]

[0038]

[0039] P k =(I-KH)P k -

[0040] Where P is the prior estimate covariance matrix, H is the observation matrix, R is the measurement state covariance matrix, and is the Kalmar gain matrix. z is the prior estimate. k Let y be the measured value and y be the prior estimate. After being projected onto the measurement space by the observation matrix H, it is compared with the measured value z. k The calculated residuals, For the updated posterior estimate, P k This is the updated posterior estimated covariance matrix.

[0041] The formula for calculating the propagation speed of the tidal bore target in step 5 is as follows:

[0042]

[0043] Where x1 and y1 are the pixel coordinates of the trajectory point at time t1 in the trajectory of the tidal bore target, x2 and y2 are the pixel coordinates of the trajectory point at time t2 in the trajectory of the tidal bore target, H is the height of the UAV above the river surface, f is the focal length of the gimbal camera, L is the size of the image sensor of the gimbal camera, and N is the number of image pixels.

[0044] The beneficial effects of this invention are as follows:

[0045] This invention innovatively combines the DeepSORT deep learning algorithm with the principle of UAV monocular ranging to achieve efficient and rapid measurement of tidal bore propagation speed.

[0046] A classic tidal bore target detection model is proposed. In the process of collecting tidal bore data, the temporal and spatial variation characteristics of tidal bores are fully considered. In terms of time, the tidal bore data collection covers tidal bore data under different tidal bore intensities such as spring tide, mid-tidal, and neap tide. In terms of space, the tidal bore data collection covers tidal bore data of different typical river sections, so that the tidal bore target detection model has better robustness and representativeness, and is more in line with the identification and tracking in the actual situation of tidal bores.

[0047] A formula for calculating the propagation speed of tidal bore based on the principle of monocular ranging is proposed. First, the number of pixels that the tidal bore target moves per second in the image is calculated, which is the pixel velocity of the tidal bore target in the image coordinate system. Then, the actual size of the pixels in the image is calculated by monocular ranging, and the actual propagation speed of the tidal bore target is calculated accordingly. Attached Figure Description

[0048] Figure 1 This is a flowchart of a method for calculating tidal bore propagation speed based on drones and the DeepSORT algorithm;

[0049] Figure 2 This is a diagram showing the recognition results of object detection;

[0050] Figure 3 This is a diagram showing the effect of target tracking;

[0051] Figure 4 This is a rendering of the tidal bore's movement trajectory. Detailed Implementation

[0052] The specific implementation of the present invention is as follows:

[0053] Step 1: First, mount the camera on the drone using a three-axis gimbal. Then, use the remote controller to control the drone to fly above the river and hover. Adjust the gimbal camera's pitch angle to a top-down angle and adjust the gimbal camera's attitude so that the image is as parallel to the shore as possible. Use the remote controller to send the real-time image captured by the gimbal camera to the PC.

[0054] Step 2: The drone transmits the images captured by the gimbal camera to the PC via the remote controller. Then, the PC uses a trained tidal bore target detection model to perform frame-by-frame recognition of the transmitted images and displays the results, such as... Figure 2 As shown;

[0055] Step 3: Input the target detection results from Step 2 into the target tracking model, extract the feature vector of each detected tidal bore target, and then match the tidal bore targets in the current frame with those in previous frames by calculating the similarity between the tidal bore target features. Perform state estimation and prediction for each tidal bore target to achieve continuous tidal bore target tracking, and assign an ID to each tidal bore target, such as... Figure 3 As shown;

[0056] Step 4: Based on the target tracking results, a trajectory management mechanism is used to track the target's motion trajectory. When a new target is associated with an existing trajectory, that trajectory needs to be initialized. This includes assigning a unique trajectory ID to the trajectory and initializing its state, position, velocity, and other information. In each frame, associated targets update their state estimates and position information. Based on the results of target association and data association, the target detection results in the current frame are associated with the corresponding trajectories, and Kalman filtering is used to update the trajectory's state estimate, ensuring the accuracy and consistency of the tracked target trajectory and enabling it to handle situations where targets are lost or new targets appear, thereby improving the robustness and accuracy of target tracking. Figure 4 As shown;

[0057] Step 5: Calculate the actual size represented by the pixels in the image by using the drone's flight altitude and the parameters of the gimbal camera, and then calculate the propagation speed of the tidal bore target by the number of pixels traversed by the tidal bore target's trajectory over a period of time.

[0058] Step 6: Complete the task of measuring the propagation speed of the tidal bore in the designated river section, save the results in the database, and return the drone to the starting point.

[0059] The construction steps of the tidal surge target detection model in step 2 are as follows:

[0060] Step 2.1: Data Collection. To ensure the robustness and representativeness of the target detection model, the temporal and spatial variations of tidal bores should be fully considered during data collection. Temporally, the tidal bore patterns differ across different tidal periods, and data collection should cover tidal bore patterns under different tidal bore intensities, such as spring tides, mid-tidal periods, and neap tides. Spatially, the tidal bore patterns formed by geographical differences in different river sections vary, and data collection should cover tidal bore patterns in different typical river sections.

[0061] Step 2.2: Training Dataset. The collected tidal bore data is labeled using labelImg software. The label format is txt file, and the label file corresponds one-to-one with the image file. Each label file corresponds to the target information in one image. After labeling, the image files and label files are divided into training set, validation set and test set in a ratio of 4:1:1. The training set and validation set are used to train the neural network, and the test set is used to test the accuracy of the model generated after training.

[0062] Step 3, tidal surge target tracking, includes the following steps:

[0063] Step 3.1: After performing object detection on the input frame, a deep learning feature extraction network is used for each target object to extract features from its target region and transform them into high-dimensional feature vectors. These feature vectors can describe the appearance, motion, and semantic information of the target object.

[0064] Step 3.2: In each frame, target association is performed using the appearance and motion features between targets. The similarity between targets is evaluated by calculating the similarity measure between feature vectors, thereby determining the correspondence of the same target in different frames.

[0065] Similarity measure c i,j It is calculated using the following formula:

[0066]

[0067]

[0068] c i,j =λd1(i,j)+(1-λ)d2(i,j)

[0069] Step 3.3: Use Kalman filtering to estimate and predict the state of the tidal bore target, including its position, velocity, and acceleration. Kalman filtering can fuse prior and observational information by predicting and updating the state, thereby achieving the estimation and prediction of the target state. It can effectively estimate the target state even in the presence of measurement noise and system model errors, and provides the ability to predict future states, enabling the DeepSORT algorithm to continuously track the target.

[0070] Kalman filtering achieves estimation and prediction of the target state through a prediction phase and an update phase. The prediction phase includes:

[0071]

[0072] P k - =FP k-1 F T +Q

[0073] Update phase:

[0074] K = P k - H T HP k - H T +R) -1

[0075]

[0076]

[0077] P k =(I-KH)P k -

[0078] Step 3.4: Use the Hungarian algorithm to associate the target detection results in the current frame with the previously tracked targets, assign a new ID to the newly detected targets, and retain the existing IDs of the tracked targets to ensure that each detected tidal target has a unique ID and to continue tracking.

[0079] The propagation speed of the tidal bore target in step 5 is calculated using the following formula:

[0080]

[0081] The above are preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions made by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for calculating tidal bore propagation velocity based on UAVs and the DeepSORT algorithm, characterized in that, Includes the following steps: Step 1: Use the remote controller to control the drone to fly above the river and hover, and use the gimbal camera on the drone to acquire real-time images, and send the acquired real-time images to the PC. Step 2: Using the tidal bore target detection model trained in the detector of the DeepSORT algorithm, target detection is performed on the input real-time image to identify the tidal bore targets in the image; Step 3: Based on the target detection results of consecutive frames, use the target tracking model in the tracker of the DeepSORT algorithm to lock the tidal bore target that needs to be tracked in the target detection results, and assign a target ID to it to realize tidal bore target tracking; Step 4: Based on the target tracking results, record the image coordinates of the tidal bore targets in each frame of the image in real time, and draw the motion trajectory of each tidal bore target in the image. Step 5: Calculate the propagation speed of each tidal bore target; Step 6: Complete the task of measuring the propagation speed of the tidal bore in the designated river section, save the results in the database, and return the drone to the starting point; The tidal bore target tracking in step 3 includes the following steps: Step 3.1: After performing target detection on the input frame, for each tidal bore target, the feature extraction network extracts features from its target region and transforms them into high-dimensional feature vectors. These feature vectors can describe the appearance, motion and semantic information of the target object. Step 3.2: In each frame, target association is performed using the appearance and motion features between targets. The similarity between targets is evaluated by calculating the similarity measure between feature vectors, thereby determining the correspondence of the same target in different frames. Step 3.3: Use Kalman filtering to estimate and predict the state of the tidal bore target, including its position, velocity, and acceleration. Kalman filtering can fuse prior information and observation information by predicting and updating the state, thereby achieving the estimation and prediction of the target state. It can effectively estimate the state of the target and provide the ability to predict the future state even in the presence of measurement noise and system model errors, enabling the target tracking model to continuously track the target. Step 3.4: Use the Hungarian algorithm to associate the target detection results in the current frame with the previously tracked targets, assign a new ID to the newly detected targets, and retain the existing IDs of the tracked targets to ensure that each detected tidal target has a unique ID and to perform continuous tracking. Furthermore, the similarity metric described in step 3.2 of the tidal surge target tracking process is obtained through a linear weighted calculation of Mahalanobis distance and cosine distance. The Mahalanobis distance calculation formula is as follows: Where, d j y represents the position of the j-th detection box. i S represents the predicted position of the target by the i-th tracker. i This represents the covariance matrix between the detection box and the predicted box; Cosine distance calculation formula: Where, r j This represents the feature vector extracted from the j-th detection box. This represents the k-th vector of the most recent matching result in the i-th predicted trajectory; The final similarity measurement formula is: c i,j =λd1(i,j)+(1-λ)d2(i,j) Where λ is the weight coefficient, d1(i,j) is the Mahalanobis distance between the i-th tracker and the j-th detector, and d2(i,j) is the minimum cosine distance between all associated feature vectors of the i-th tracker and the feature vector of the j-th detection result. Furthermore, the Kalman filter described in step 3.3 of the tidal surge target tracking process estimates and predicts the target state through a prediction phase and an update phase. The prediction phase includes: x k - =Fx k-1 P k - =FP k-1 F T +Q Where F is the state transition matrix, x k-1 x is the posterior estimate from the previous time step. k - P is the prior estimate of k at the current time. k-1 Let P be the posterior mean square error matrix of the previous time step. k - Let be the mean squared error matrix of the prior estimate at time k, and Q be the noise matrix; Update phase: K=P k - H T (HP k - H T +R) -1 y=z k -Hx k - x k =x k - +K y P k =(I-KH)P k - Where P is the prior estimate covariance matrix, H is the observation matrix, R is the measurement state covariance matrix, and is the Kalmar gain matrix, x k - z is the prior estimate. k Let y be the measured value and y be the prior estimate of x. k - After being projected onto the measurement space by the observation matrix H, it is compared with the measured value z. k The calculated residual, x k For the updated posterior estimate, P k This is the updated posterior estimated covariance matrix.

2. The method for calculating tidal bore propagation velocity based on UAV and DeepSORT algorithm according to claim 1, characterized in that, The construction steps of the tidal surge target detection model in step 2 are as follows: Step 2.1: Collect the dataset. In order to make the target detection model more robust and representative, the data collection process should fully consider the temporal and spatial variation characteristics of the tidal bore. In terms of time, the tidal bore variation pattern is different in different tides. The data collection should cover the tidal bore pattern data under three different tidal bore intensities: spring tide, mid-tidal tide, and neap tide. In terms of space, the tidal bore pattern formed by different river sections due to geographical differences is different. The data collection should cover the tidal bore pattern data of different typical river sections. Step 2.2: Training dataset. The collected tidal bore data is labeled using labelImg software. The label format is txt file. The labeled file corresponds one-to-one with the image file. Each labeled file corresponds to the target information in one image. After labeling, the image files and labeled files are divided into training set, validation set and test set in a ratio of 4:1:

1. The training set and validation set are used to train the neural network, and the test set is used to test the accuracy of the model generated after training.

3. The method for calculating tidal bore propagation velocity based on UAV and DeepSORT algorithm according to claim 1, characterized in that, The formula for calculating the propagation speed of the tidal bore target in step 5 is as follows: Where x1 and y1 are the pixel coordinates of the trajectory point at time t1 in the trajectory of the tidal bore target, x2 and y2 are the pixel coordinates of the trajectory point at time t2 in the trajectory of the tidal bore target, H is the height of the UAV above the river surface, f is the focal length of the gimbal camera, L is the size of the image sensor of the gimbal camera, and N is the number of image pixels.

Images