Eagre observation method and system

Abstract

The invention relates to the technical field of river observation, in particular to an eagre observation method and system, and the method comprises the steps: controlling an unmanned plane carrying a pan-tilt camera to fly above a target river section, and obtaining a source image at an inclined visual angle; calculating an actual observation range of the image in a real geographic coordinate system, and determining coordinates of four corresponding corner points of the image in an overlooking view angle according to the actual observation range; obtaining a perspective transformation matrix based on four-corner corresponding points of the source image and the target top view image; and converting a video frame picture in an inclined view angle into a video frame picture in an overlooking view angle in real time by using the matrix. The method has the advantages that the size of an observation picture of the unmanned aerial vehicle is increased, the observation range of the unmanned aerial vehicle is expanded, and the tidal bore target under the overlook view angle is completely observed.

Description

Technical Field

[0001] This invention relates to the field of river flow observation technology, and in particular to a method and system for observing tidal bores. Background Technology

[0002] Field observation is the foundation of tidal bore research. Traditional observation methods mainly rely on visual observation, but with the advancement of observation techniques, instruments and equipment of various types, such as pressure-based, acoustic, and remote sensing methods, are now widely used.

[0003] In the patent "An Automated Tidal Bore Observation System and Method Thereof" (Publication No.: CN110411419A), to address the problems of low efficiency, poor real-time performance, low accuracy of tide level readings, and the inability to eliminate the influence of high tide at the water gauge point, this patent provides an automated tidal bore observation system and method. This system employs an intelligent and information-based observation system to improve the efficiency and accuracy of tidal bore observation data. However, this method is limited by the observation points and can only observe data from specific locations. In the patent "A Method for Observing Tidal Bore Propagation Speed ​​Combining Flight Route Tasks and Virtual Control" (Publication No.: CN115079716A), a method combines UAV flight route tasks with virtual control to maintain dynamic tracking between the UAV and the tidal bore, thereby calculating the propagation speed of the tidal bore at different spatial points. This observation method allows for relatively easy observation of the large-scale tidal bore propagation process using UAVs.

[0004] However, drones are susceptible to flight policy restrictions, which limit their flight altitude. Tidal bores, as a target for drone observation, span a large geographical area. For example, during the observation of the Qiantang River tidal bore, due to the proximity of some areas to Xiaoshan Airport, there are drone altitude restriction zones in the sections from Jiangdong Bridge to Jiubao Bridge and from Pengbu Bridge to Xixing Bridge, limiting drone flight altitude to no more than 120 meters. From a 120-meter overhead viewpoint, drones can only observe a portion of the target tidal bore, resulting in lower accuracy of the tidal bore observation data.

[0005] In addition, due to the wideness of the river channel, existing technologies for measuring the propagation speed of tidal bores (i.e., the forward speed of the tidal bore) are mostly local measurements. Furthermore, the tidal bore is obstructed by groynes and bridges during its journey, causing significant changes in the local propagation speed. Local measurement methods cannot accurately measure the propagation speed of the entire target tidal bore.

[0006] Therefore, this case is brought. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for observing tidal bores, which solves the problem of low accuracy of tidal bore observation data under altitude-restricted conditions by unmanned aerial vehicles (UAVs) in the prior art.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A method for observing tidal bores, characterized by comprising the following steps: S1. Select the corresponding target river section according to the task, and control the drone equipped with a gimbal camera to fly above the surface of the target river section; S2. Adjust the drone's flight altitude and the gimbal camera's pitch angle to ensure the target area is within the image acquisition range, capture a source image from an oblique perspective, and calculate the actual observation range of the source image in the real coordinate system. S3. Based on the actual observation range of the source image under the tilted view, calculate the coordinates of the four corner points corresponding to the actual observation range in the target image under the top view. S4. Calculate the perspective transformation matrix based on the coordinates of the four corner points of the source image and the corresponding coordinates of the four corner points of the target image; S5. Use the perspective transformation matrix to perform geometric transformation on the source image, and convert the video frame images captured by the gimbal camera from the tilted viewpoint into video frame images from the top viewpoint through the perspective transformation matrix. S6. Output the transformed target image result, complete the target river section observation task, and the UAV returns to the starting point.

[0010] Furthermore, step S5 also includes the following process: calculating the propagation speed of the tidal bore based on the video frames from the top-down view.

[0011] Furthermore, the calculation of the propagation speed includes the following steps: S51. Using the tidal bore target detection model in the detector of the target detection algorithm, target detection is performed on the video frame from the top-down view to identify the tidal bore target; S52. Based on the target detection results, use the target tracking model in the tracker of the target detection algorithm to locate the tidal bore target that needs to be tracked in the target detection results and assign it a target ID; S53. Based on the target tracking results, record the image coordinates of the tidal bore target in each frame of the image in real time, and draw the motion trajectory of each tidal bore target in the image. S54. Calculate the propagation speed of each tidal bore target based on the described trajectory and known spatial scale information.

[0012] Furthermore, in S2, the pitch angle of the gimbal camera is between 30° and 60°, preferably 45°.

[0013] A tidal bore observation system based on the method described above includes: The drone, equipped with a gimbal camera, is used to collect video from an oblique perspective above the target section of the river. The image processing module is used to calculate the actual observation range of the source image in the real coordinate system, and to generate video frame images from the top view based on the coordinates of the four corner points corresponding to the actual observation range in the target image from the top view, and to construct a perspective transformation matrix. The tidal bore analysis module is used to detect and track tidal bore targets in the overhead view video and calculate the propagation speed of the tidal bore targets. The communication module is used to output the tidal bore observation results.

[0014] The advantages of this invention are:

[0015] 1. Overcoming flight altitude limitations to achieve complete observation of the target tidal bore. When drones cannot ascend to the ideal altitude due to airspace control or safety requirements, traditional overhead-view photography is insufficient to cover the entire target tidal bore. This solution first uses an oblique angle to ensure the entire target tidal bore falls within the shooting range, and then corrects it into a virtual overhead image. This preserves the integrity of the target while obtaining video frames from an overhead perspective, which is beneficial for accurately measuring the characteristic parameters of the target tidal bore.

[0016] 2. Under flight altitude restrictions, the complete observation of the target tidal bore through an oblique perspective ensures the integrity of the target tidal bore from a top-down perspective, which is beneficial for measuring the propagation speed of the target tidal bore and thus improving the accuracy of tidal bore observation data. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the tidal bore observation method in the embodiment; Figure 2 This is a schematic diagram illustrating the conversion of the UAV's observation range from an oblique viewpoint to a top-down viewpoint in the embodiment. Figure 3 This is a schematic diagram illustrating the change in image coordinates when the observation range of the UAV changes from an oblique viewpoint to a top-down viewpoint in the embodiment. Figure 4 These are actual images taken during actual observation of the river surface in the embodiments; Figure 5 In the example, Figure 4 The image from the tilted viewpoint is converted into an image from the top-down viewpoint; Figure 6 The specific composition of each part of the perspective transformation matrix in the embodiment is shown below; Figure 7 This is the process of obtaining the target tidal bore propagation speed in the embodiment. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to embodiments. It should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., used in this document indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0019] This embodiment proposes a method for observing tidal bores. By maneuvering a drone equipped with a gimbal camera to fly above the target river section, adjusting the drone's flight altitude and the gimbal camera's pitch angle, the target tidal bore is brought within the image acquisition range. Inverse perspective transformation projects the image plane from an oblique viewpoint onto a new image plane from a top-down viewpoint, thus expanding the drone's observation range from the top-down perspective. Furthermore, the propagation speed of the target tidal bore can be calculated based on the video frames from the transformed top-down viewpoint. Figure 1 As shown, the specific steps include: S1. Select the corresponding target river section according to the task, and control the drone equipped with a gimbal camera to fly above the surface of the target river section; S2. Adjust the drone's flight altitude and the gimbal camera's pitch angle to ensure the target area is within the image acquisition range, capture a source image from an oblique perspective, and calculate the actual observation range of the source image in the real coordinate system. S3. Based on the actual observation range of the source image under the tilted view, calculate the coordinates of the four corner points corresponding to the actual observation range in the target image under the top view. S4. Calculate the perspective transformation matrix based on the coordinates of the four corner points of the source image and the corresponding coordinates of the four corner points of the target image; S5. Use the perspective transformation matrix to perform geometric transformation on the source image, and convert the video frame images captured by the gimbal camera from the tilted viewpoint into video frame images from the top viewpoint through the perspective transformation matrix. S6. Output the transformed target image result, complete the target river section observation task, and the UAV returns to the starting point.

[0020] Projecting an image plane from a tilted perspective onto a new image plane from a top-down perspective, and then converting the tilted-perspective image captured by the drone into a top-down perspective image through image transformation, can be understood as follows: when the entire tidal bore cannot be observed from a top-down perspective due to altitude limitations, the entire tidal bore is observed from a tilted perspective, and then the tilted perspective is converted into a top-down perspective through image transformation. This method is beneficial for increasing the size of the drone's observation frame under flight altitude limitations, expanding the drone's observation range, and allowing for a complete observation of the target tidal bore from a top-down perspective. This, in turn, facilitates the measurement of the tidal bore's propagation speed, thereby improving the accuracy of tidal bore observation data. It should be noted that the video frames from a top-down perspective are more conducive to measuring the characteristic parameters of the target tidal bore. These characteristic parameters can be the tidal bore's propagation speed, or other characteristic parameters such as the degree of wave breakage. Measuring these characteristic parameters in the video frames from a top-down perspective is more accurate.

[0021] As one implementation method, see Appendix Figure 2 As shown, attached Figure 2 This is a vertical view of the gimbal camera. In this image, point O represents the drone's position from the tilted perspective, △AOC represents the drone's shooting range from the tilted perspective, line segment AC represents the width of the river surface from the tilted perspective (value l), θ represents the vertical viewing angle of the gimbal camera, α represents the gimbal camera's pitch angle from the tilted perspective, line segment OD represents the drone's flight altitude from the tilted perspective (value h), and ds and dl represent the distances from the drone to points A and C on the river surface from the tilted perspective. Specifically, the gimbal camera's pitch angle α ranges from 30° to 60°. If the α value is too large, it will result in an excessively large image frame with too many noise points, which is detrimental to the identification of the tidal bore target in subsequent steps. If the α value is too small, the target tidal bore cannot be located within the image range. It should be noted that it is not required that the target tidal bore appear completely within the image range. When most of the target tidal bore appears within the image range, the measurement of the tidal bore is already of a certain accuracy. Preferably, in this embodiment, the value of α is taken as 45°, which helps to reduce noise points in the image and also helps to keep the target tidal bore within the observation range. Extend line segment OC to point B, making the length of OB equal to OA, and draw the altitude OF of △AOB. Then, rotate △AOB with A as the origin so that line segment AB coincides with the river surface, that is, △AOB coincides with △AO'B', and O' is the position of the UAV from the top-down view. It can be understood that, under the limitation of flight altitude, this method allows more target tidal bores to be located in the image from the top-down view, thereby improving the accuracy of tidal bore observation data. The corner coordinate calculation process in step S4 is as follows:

[0022] Appendix Figure 2In the diagram, the vertical viewing angle θ of the gimbal camera is 2arctan(film height / (2 * lens focal length)). Film height, width, diagonal value, and lens focal length are all parameters of the gimbal camera. The gimbal camera pitch angle α and the drone's flight altitude h are set values ​​for controlling the drone. Therefore, the following can be calculated using trigonometric functions: Cos(θ-α) = h / ds, that is, ds=h / Cos(θ-α); Cos(θ+α) = h / dl, that is, dl=h / Cos(θ+α).

[0023] Appendix Figure 3 This represents the observation range of the drone from a horizontal perspective. Rectangle X1X2X3X4 represents the image taken from a tilted viewpoint, and trapezoid X1'X2'X3'X4' represents the image converted to a top-down viewpoint. The distance from point O to the upper edge W of the trapezoid is also shown. 上 The distance from point O to the lower edge W of the trapezoid is dl. 下 The distance is ds. Set the resolution of the source image captured by the drone to a*b, that is, the length of the rectangle X1X2X3X4 is a and the width is b. With X3 as the origin, the coordinate values ​​of each corner point of the rectangle X1X2X3X4 can be obtained.

[0024] Horizontal viewing angle φ = 2arctan(film width / (2 * lens focal length)); W 上 =2*dl*tanφ; W 下 =2*ds*tanφ; This allows us to obtain the coordinates of each corner point of X1'X2'X3'X4'.

[0025] like Figure 4 and Figure 5 The image shown is the source image that was actually captured and the target image that was converted. Figure 5 The black part is Figure 2 The CB' portion in a vertical view.

[0026] In step S4, the perspective transformation matrix is ​​as follows: Figure 6 As shown, Zc represents the depth in the gimbal camera coordinate system (i.e., the Z coordinate of the object in the camera coordinate system); u and v represent the coordinates in the pixel coordinate system; dx and dy represent the physical size of each pixel on the image sensor (unit: mm / pixel); u0 and v0 represent the coordinates of the principal point (image center) in the pixel coordinate system (unit: pixel); f represents the focal length of the camera (unit: mm); R represents a 3×3 rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system; t represents a 3×1 translation vector, representing the translation from the world coordinate system to the camera coordinate system; 0 TThis represents a 1x3 zero vector; X, Y, and Z represent the coordinates of a 3D point in the world coordinate system, using homogeneous coordinates [X,Y,Z,1]. T The overall construction process: T1. Transform the world coordinates [X,Y,Z,1] to the camera coordinate system using the extrinsic parameter matrix (rotation R and translation t); T2. Project onto the image plane coordinate system using the projection matrix (focal length f); T3. Transform to pixel coordinate system using pixel coordinate system transformation matrix (pixel dx, dy and principal point u0, v0); T4. Finally, the pixel coordinates [u,v] are obtained, along with the depth information Zc.

[0027] Furthermore, as an implementation method, step S5 also includes the following process: calculating the propagation speed of the target tidal bore based on the video frames from the top-down perspective. This method allows more target tidal bores to be captured in the video frames from the top-down perspective, which is beneficial for measuring the propagation speed of the target tidal bore and thus improves the accuracy of tidal bore observation data.

[0028] like Figure 7 As shown, the calculation of the propagation speed includes the following steps: S51. Using the tidal bore target detection model (pre-trained) in the detector of the target detection algorithm, target detection is performed on the video frame images from the top-down perspective to identify the tidal bore target; S52. Based on the target detection results, use the target tracking model (pre-trained) in the tracker of the target detection algorithm to lock the tidal bore target that needs to be tracked in the target detection results, and assign it a target ID to achieve tidal bore target tracking; S53. Based on the target tracking results, record the image coordinates of the tidal bore target in each frame of the image in real time, and draw the motion trajectory of each tidal bore target in the image. S54. Calculate the propagation speed of each tidal bore target based on the described motion trajectory and known spatial scale information.

[0029] Specifically, in this embodiment, the tilted-view image captured by the drone is converted into a top-down view image. It can be understood that in a video frame from a tilted perspective, the target tidal bore is also tilted, which is not conducive to measuring the tidal bore's propagation speed. After converting the tilted-view image into a top-down view image, the top-down view video frame is more conducive to measuring the tidal bore's propagation speed. At the same time, the complete observation of the target tidal bore through the tilted-view perspective ensures the integrity of the target tidal bore from the top-down perspective, improving the completeness of observations and the accuracy of speed measurement for wide channels and complex tidal patterns. This method is beneficial for measuring the propagation speed of the target tidal bore, thereby improving the accuracy of tidal bore observation data.

[0030] Furthermore, in this embodiment, the object detection algorithm is a computer vision technique designed to automatically identify and locate target objects of a specific category from image or video frames. Its location is typically marked with a bounding box, along with a category label and confidence score. Common object detection algorithms include two-stage models (such as Faster R-CNN) and one-stage models (such as the YOLO series and SSD), each offering a trade-off between accuracy and speed. Building upon this, if continuous tracking of multiple targets in a video is required, a multi-object tracking (MOT) algorithm must be combined. DeepSORT (DeepSimple Online and Realtime Tracking) is an efficient and robust tracking framework: it first uses an object detector (such as YOLO) to detect targets frame by frame, then fuses motion information (using Kalman filtering to predict position) with appearance features (using a deep neural network to extract ReID features and calculate cosine similarity), and employs the Hungarian algorithm for optimal matching, thereby assigning a unique ID to each target and achieving stable cross-frame tracking. This method is beneficial for measuring the propagation speed of the target tidal bore in video frames from a top-down perspective, thereby improving the accuracy of tidal bore observation data.

[0031] Furthermore, as an implementation method, this embodiment also proposes a tidal bore observation system for the above method, comprising: The drone, equipped with a gimbal camera, is used to collect video from an oblique perspective above the target section of the river. The image processing module is used to calculate the actual observation range of the source image in the real coordinate system, and to generate video frame images from the top view based on the coordinates of the four corner points corresponding to the actual observation range in the target image from the top view, and to construct a perspective transformation matrix. The tidal bore analysis module is used to detect and track tidal bore targets in the overhead view video and calculate the propagation speed of the tidal bore targets. The communication module is used to output the tidal bore observation results.

[0032] The above implementation is only used to explain the concept of the present invention, and is not intended to limit the protection of the present invention. Any non-substantial modifications made to the present invention using this concept should fall within the protection scope of the present invention.

Claims

1. A method for observing tidal bores, characterized in that, Includes the following steps: S1. Select the corresponding target river section according to the task, and control the drone equipped with a gimbal camera to fly above the surface of the target river section; S2. Adjust the drone's flight altitude and the gimbal camera's pitch angle to ensure the target area is within the image acquisition range, capture a source image from an oblique perspective, and calculate the actual observation range of the source image in the real coordinate system. S3. Based on the actual observation range of the source image under the tilted view, calculate the coordinates of the four corner points corresponding to the actual observation range in the target image under the top view. S4. Calculate the perspective transformation matrix based on the coordinates of the four corner points of the source image and the corresponding coordinates of the four corner points of the target image; S5. Use the perspective transformation matrix to perform geometric transformation on the source image, and convert the video frame images captured by the gimbal camera from the tilted viewpoint into video frame images from the top viewpoint through the perspective transformation matrix. S6. Output the transformed target image result, complete the target river section observation task, and the UAV returns to the starting point.

2. The tidal bore observation method as described in claim 1, characterized in that, Step S5 also The process includes the following: calculating the propagation speed of the tidal bore based on video frames viewed from above.

3. The tidal bore observation method as described in claim 2, characterized in that, The calculation of the propagation speed includes the following steps: S51. Using the tidal bore target detection model in the detector of the target detection algorithm, target detection is performed on the video frame from the top-down view to identify the tidal bore target; S52. Based on the target detection results, use the target tracking model in the tracker of the target detection algorithm to locate the tidal bore target that needs to be tracked in the target detection results and assign it a target ID; S53. Based on the target tracking results, record the image coordinates of the tidal bore target in each frame of the image in real time, and draw the motion trajectory of each tidal bore target in the image. S54. Calculate the propagation speed of each tidal bore target based on the described motion trajectory and known spatial scale information.

4. The tidal bore observation method as described in claim 1, characterized in that, In the S2, the pitch angle of the gimbal camera is between 30° and 60°.

5. The tidal bore observation method as described in claim 4, characterized in that, In S2, the gimbal camera has a pitch angle of 45°.

6. A tidal bore observation system based on the method of any one of claims 1 to 5, characterized in that, include: The drone, equipped with a gimbal camera, is used to collect video from an oblique perspective above the target section of the river. The image processing module is used to calculate the actual observation range of the source image in the real coordinate system, and to generate video frame images from the top view based on the coordinates of the four corner points corresponding to the actual observation range in the target image from the top view, and to construct a perspective transformation matrix. The tidal bore analysis module is used to detect and track tidal bore targets in the overhead view video and calculate the propagation speed of the tidal bore targets. The communication module is used to output the tidal bore observation results.

Images