A river surface water flow velocity detection method based on three-eye multi-modal vision

Abstract

The application discloses a river surface water flow velocity detection method based on a three-eye multi-modal vision, and the process is as follows: step 1, video images of a river to be measured are collected through a left infrared camera, a right infrared camera and an RGB camera; step 2, target detection is carried out based on infrared video images of the left infrared camera and RGB video images of the RGB camera, and anchor frame center point pixel coordinates of each floating object target in each image are obtained; step 3, taking the RGB camera as a reference camera, the anchor frame center point pixel coordinates of each floating object target in each image of the reference camera are converted into coordinates under a world coordinate system; step 4, a multi-target tracking algorithm is used to track the floating object target to obtain a motion trajectory, and the floating object speed is estimated in combination with the motion trajectory and the coordinates under the world coordinate system, and then the water flow velocity is obtained. The application does not need other measuring instruments and can realize efficient positioning and detection.

Description

Technical Field

[0001] This invention relates to the field of river flow velocity detection methods, specifically a method for detecting surface water flow velocity in rivers based on tri-view multimodal vision. Background Technology

[0002] Measuring the velocity of water flow in rivers is of great significance for environmental protection and flood prediction. Currently, there are many methods for estimating river flow velocity using machine vision. For example, Chinese patent document CN118736463A discloses a method for measuring the velocity of water flow on the river surface based on deep learning, and Chinese patent document CN117788879A discloses a method for measuring the velocity of water flow on the river surface based on machine vision. Both of these methods require the use of markers of known size along the riverbank to determine the velocity, and their detection effect in low-light environments needs to be verified, which greatly limits their practical application. Summary of the Invention

[0003] This invention provides a method for detecting surface water velocity in rivers based on tri-view multimodal vision, in order to solve the problem that existing non-contact methods for measuring surface water velocity in rivers require the assistance of other tools for training during the measurement process.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for detecting surface water flow velocity in a river channel based on trioptic multimodal vision is described below:

[0006] Step 1: Acquire video images of the river channel to be measured using a tri-lens camera. The river channel contains several floating objects.

[0007] The tri-lens camera includes a left infrared camera, a right infrared camera, and an RGB camera. The left infrared camera, the right infrared camera, and the RGB camera simultaneously acquire video images of the river channel to be measured. The left infrared camera, the right infrared camera, and the RGB camera acquire video images of the river channel to be measured at the same frame rate and frame rate. The left infrared camera and the right infrared camera each acquire infrared video images of the river channel to be measured, and the RGB camera acquires RGB video images of the river channel to be measured.

[0008] Step 2: Perform multimodal detection of floating objects based on the infrared video images acquired by the left infrared camera and the RGB video images acquired by the RGB camera. As a result, the anchor frame and the pixel coordinates of the anchor frame center point of each floating object are obtained in each frame of the image acquired by the left infrared camera, and the anchor frame and the pixel coordinates of the anchor frame center point of each floating object are obtained in each frame of the image acquired by the RGB camera.

[0009] Step 3: Using the RGB camera as the reference camera, convert the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera into coordinates in the world coordinate system to obtain the coordinates of the center point of each floating target anchor frame in the world coordinate system in each frame of the reference camera.

[0010] Step 4: Use a multi-target tracking algorithm to track floating targets in each frame of images captured by the reference camera, thereby obtaining the motion trajectory of each floating target and the number of image frames contained in each motion trajectory;

[0011] The coordinates of the center point of the anchor frame of the corresponding floating object in the frame corresponding to the starting point of the trajectory of each floating object are taken as the world coordinates of the starting point of each floating object; the coordinates of the center point of the anchor frame of the corresponding floating object in the frame corresponding to the ending point of the trajectory of each floating object are taken as the world coordinates of the ending point of each floating object.

[0012] Then, based on the starting point coordinates, ending point coordinates, number of image frames contained in the corresponding motion trajectory, and video frame rate of the reference camera, the velocity estimate of each floating object is calculated.

[0013] Finally, based on the velocity estimates of each floating object, the water flow velocity on the surface of the river channel under test is calculated.

[0014] Furthermore, in step 2, the BV-YOLOv network is used to perform multimodal detection of floating objects on the infrared video images captured by the left infrared camera and the RGB video images captured by the RGB camera.

[0015] Furthermore, the backbone structure of the BV-YOLOv network is a cross-modal fusion backbone network formed by adding CFT modules to the backbone part of the YOLOv11 network. The Neck and Head structures of the BV-YOLOv network are the same as those of the Neck and Head parts of the YOLOv11 network.

[0016] Furthermore, in step 3, based on the corresponding frame images in the infrared video images acquired by the left infrared camera and the right infrared camera, and combined with the intrinsic and extrinsic parameters of the left infrared camera and the right infrared camera, the depth map corresponding to the corresponding frame image is obtained.

[0017] Using an RGB camera as a reference camera, the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera are substituted into the depth map of the corresponding frame to obtain the depth value of the center point of each floating target anchor frame in each frame of the reference camera.

[0018] Then, based on the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera, the intrinsic parameters of the reference camera, and the depth value of the center point of each floating target anchor frame in each frame of the reference camera, the coordinates of the center point of each floating target anchor frame in the reference camera coordinate system are obtained.

[0019] Finally, the coordinates of the center point of each floating target anchor frame in each frame of the reference camera in the reference camera coordinate system are converted into coordinates in the world coordinate system to obtain the coordinates of the center point of each floating target anchor frame in the world coordinate system in each frame of the reference camera.

[0020] Furthermore, in step 4, the ByteTrack algorithm is used to track floating targets in each frame of images captured by the reference camera.

[0021] Furthermore, in step 4, the mean of the velocity estimates of each floating object is calculated as the flow velocity of the river channel to be measured.

[0022] Furthermore, in step 4, the calculated flow velocity of the river channel to be measured is divided into multiple intervals, and the least squares estimation model is used to correct the calculated water flow velocity on the surface of the river channel to be measured based on each interval.

[0023] Compared with the prior art, the advantages of the present invention are:

[0024] In this invention, a tri-lens camera with three cameras (two infrared and one RGB) is used to estimate the water flow velocity in a non-contact manner by monitoring floating objects such as algae, plastic and paper on the river surface. This method does not require other measuring instruments during training or use, nor does it require bank markers or floating markers of known size to assist in network training, and can achieve efficient positioning and detection.

[0025] In this invention, the depth value is calculated directly based on the images acquired by the left and right infrared cameras, while the detection of floating objects adopts a multimodal BV-YOLO network that processes infrared and RGB images, which can achieve good detection performance even under low light conditions. This method improves the accuracy of water flow velocity estimation on the river surface and the ease of training, and has great application value. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the algorithm framework and process of an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of a trinocular camera according to an embodiment of the present invention.

[0028] Figure 3This is a schematic diagram of the backbone structure of BV-YOLO according to an embodiment of the present invention.

[0029] Figure 4 This is a structural diagram of the CFT according to an embodiment of the present invention.

[0030] Figure 5 This is a plan view of a binocular device according to an embodiment of the present invention.

[0031] Figure 6 This is a bird's-eye view of a binocular device according to an embodiment of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] like Figure 1 As shown in the figure, this embodiment discloses a method for detecting the surface water flow velocity of a river channel based on trioptic multimodal vision. The process is as follows:

[0034] Step 1: Acquire video images of the river channel to be measured using a tri-lens camera. The river channel contains several floating objects.

[0035] like Figure 2 As shown, the tri-lens camera used in this embodiment includes a left infrared camera, a right infrared camera, and an RGB camera, wherein the left infrared camera and the right infrared camera constitute a binocular device.

[0036] The left infrared camera, right infrared camera, and RGB camera simultaneously acquire video images of the river channel to be measured. Thus, the video images acquired by each camera contain floating objects (such as algae, plastic, paper, etc.) in the river channel to be measured. Furthermore, the left infrared camera, right infrared camera, and RGB camera acquire the same number of frames and the same video frame rate.

[0037] The left and right infrared cameras each capture infrared video images of the river channel being measured, while the RGB camera captures RGB video images of the river channel. The images captured by the left and right infrared cameras are used for subsequent multimodal detection of floating objects, and the images captured by the left and right infrared cameras are used to obtain the depth map.

[0038] Step 2: Using the BV-YOLO network, perform multimodal detection of floating objects on the infrared video images captured by the left infrared camera and the RGB video images captured by the RGB camera. The BV-YOLO network predicts the anchor frame of each floating object in each frame captured by the left infrared camera and the anchor frame of each floating object in each frame captured by the RGB camera. Thus, the pixel coordinates of the center point of the anchor frame of each floating object in each frame captured by the left infrared camera and the pixel coordinates of the center point of the anchor frame of each floating object in each frame captured by the RGB camera are obtained.

[0039] The BV-YOLOv network used in this embodiment is an improved YOLOv11 network. Specifically, the backbone structure of the BV-YOLOv network is a cross-modal fusion backbone network formed by embedding CFT modules in the backbone part of the YOLOv11 network. The Neck and Head parts of the BV-YOLOv network are the same as those of the YOLOv11 network.

[0040] like Figure 3 As shown, the backbone structure of the BV-YOLOv network takes infrared video images captured by the left infrared camera and RGB video images captured by the RGB camera as inputs for dual-branch feature extraction. Each frame of the input RGB image and each frame of the input infrared image are 640 pixels in size. 640 3 means 640 pixels long, 640 pixels wide, and 3 channels. Figure 3 The symbol ⊕ represents the Add module, which is used to add two tensors element by element to generate a new tensor. Figure 3 The Conv module in YOLO is a commonly used convolution module in YOLO 11. It uses standard convolution operations and extracts spatial features of the image at different levels by adjusting the size of the convolution kernel, stride, and padding.

[0041] The structural diagram of the CFT module in the backbone structure of the BV-YOLOv network is shown below. Figure 4 As shown, the core of the CFT module is to achieve feature fusion both intra-modal and inter-modal through the self-attention mechanism of the Transformer. The principle of the CFT module is as follows:

[0042] 1. Feature preprocessing:

[0043] The CFT module extracts the feature map of each frame of RGB image. Flattened as a feature map The feature map of each frame of infrared image , feature map and Furthermore, sine and cosine positional encoding is added to preserve the spatial location information of the features. Here, H, W, and C represent the image height, width, and number of channels, respectively.

[0044] 2. Self-attention mechanism:

[0045] The CFT module calculates attention weights by projecting and scaling the dot product of the query (Q), key (K), and value (V), forming the relevance matrix α. Matrix α is shown in the following formula:

[0046]

[0047] in: Indicates self-attention within the RGB mode; This indicates self-attention within the infrared mode; and These represent cross-modal attention, used to capture complementary information between RGB and infrared.

[0048] 3. Multi-head attention and residual connections:

[0049] The input to the multi-head attention module is the query matrix Q, the key matrix K, and the value matrix V. For h attention heads (h=8 here), Q, K, and V need to be decomposed into... Dimensions, then calculate using the following formula:

[0050]

[0051] in: Indicates the first Size;

[0052] Indicates the first The query vector of the head; Indicates the first The key vector of each head; Represents a value vector; , , All are two-dimensional tensors, and their operations can be represented by matrices.

[0053] The CFT module uses a multi-head attention mechanism to capture multi-subspace relationships. After the output is processed by a multilayer perceptron (MLP), it is connected back to the original modality branch through residual connections to avoid gradient vanishing and enhance feature representation.

[0054] In this embodiment, basic features are extracted from each frame of the input RGB image and each frame of the infrared image using the backbone structure of the BV-YOLOv network. Then, the basic features of each RGB image and each infrared image are fed into the Neck part of the BV-YOLOv network for fusion to obtain fused features. Next, based on the fused features, the Head part of the BV-YOLOv network predicts the anchor bounding box of each floating target in each RGB image and each infrared image, thereby achieving the detection of each floating target.

[0055] After detection by the BV-YOLOv network, each frame of the RGB image and each frame of the infrared image contains an anchor box [x1, y1, x2, y2] containing four numbers. Here, (x1, y1) represents the coordinates of the upper left corner of the anchor box of the floating target in the corresponding image, and (x2, y2) represents the coordinates of the lower right corner of the floating target in the corresponding image. Thus, the pixel coordinates of the center point of the anchor box of the floating target in each frame of the RGB image and each frame of the infrared image can be obtained.

[0056] In this embodiment, the BV-YOLOv network is trained using a dataset constructed from historical video images of the river channel to be measured. The historical video data includes images captured by a left infrared camera (left infrared image), a right infrared camera (right infrared image), and an RGB camera. During dataset acquisition, the video images captured by the left infrared camera and the RGB camera should overlap. When fusing the RGB and infrared images, the left infrared image is used for fusion.

[0057] The training process of the BV-YOLOv network begins with labeling the floating objects in the prepared dataset using labelmeChinese32, including the category and location information of the floating objects. Since the RGB images and the left infrared image overlap, labeling the RGB images also provides the category and location information of the floating objects in the left infrared image. After creating the training set required for training, the training set is divided, pairing RGB images at the same time point with the left infrared image.

[0058] The BV-YOLOv network training process uses AdamW as the optimizer, with an initial learning rate of 0.0015 and a final learning rate of 0.0001. The batch size is set to 16. The network input should be a pair of RGB and infrared images with a size of 640×640. Training is performed for 100 epochs with a patience level of 20. If the model does not show significant improvement within 20 training epochs, training is terminated.

[0059] After the BV-YOLOv network is trained, the optimal model weights are retained. The trained BV-YOLOv network is then used for detecting floating objects in the river channel to be measured.

[0060] Step 3: Using either the left infrared camera or the RGB camera as the reference camera, convert the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera into world coordinates to obtain the world coordinates of the center point of each floating target anchor frame in each frame of the reference camera.

[0061] In this embodiment, the depth map corresponding to the corresponding frame image is first obtained based on the corresponding frame image in the infrared video image captured by the left infrared camera and the right infrared camera in the binocular device, combined with the intrinsic and extrinsic parameters of the left infrared camera and the right infrared camera.

[0062] Obtaining the corresponding frame depth map requires the intrinsic and extrinsic parameters of the left and right infrared cameras. Obtaining these parameters necessitates calibration of the left and right infrared cameras. In this embodiment, the calibration of both the left and right infrared cameras employs the Zhang Zhengyou camera calibration method. This method is a practical approach to camera calibration using a planar checkerboard pattern. The calibration process is as follows:

[0063] In the preparation phase of the experiment, a calibration board composed of black and white squares (each square has an actual side length of 20mm) was prepared and placed on a flat surface. The calibration board was photographed from various angles using the left and right infrared cameras, resulting in images from the left and right infrared cameras (denoted as Image left and Image right). These images were then imported into the Stereo Camea Caliberato module in Matlab. Throughout the process, Matlab automatically detected the corner points of the checkerboard. The Stereo Camea Caliberato module used these detected corner points for camera calibration, calculating the intrinsic and extrinsic parameter matrices for the left and right infrared cameras.

[0064] In this embodiment, the corresponding frame depth map is obtained based on the triangulation principle of the binocular device. Specifically, the left and right infrared cameras simultaneously capture images of the same scene from different positions (with a certain baseline distance). Then, the disparity is calculated by matching corresponding feature points in the images from the left and right infrared cameras. Disparity refers to the lateral offset of the image position of the same object in the left and right camera images. When the binocular device captures the same object, the object's position in the two photos will differ due to the different camera perspectives. Figure 5As shown, by calculating this positional difference, a depth map can be generated, where depth refers to vertical distance.

[0065] Depth values ​​and parallax are inversely proportional. Generally, the larger the parallax, the smaller the corresponding depth value, meaning the object is closer to the camera; conversely, the smaller the parallax, the larger the depth value, meaning the object is farther away. Each pixel in the parallax map has a corresponding parallax value. In this embodiment, the depth map is generated according to the following process.

[0066] S1: Traverse the disparity map pixels: Starting from the top left corner of the disparity map, traverse each pixel row by row.

[0067] S2: From a bird's-eye view perspective, the bird's-eye view of the binocular device is as follows: Figure 6 As shown, for each pixel, its disparity value d= By combining the baseline distance and the camera focal length, the corresponding depth value Z is calculated. Figure 5 and Figure 6 The coordinates of a pixel in the camera coordinate system can be derived using the following formula:

[0068]

[0069] Where: T represents the baseline distance between the left and right infrared cameras; This represents the focal length of the camera; the left and right infrared cameras have the same focal length. The x-coordinate of a pixel in the image captured by the left infrared camera; The x-coordinate of a pixel in the image captured by the right infrared camera; This represents the x-coordinate of the pixel in the camera coordinate system. The vertical coordinate of the pixel in the camera coordinate system. The vertical coordinate of a pixel in the image captured by the left infrared camera; The vertical coordinate of a pixel in the image captured by the right infrared camera.

[0070] S3: Assign the calculated depth value to the corresponding pixel position in the depth map. After traversing all pixels, a complete depth map is generated. The color value of each pixel in the depth map corresponds to the depth information of the corresponding spatial position point.

[0071] The aforementioned depth map acquisition process does not rely on additional measuring instruments. It only requires the parallax information of the images collected by the left and right infrared cameras, combined with the internal and external parameters of the left and right infrared cameras, to achieve efficient positioning and detection.

[0072] After obtaining the depth map of the corresponding frame, a large amount of noise still exists in the depth map, requiring post-processing. Filtering is applied to the depth map image to eliminate noise and restore image integrity. In this embodiment, the post-processing order of the depth map is: depth frame → spatial filter → temporal filter → aperture fill filter.

[0073] In this embodiment, an RGB camera is used as a reference camera. The pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera are substituted into the depth map of the corresponding frame to obtain the depth value Z of the center point of each floating target anchor frame in each frame of the reference camera.

[0074] Then, based on the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera, the intrinsic parameters of the reference camera, and the depth value of the center point of each floating target anchor frame in each frame of the reference camera, the coordinates of the center point of each floating target anchor frame in each frame of the reference camera in the camera coordinate system are obtained. The calculation formula is as follows:

[0075]

[0076] Where: (u, v) are the pixel coordinates of the center point of the anchor frame; Z represents the depth value of the pixel coordinates (u, v), that is, the distance of the point in the real physical world corresponding to the pixel coordinates (u, v) of the anchor frame from the camera. , These are the coordinates of the optical center of the reference camera in the pixel coordinate system. The focal length of the reference camera. , , These are all intrinsic parameters of the reference camera, which can be obtained through the calibration of the reference camera; , , It is the coordinate of the center point of the anchor frame of each floating object target in each frame of the reference camera in the camera coordinate system.

[0077] In this embodiment, the coordinates of the center point of each floating target anchor frame in each frame of the reference camera are converted from the camera coordinate system to the world coordinate system, thus obtaining the world coordinates of the center point of each floating target anchor frame in each frame of the reference camera. The conversion is shown in the following formula:

[0078]

[0079] in: , , R is the world coordinate of the center point of the anchor frame of each floating target in each frame of the reference camera. R is the rotation matrix, which is a rotation transformation between coordinate systems. It does not change the shape or size of the point, only its orientation. T is the translation matrix, which is the translation relationship between coordinate systems. It can be used to move the origin of one coordinate system to the origin of another coordinate system.

[0080] Based on the above formula, the transformation relationship from the reference camera coordinate system to the world coordinate system is obtained as shown in the following formula:

[0081]

[0082] For ease of calculation, the origins of the world coordinate system and the camera coordinate system are set to the same point in this embodiment, thereby reducing the calculation of the translation matrix.

[0083] Step 4: Using the ByteTrack algorithm, track each floating target in each frame of the reference camera detected by the BV-YOLO network, thereby obtaining the motion trajectory of each floating target and the number of image frames contained in each motion trajectory.

[0084] The ByteTrack algorithm performs object detection in each frame of an image, extracting the object's bounding box, confidence score, and appearance feature vector. Then, it calculates the IoU value of the bounding box and initially matches the detection result of the current frame with historical trajectories: if the IoU value exceeds a set threshold, a match is considered successful. For detection boxes that do not match based on IoU value, the cosine distance between the bounding box and the historical trajectory is further calculated based on the appearance feature vector to complete the embedding space matching, thereby enhancing the ability to discriminate occluded or overlapping scenes.

[0085] In multi-target tracking, Kalman filters and the Hungarian algorithm are used to predict floating targets and solve the many-to-many matching problem. The Kalman filter predicts target positions, enabling the system to continue tracking targets even when some targets temporarily disappear due to occlusion or missed detection. It combines historical data (such as target velocity and position information) to predict the target's position in future frames, compensating for information gaps caused by target loss or occlusion. In handling floating objects, the Kalman filter effectively compensates for situations where targets are temporarily undetected, allowing the target's trajectory to continue more smoothly and avoiding the loss of the entire trajectory due to a target not being detected in a single frame. The Hungarian algorithm solves the many-to-many matching problem. In target tracking, there may be overlaps or matching difficulties between multiple detection boxes and multiple historical trajectories. The Hungarian algorithm optimizes calculations to find the optimal matching scheme, ensuring that each detection box matches the most suitable trajectory, thus avoiding mismatches or inaccurate matching. In floating object scenarios, when a target temporarily disappears or a new target appears, the Hungarian algorithm can flexibly adjust the matching to ensure the continuity and stability of the target.

[0086] In this embodiment, the coordinates of the center point of the anchor frame of the corresponding floating object in the frame image corresponding to the starting point of the movement trajectory of each floating object are used as the world coordinates of the starting point of each floating object; and the coordinates of the center point of the anchor frame of the corresponding floating object in the frame image corresponding to the ending point of the movement trajectory of each floating object are used as the world coordinates of the ending point of each floating object.

[0087] Then, based on the starting point coordinates, ending point coordinates, number of image frames included in the corresponding motion trajectory, and video frame rate of the reference camera for each floating target, the velocity estimate of each floating target is calculated. As shown in the formula below:

[0088]

[0089] in: The starting point coordinates for each floating object; The coordinates of the endpoint for each floating target; T is the number of image frames contained in the trajectory of each floating target; F is the video frame rate of the reference camera.

[0090] Finally, based on the velocity estimates of each floating object, the water flow velocity on the surface of the river channel under test is calculated. As shown in the formula below:

[0091]

[0092] Where: n is the number of floating targets.

[0093] In this embodiment, to reduce the error in flow velocity measurement, the calculated flow velocity of the river channel to be measured is divided into multiple intervals. A linear least squares estimation model is used to correct the calculated surface flow velocity of the river channel based on each interval. It is important to note that the flow velocity here refers to the flow velocity at different times; these velocities are not necessarily the same. Therefore, different intervals can be defined, and the linear least squares method is used for estimation within each interval. Specifically, the linear least squares method is shown in the following formula:

[0094]

[0095] in: This represents the corrected flow velocity value; w and b are the parameters of the least squares estimation model; x is the original predicted flow velocity value, i.e., x = .

[0096] In this embodiment, since the flow velocity obtained in step 4 is not the flow velocity at the same time, but the flow velocity at different times, different intervals are divided according to the water flow velocity, and parameters for each interval are selected. and Data is fitted separately within each interval to make the flow velocity measurement more accurate. In this embodiment, the obtained water flow velocity is divided into several small intervals ( ),( ),…,( The formula for the least squares method for velocity intervals is as follows:

[0097]

[0098] in: The indicator function is shown in the following formula:

[0099]

[0100] in: The original flow velocity prediction value; This is the lower bound when the indicator function is 1; This indicates the upper limit when the indicator function is 1.

[0101] The flow velocity interval least squares method is used to estimate different intervals of the model. and The parameters are used to minimize the objective function MSE (mean squared error).

[0102] , ,…, Let x be the predicted flow velocity and y be the actual flow velocity. The parameters for each interval correction are shown in the following formula:

[0103]

[0104]

[0105] The objective function MSE (mean squared error) is shown in the following formula:

[0106]

[0107] Obtain the parameters for each interval. and Then, for the predicted flow velocity x in different intervals, select the appropriate... and , bring in In this process, the predicted flow velocity is corrected.

[0108] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0109] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A method for detecting surface water flow velocity in a river channel based on trioptic multimodal vision, characterized in that, The process is as follows: Step 1: Acquire video images of the river channel to be measured using a tri-lens camera. The river channel contains several floating objects. The tri-lens camera includes a left infrared camera, a right infrared camera, and an RGB camera. The left infrared camera, the right infrared camera, and the RGB camera simultaneously acquire video images of the river channel to be measured. The left infrared camera, the right infrared camera, and the RGB camera acquire video images of the river channel to be measured at the same frame rate and frame rate. The left infrared camera and the right infrared camera each acquire infrared video images of the river channel to be measured, and the RGB camera acquires RGB video images of the river channel to be measured. Step 2: Perform multimodal detection of floating objects based on the infrared video images acquired by the left infrared camera and the RGB video images acquired by the RGB camera. As a result, the anchor frame and the pixel coordinates of the anchor frame center point of each floating object are obtained in each frame of the image acquired by the left infrared camera, and the anchor frame and the pixel coordinates of the anchor frame center point of each floating object are obtained in each frame of the image acquired by the RGB camera. Step 3: Using the RGB camera as the reference camera, convert the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera into coordinates in the world coordinate system to obtain the coordinates of the center point of each floating target anchor frame in the world coordinate system in each frame of the reference camera. Step 4: Use a multi-target tracking algorithm to track floating targets in each frame of images captured by the reference camera, thereby obtaining the motion trajectory of each floating target and the number of image frames contained in each motion trajectory; The coordinates of the center point of the anchor frame of the corresponding floating object in the frame corresponding to the starting point of the trajectory of each floating object are taken as the world coordinates of the starting point of each floating object; the coordinates of the center point of the anchor frame of the corresponding floating object in the frame corresponding to the ending point of the trajectory of each floating object are taken as the world coordinates of the ending point of each floating object. Then, based on the starting point coordinates, ending point coordinates, number of image frames contained in the corresponding motion trajectory, and video frame rate of the reference camera, the velocity estimate of each floating object is calculated. Finally, based on the velocity estimates of each floating object, the water flow velocity on the surface of the river channel under test is calculated.

2. The method for detecting river surface water flow velocity based on trioptic multimodal vision according to claim 1, characterized in that, In step 2, the BV-YOLOv network is used to perform multimodal detection of floating objects on the infrared video images captured by the left infrared camera and the RGB video images captured by the RGB camera.

3. The method for detecting surface water flow velocity in a river channel based on trioptic multimodal vision according to claim 2, characterized in that, The backbone structure of the BV-YOLOv network is a cross-modal fusion backbone network formed by adding CFT modules to the backbone part of the YOLOv11 network. The Neck and Head structures of the BV-YOLOv network are the same as those of the Neck and Head parts of the YOLOv11 network.

4. The method for detecting river surface water flow velocity based on trioptic multimodal vision according to claim 1, characterized in that, In step 3, based on the corresponding frame images in the infrared video images captured by the left and right infrared cameras, and combined with the intrinsic and extrinsic parameters of the left and right infrared cameras, the depth map corresponding to the corresponding frame image is obtained. Using an RGB camera as a reference camera, the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera are substituted into the depth map of the corresponding frame to obtain the depth value of the center point of each floating target anchor frame in each frame of the reference camera. Then, based on the pixel coordinates of the center point of each floating target anchor frame in each frame of the reference camera, the intrinsic parameters of the reference camera, and the depth value of the center point of each floating target anchor frame in each frame of the reference camera, the coordinates of the center point of each floating target anchor frame in the reference camera coordinate system are obtained. Finally, the coordinates of the center point of each floating target anchor frame in each frame of the reference camera in the reference camera coordinate system are converted into coordinates in the world coordinate system to obtain the coordinates of the center point of each floating target anchor frame in the world coordinate system in each frame of the reference camera.

5. The method for detecting surface water flow velocity in a river channel based on trioptic multimodal vision according to claim 1, characterized in that, In step 4, the ByteTrack algorithm is used to track floating targets in each frame of images captured by the reference camera.

6. The method for detecting river surface water flow velocity based on trioptic multimodal vision according to claim 1, characterized in that, In step 4, the mean of the velocity estimates of each floating object is calculated as the flow velocity of the river channel to be measured.

7. The method for detecting river surface water flow velocity based on trioptic multimodal vision according to claim 1, characterized in that, In step 4, the calculated flow velocity of the river channel to be measured is divided into multiple intervals, and the least squares estimation model is used to correct the calculated water flow velocity on the surface of the river channel to be measured based on each interval.

Images