Water surface flow velocity detection method, device, equipment and medium

Abstract

The invention relates to the technical field of image recognition, and discloses a water surface flow velocity detection method, device and equipment and a medium, and the method comprises the steps: obtaining multi-frame to-be-detected image data; performing feature extraction, feature fusion and prediction on the image data of the current frame by using the target detection network to obtain a first target detection frame corresponding to the image data of the current frame, and performing feature extraction, feature fusion and prediction on the image data of the next frame to obtain a second target detection frame corresponding to the image data of the next frame; using a target tracking algorithm to update the historical trajectories of all the water surface tracers under the current frame based on the first target detection frame and the second target detection frame to obtain the target trajectories of all the water surface tracers under the current frame; and obtaining the water surface flow velocity of the image data of the current frame based on the target trajectory. According to the invention, the attention mechanism is fused in the target detection network, and the target tracking algorithm is used to track the water surface tracer, so that the water surface flow velocity detection precision is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and particularly to a method, device, equipment and medium for detecting water surface flow velocity. Background Art

[0002] Currently, in the field of water surface flow velocity detection, it mainly relies on hardware devices such as current meters to determine the water surface flow velocity by comparing the frequency difference between the transmitted and reflected signals. This velocity measurement method requires a lot of manpower and maintenance costs, and is greatly affected by environmental and equipment parameters in terms of measurement accuracy.

[0003] The prior art usually places water surface tracer particles in the water flow and tracks the positions of these water surface tracer particles at different times, and calculates the water surface flow velocity based on the displacement and time changes. Since there are errors when a single type of water surface tracer particle is affected by interference factors such as light changes, water surface fluctuations, and background noise, multiple types of water surface tracer particles need to be used simultaneously for calculating the water surface flow velocity.

[0004] In recent years, with the rapid development of deep learning, the image recognition method based on convolutional neural network has been widely used and shown great advantages in various recognition fields such as object detection and semantic segmentation. However, when using a convolutional neural network to detect water surface tracer particles, due to the influence of environmental factors such as light, shadow, and complex background, the detection accuracy of the model for complex scenarios often fluctuates. In addition, although a convolutional neural network can detect the types and positions of water surface tracer particles, it is difficult to distinguish water surface tracer particles of the same type, resulting in low detection accuracy. Summary of the Invention

[0005] In view of this, the present disclosure provides a method, device, equipment and medium for detecting water surface flow velocity to solve the problem of low detection accuracy of water surface flow velocity.

[0006] In a first aspect, the present disclosure provides a method for detecting water surface flow velocity, the method comprising:

[0007] Obtain multiple frames of image data to be detected, the image data including water surface tracers for detecting water surface flow velocity;

[0008] Use a target detection network to perform feature extraction, feature fusion and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame, and use the target detection network to perform feature extraction, feature fusion and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame, wherein the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer;

[0009] Using a target tracking algorithm, based on a first target detection box and a second target detection box, update the historical trajectories of all water surface tracers in the current frame to obtain the target trajectories of all water surface tracers in the current frame;

[0010] Obtain the water surface flow velocity of the image data of the current frame based on the target trajectories.

[0011] In an embodiment of the present disclosure, by acquiring multiple frames of image data to be detected, the image data includes water surface tracers for water surface flow velocity detection; using a target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame, and using the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame, wherein the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer; using a target tracking algorithm, based on the first target detection box and the second target detection box, update the historical trajectories of all water surface tracers in the current frame to obtain the target trajectories of all water surface tracers in the current frame; obtain the water surface flow velocity of the image data of the current frame based on the target trajectories. Since in the embodiment of the present disclosure, the attention mechanism is fused in the target detection network and the target tracking algorithm is used to track the water surface tracers, the accuracy of water surface flow velocity detection is improved.

[0012] In an alternative embodiment, the feature extraction layer includes multiple convolutional layers. Using the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame, and using the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame, includes:

[0013] Based on the feature extraction layer, perform feature extraction on the image data of the current frame to obtain a first feature, and perform feature extraction on the image data of the next frame to obtain a second feature;

[0014] Based on the convolutional layer, perform a convolutional operation on the image data of the current frame to obtain a third feature, and select a first network feature from the preset layer from the third feature. Perform a convolutional operation on the image data of the next frame to obtain a fourth feature, and select a second network feature from the preset layer from the fourth feature;

[0015] Based on the attention mechanism, weight the first feature to obtain a weighted first feature, and weight the second feature to obtain a weighted second feature;

[0016] Based on the feature fusion layer, perform feature fusion on the first network feature and the weighted first feature to obtain the first fused feature, and perform feature fusion on the second network feature and the weighted second feature to obtain the second fused feature;

[0017] Based on the prediction layer, detect the input first fused feature and output the first target detection box corresponding to the image data of the current frame, and detect the input second fused feature and output the second target detection box corresponding to the image data of the next frame.

[0018] In the embodiments of the present disclosure, by using the feature extraction layer, attention mechanism, feature fusion layer, and prediction layer in the object detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame and the image data of the next frame, it is possible to accurately capture the features of the water surface tracer in the image data at different scales, obtain accurate target detection boxes, and improve the detection accuracy of the water surface flow velocity.

[0019] In an alternative embodiment, based on the attention mechanism, weight the first feature to obtain the weighted first feature, and weight the second feature to obtain the weighted second feature, including:

[0020] Based on the average pooling layer, perform a pooling operation on the first feature to obtain the first tensor, and perform a pooling operation on the second feature to obtain the second tensor;

[0021] Based on the one-dimensional convolutional layer, perform one-dimensional convolution on the first tensor to obtain the third tensor, and perform one-dimensional convolution on the second tensor to obtain the fourth tensor;

[0022] Use the activation function to transform the third tensor into a preset interval to obtain the first weight, and transform the fourth tensor into a preset interval to obtain the second weight;

[0023] Fuse the first feature and the first weight to obtain the weighted first feature, and fuse the second feature and the second weight to obtain the weighted second feature.

[0024] In the embodiments of the present disclosure, by weighting the first feature and the second feature based on the attention mechanism, it is possible to make full use of the channel information of the first feature and the second feature, enhance the fitting ability of the object detection network, and improve the detection accuracy of the water surface flow velocity.

[0025] In an alternative embodiment, use the object tracking algorithm to update the historical trajectories of all water surface tracers in the current frame based on the first target detection box and the second target detection box to obtain the target trajectories of all water surface tracers in the current frame, including:

[0026] Obtain the state information of the first target detection box;

[0027] Predict the trajectories of all water surface tracers in the next frame based on the status information to obtain the tracking frames of all water surface tracers in the next frame under the current frame;

[0028] Match the second target detection frame with the tracking frame of the next frame to obtain a matching result;

[0029] Update the historical trajectories of all water surface tracers in the current frame based on the matching result to obtain the target trajectories.

[0030] In the embodiments of the present disclosure, by using the target tracking algorithm to update the historical trajectories of all water surface tracers in the current frame, it is possible to continuously and stably track all water surface tracers, obtain accurate target trajectories, and improve the detection accuracy of the water surface flow velocity.

[0031] In an alternative embodiment, matching the second target detection frame with the tracking frame of the next frame to obtain a matching result includes:

[0032] Based on the historical unassociated frame numbers of all water surface tracers in the current frame, determine the status of the tracking frame of the next frame to obtain the first tracking frame with a determined status and the second tracking frame with an undetermined status;

[0033] Perform a first match between the first tracking frame and the second target detection frame to obtain a first matching result, where the first matching result includes a third tracking frame and a fourth tracking frame. The third tracking frame is the tracking frame that does not match the second target detection frame, and the fourth tracking frame is the tracking frame that does not match the second target detection frame in the case of detecting a new water surface tracer;

[0034] Perform a second match between the second tracking frame, the third tracking frame, and the fourth tracking frame to obtain a second matching result.

[0035] In the embodiments of the present disclosure, by performing the first match and the second match between the second target detection frame and the tracking frame of the next frame, the accuracy and stability of tracking the water surface tracers can be improved, and the detection accuracy of the water surface flow velocity can be improved.

[0036] In an alternative embodiment, the first matching result further includes a fifth tracking frame, where the fifth tracking frame is the tracking frame that successfully matches the second target detection frame. The second matching result includes the third tracking frame, the fourth tracking frame, and the fifth tracking frame. Updating the historical trajectories of all water surface tracers in the current frame based on the matching result to obtain the target trajectories includes:

[0037] Use the fifth tracking frame to update the historical trajectories of all water surface tracers in the corresponding current frame to obtain the first target trajectories;

[0038] Determine the state of the third tracking box based on the historical unassociated frame numbers of all water surface tracers in the current frame corresponding to the third tracking box, and obtain the first tracking box and the second tracking box;

[0039] Compare the historical unassociated frame number of the first tracking box with a preset frame number to obtain a sixth tracking box and a seventh tracking box, where the sixth tracking box is the tracking box with a historical unassociated frame number greater than the preset frame number, and the seventh tracking box is the tracking box with a historical unassociated frame number less than the preset frame number;

[0040] Create new trajectories of all water surface tracers in the corresponding current frame using the fourth tracking box and the seventh tracking box in the second matching result to obtain a second target trajectory;

[0041] Delete the historical trajectories of all water surface tracers in the current frame corresponding to the second tracking box and the sixth tracking box;

[0042] Obtain a target trajectory based on the first target trajectory and the second target trajectory.

[0043] In the embodiments of the present disclosure, by updating the historical trajectories of all water surface tracers in the current frame based on the first matching result and the second matching result, it is possible to continuously and stably track all water surface tracers, obtain accurate target trajectories, and improve the detection accuracy of water surface flow velocity.

[0044] In an alternative embodiment, obtaining the water surface flow velocity of the image data of the current frame based on the target trajectory includes:

[0045] Based on the target trajectory of each water surface tracer in the current frame, obtain the displacement information of each water surface tracer in the current frame;

[0046] Based on the displacement information and the time interval between the current frame and the next frame, obtain the flow velocity of each water surface tracer in the current frame;

[0047] Fuse the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame.

[0048] In the embodiments of the present disclosure, by obtaining the flow velocity of each water surface tracer in the current frame based on the target trajectory of each water surface tracer in the current frame and fusing the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame, it is possible to reduce the error of the water surface flow velocity, ensure the reliability of the water surface flow velocity, and improve the detection accuracy of the water surface flow velocity.

[0049] In a second aspect, the present disclosure provides a water surface flow velocity detection device, and the device includes:

[0050] An acquisition module, configured to acquire multi-frame image data to be detected, where the image data includes water surface tracers for water surface flow velocity detection;

[0051] The first obtaining module is configured to perform feature extraction, feature fusion, and prediction on the image data of the current frame by using a target detection network, so as to obtain a first target detection box corresponding to the image data of the current frame, and perform feature extraction, feature fusion, and prediction on the image data of the next frame by using the target detection network, so as to obtain a second target detection box corresponding to the image data of the next frame. The target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer;

[0052] The second obtaining module is configured to update the historical trajectories of all water surface tracers in the current frame by using a target tracking algorithm based on the first target detection box and the second target detection box, so as to obtain the target trajectories of all water surface tracers in the current frame;

[0053] The third obtaining module is configured to obtain the water surface flow velocity of the image data of the current frame based on the target trajectories.

[0054] In a third aspect, the present disclosure provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the water surface flow velocity detection method according to the first aspect or any corresponding embodiment thereof.

[0055] In a fourth aspect, the present disclosure provides a computer-readable storage medium, on which computer instructions are stored. The computer instructions are used to cause a computer to execute the water surface flow velocity detection method according to the first aspect or any corresponding embodiment thereof.

[0056] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to cause a computer to execute the water surface flow velocity detection method according to the first aspect or any corresponding embodiment thereof. Description of the Drawings

[0057] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0058] Figure 1 It is a flowchart of the water surface flow velocity detection method according to an embodiment of the present disclosure;

[0059] Figure 2 It is a structural diagram of the target detection network according to an embodiment of the present disclosure;

[0060] Figure 3 is a schematic structural diagram of an attention mechanism according to an embodiment of the present disclosure;

[0061] Figure 4 is a schematic flowchart of an object tracking algorithm according to an embodiment of the present disclosure;

[0062] Figure 5 is a schematic flowchart of another method for detecting water surface flow velocity according to an embodiment of the present disclosure;

[0063] Figure 6 is a schematic flowchart of yet another method for detecting water surface flow velocity according to an embodiment of the present disclosure;

[0064] Figure 7 is a block diagram of the structure of a water surface flow velocity detection device according to an embodiment of the present disclosure;

[0065] Figure 8 is a schematic hardware structure diagram of a computer device according to an embodiment of the present disclosure. Detailed implementation manners

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0067] Currently, in the field of water surface flow velocity detection, it mainly relies on hardware devices such as current meters, and determines the water surface flow velocity by comparing the frequency differences between the transmitted and reflected signals. This method of measuring velocity requires a lot of manpower and maintenance costs, and is greatly affected by the environment and equipment parameters in terms of measurement accuracy.

[0068] The prior art usually puts water surface tracer particles into the water flow, and tracks the positions of these water surface tracer particles at different times, and calculates the water surface flow velocity based on the displacement and time changes. Since there are errors when a single type of water surface tracer particle is affected by interference factors such as light changes, water surface fluctuations, and background noise, multiple types of water surface tracer particles need to be used simultaneously for calculating the water surface flow velocity.

[0069] In recent years, with the rapid development of deep learning, image recognition methods based on convolutional neural networks have been widely used and shown great advantages in various recognition fields such as object detection and semantic segmentation. However, when using convolutional neural networks to detect water surface tracer particles, due to the influence of environmental factors such as illumination, shadow, and complex background, the detection accuracy of the model for complex scenarios often fluctuates. In addition, although convolutional neural networks can detect the category and location of water surface tracer particles, it is difficult to distinguish water surface tracer particles of the same category, resulting in low detection accuracy.

[0070] To solve the above problems, according to an embodiment of the present disclosure, an embodiment of a water surface flow velocity detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0071] In this embodiment, a water surface flow velocity detection method is provided, as Figure 1 shown, Figure 1 is a flowchart of the water surface flow velocity detection method according to an embodiment of the present disclosure. This process can be applied to a server and includes the following steps:

[0072] Step S101, obtain multiple frames of image data to be detected, where the image data includes water surface tracers for water surface flow velocity detection.

[0073] Optionally, in an embodiment of the present disclosure, the server can obtain the video data to be detected by means of web crawlers or on-site shooting. After obtaining the video data, the server performs frame splitting on the video data to obtain multiple frames of image data to be detected.

[0074] Specifically, the water surface tracer is an object used to mark and track the water flow movement during the water surface flow velocity detection process. Since the water surface tracer has good floatability and visibility and can move naturally with the water flow, the server can intuitively understand the movement trajectory of the water flow by recording and analyzing the water surface tracer at different times and positions. The water surface tracer can be artificially made tracers such as plastic balls, foam products, fluorescent markers, etc., or tracers of natural materials such as leaves and wooden chips.

[0075] In addition, the server can also preprocess the image data. For example: the server first scales the image data to the same size, then performs a normalization operation on the image data, calculates the mean and standard deviation of the pixel values of each channel of the image data, and then subtracts the mean from the normalized image data and divides by the standard deviation to obtain the preprocessed image data.

[0076] Step S102: Use the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain the first target detection box corresponding to the image data of the current frame. Use the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain the second target detection box corresponding to the image data of the next frame. Among them, the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer.

[0077] Optionally, in the embodiments of the present disclosure, as Figure 2 shown, the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer. The first target detection box refers to the target detection box corresponding to the image data of the current frame, and the second target detection box refers to the target detection box corresponding to the image data of the next frame.

[0078] Specifically, the server first inputs the image data of the current frame and the image data of the next frame into the target detection network respectively. Use the convolutional kernels of the feature extraction layer to slide on the image data of the current frame and perform convolutional calculations with the pixels of the image data of the current frame. Use the convolutional kernels of the feature extraction layer to slide on the image data of the next frame and perform convolutional calculations with the pixels of the image data of the next frame.

[0079] Then, as Figure 3 shown, the server uses the attention mechanism to weight the features of different channels in the features of the image data of the current frame output by the feature extraction layer, and weight the features of different channels in the features of the image data of the next frame output by the feature extraction layer.

[0080] Next, the server uses the feature fusion layer to perform feature fusion on the multiple features output during the feature extraction process of the image data of the current frame and the weighted features, and perform feature fusion on the multiple features output during the feature extraction process of the image data of the next frame and the weighted features.

[0081] After that, the server uses the prediction layer to predict the fusion features corresponding to the image data of the current frame to obtain the first target detection box, and uses the prediction layer to predict the fusion features corresponding to the image data of the next frame to obtain the second target detection box.

[0082] Step S103: Use the target tracking algorithm to update the historical trajectories of all water surface tracers in the current frame based on the first target detection box and the second target detection box to obtain the target trajectories of all water surface tracers in the current frame.

[0083] Optionally, in the embodiments of the present disclosure, the historical trajectory refers to the set of all trajectories of each water surface tracer before the current frame of image data, and the target trajectory refers to the set of all trajectories of each water surface tracer at the current frame of image data and before the current frame of image data.

[0084] Specifically, as Figure 4 shown, the server first uses the Kalman filter to predict the state of the next frame based on the state information of the first target detection box, and obtains the tracking box of the next frame. Then, the server uses the Hungarian algorithm to match the tracking box of the next frame with the second target detection box to obtain the matching result. After that, the server uses the Kalman filter to update the historical trajectories of all water surface tracers in the current frame according to the matching result, and obtains the target trajectories of all water surface tracers in the current frame.

[0085] Step S104, obtaining the water surface flow velocity of the image data of the current frame based on the target trajectory.

[0086] Optionally, in the embodiments of the present disclosure, the server first calculates the displacement of each water surface tracer from the current frame to the next frame according to the coordinate information of the first target detection box and the coordinate information of the second target detection box in the target trajectory of each water surface tracer in the current frame, then calculates the time elapsed from the current frame to the next frame according to the frame rate of the video data to be detected, then calculates the flow velocity of each water surface tracer according to the displacement and time of each water surface tracer from the current frame to the next frame, and then performs a fusion calculation (such as average calculation, median calculation, mode calculation, weighted calculation, etc.) on the flow velocities of each water surface tracer to obtain the water surface flow velocity of the image data of the current frame.

[0087] In the embodiments of the present disclosure, by acquiring multiple frames of image data to be detected, the image data includes water surface tracers for water surface flow velocity detection; using a target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain the first target detection box corresponding to the image data of the current frame, using the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain the second target detection box corresponding to the image data of the next frame, where the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer; using a target tracking algorithm based on the first target detection box and the second target detection box to update the historical trajectories of all water surface tracers in the current frame to obtain the target trajectories of all water surface tracers in the current frame; obtaining the water surface flow velocity of the image data of the current frame based on the target trajectory. Since the embodiments of the present disclosure fuse the attention mechanism in the target detection network and use the target tracking algorithm to track the water surface tracers, the accuracy of water surface flow velocity detection is improved.

[0088] In some alternative embodiments, the present embodiment provides a method for detecting water surface flow velocity, as Figure 5 shown, Figure 5 FIG. is a schematic flowchart of another method for detecting water surface flow velocity according to an embodiment of the present disclosure. This process can be applied to a server and includes the following steps:

[0089] Step S501: Obtain multiple frames of image data to be detected. The image data includes water surface tracers for water surface flow velocity detection. For details, please refer to Figure 1 step S101 of the embodiment shown, which will not be elaborated here.

[0090] Step S502: Use a target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame. Use the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame. Among them, the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer. The attention mechanism is located between the feature extraction layer and the feature fusion layer.

[0091] Specifically, the above step S502 includes:

[0092] Step S5021: Based on the feature extraction layer, perform feature extraction on the image data of the current frame to obtain a first feature, and perform feature extraction on the image data of the next frame to obtain a second feature.

[0093] Optionally, in the embodiment of the present disclosure, the first feature refers to the feature obtained by passing the image data of the current frame through the feature extraction layer, and the second feature refers to the feature obtained by passing the image data of the next frame through the feature extraction layer. As Figure 2 shown, the feature extraction layer includes a convolution module, a residual module, and a pooling module. Among them, the convolution module includes a convolution layer, a normalization layer, and an activation function layer. The residual module includes a convolution layer, a residual network, and a channel splicing operator. The pooling module includes a convolution layer and a pooling layer.

[0094] Specifically, the server first inputs the image data of the current frame and the image data of the next frame into the convolutional module respectively. By using convolutional kernels of different sizes and numbers in the convolutional layer, it slides on the image data of the current frame and the image data of the next frame respectively to perform convolutional operations, extracts the local features of the image data of the current frame and the local features of the image data of the next frame. Then, it uses the normalization layer to perform normalization operations on the local features of the image data of the current frame and the local features of the image data of the next frame output by the convolutional layer in the channel dimension, making the feature mean of each channel 0 and the variance 1. And it uses the activation function layer to perform non-linear transformations on the normalized features of the image data of the current frame and the normalized features of the image data of the next frame, setting the feature values less than 0 to 0 and keeping the feature values greater than 0 unchanged.

[0095] Then, the server inputs the features of the image data of the current frame and the features of the image data of the next frame output by the convolutional module into the residual module respectively. It uses the convolutional layer to perform convolutional operations on the features of the image data of the current frame and the features of the image data of the next frame again to extract deeper features. At the same time, it uses the residual network to perform feature extraction and residual connection on the features of the image data of the current frame and the image data of the next frame output by the convolutional layer respectively. Then, it uses the channel splicing operator to splice the features of the image data of the current frame output by the convolutional layer and the features of the image data of the current frame output by the residual network, and splices the features of the image data of the next frame output by the convolutional layer and the features of the image data of the next frame output by the residual network.

[0096] After that, the server inputs the features of the image data of the current frame and the features of the image data of the next frame output by the residual module into the pooling module respectively. It uses the convolutional layer to perform convolutional operations on the features of the image data of the current frame and the features of the image data of the next frame further to extract deeper features. Then, it uses the pooling layer to perform downsampling on the features of the image data of the current frame and the features of the image data of the next frame output by the convolutional layer respectively, obtaining the first feature corresponding to the image data of the current frame and the second feature corresponding to the image data of the next frame.

[0097] It should be noted that, as Figure 2 shown, the server uses multiple convolutional modules, multiple residual modules and pooling modules in the feature extraction layer to perform feature extraction on the input image data, obtaining multi-scale feature maps (such as feature maps with 2x, 4x, 8x downsampling).

[0098] Step S5022: Perform a convolutional operation on the image data of the current frame based on the convolutional layer to obtain a third feature, and select a first network feature from the preset layer from the third feature. Perform a convolutional operation on the image data of the next frame to obtain a fourth feature, and select a second network feature from the preset layer from the fourth feature.

[0099] Optionally, in the embodiments of the present disclosure, the third feature refers to the feature output during the feature extraction of the image data of the current frame in the feature extraction layer, the fourth feature refers to the feature output during the feature extraction of the image data of the next frame in the feature extraction layer, the first network feature refers to the feature from the preset layer in the third feature, and the second network feature refers to the feature from the preset layer in the fourth feature. The preset layer is included in the feature extraction layer. As Figure 2 shown, the preset layer refers to Figure 2 modules 4 and 6 in

[0100] Specifically, the server uses the convolutional layer in the feature extraction layer to perform a convolution operation on the image data of the current frame to obtain the third feature, and selects the first network feature from the preset layer ( Figure 2 modules 4 and 6 in Figure 2 modules 4 and 6) from the third feature. The server uses the convolutional layer in the feature extraction layer to perform a convolution operation on the image data of the next frame to obtain the fourth feature, and selects the second network feature from the preset layer (

[0101] Step S5023: Weight the first feature based on the attention mechanism to obtain the weighted first feature, and weight the second feature to obtain the weighted second feature.

[0102] Optionally, in the embodiments of the present disclosure, as Figure 3 shown, the server uses the attention mechanism to weight the first feature and the second feature respectively, adjusts the channel attention of the first feature and the channel attention of the second feature, and obtains the weighted first feature and the weighted second feature.

[0103] In some alternative embodiments, the above step S5023 includes:

[0104] Step a1: Perform a pooling operation on the first feature based on the average pooling layer to obtain the first tensor, and perform a pooling operation on the second feature to obtain the second tensor.

[0105] Step a2: Perform a one-dimensional convolution on the first tensor based on the one-dimensional convolutional layer to obtain the third tensor, and perform a one-dimensional convolution on the second tensor to obtain the fourth tensor.

[0106] Step a3: Use the activation function to transform the third tensor into a preset interval to obtain the first weight, and transform the fourth tensor into a preset interval to obtain the second weight.

[0107] Step a4: Fuse the first feature and the first weight to obtain the weighted first feature, and fuse the second feature and the second weight to obtain the weighted second feature.

[0108] Optionally, in the embodiments of the present disclosure, the first tensor refers to the tensor after pooling the first feature, the second tensor refers to the tensor after pooling the second feature, the third tensor refers to the tensor after performing one-dimensional convolution on the first tensor, the fourth tensor refers to the tensor after performing one-dimensional convolution on the second tensor, the first weight refers to the weight of the first feature, and the second weight refers to the weight of the second feature. The preset interval refers to the value range of the first weight and the second weight, such as the interval (0, 1).

[0109] Specifically, the server first uses the average pooling layer to perform a pooling operation on the first feature, adjusts the size of the first feature from W×H×C to 1×1×C to obtain the first tensor, and performs a pooling operation on the second feature, adjusts the size of the second feature from W×H×C to 1×1×C to obtain the second tensor. Wherein, W represents the width of the feature map, H represents the height of the feature map, and C represents the number of channels of the feature map.

[0110] Then, the server uses the one-dimensional convolution layer to perform one-dimensional convolution on the first tensor to obtain the third tensor, and performs one-dimensional convolution on the second tensor to obtain the fourth tensor.

[0111] Next, the server uses an activation function (such as the sigmoid activation function) to transform the third tensor into the interval (0, 1) to obtain the first weight, and transforms the fourth tensor into the interval (0, 1) to obtain the second weight.

[0112] Finally, the server multiplies the first feature and the first weight to obtain the weighted first feature, and multiplies the second feature and the second weight to obtain the weighted second feature.

[0113] In the above embodiments, by weighting the first feature and the second feature based on the attention mechanism, the channel information of the first feature and the second feature can be fully utilized, the fitting ability of the target detection network can be enhanced, and the detection accuracy of the water surface flow velocity can be improved.

[0114] Step S5024, based on the feature fusion layer, perform feature fusion on the first network feature and the weighted first feature to obtain the first fusion feature, and perform feature fusion on the second network feature and the weighted second feature to obtain the second fusion feature.

[0115] Optionally, in the embodiments of the present disclosure, the first fusion feature refers to the fusion feature corresponding to the image data of the current frame, and the second fusion feature refers to the fusion feature corresponding to the image data of the next frame. As Figure 2 shown, the feature fusion layer includes a feature pyramid network (i.e., Figure 2 modules 11-module 17 in) and a path aggregation network (i.e., Figure 2Modules 18 - Module 24). Among them, the Feature Pyramid Network includes a convolutional module, an upsampling module, a splicing module, and a residual module, and the Path Aggregation Network includes a residual module, a convolutional module, and a splicing module.

[0116] Specifically, the server uses the Feature Pyramid Network in the feature fusion layer to perform convolution operations, upsampling operations, residual connections, and splicing operations on the first network feature (i.e., Figure 2 the features output by Modules 4 and 6 in Figure 2 and the weighted first feature (i.e., Figure 2 the features output by Module 10 in Figure 2 Then, the server uses the Path Aggregation Network in the feature fusion layer to perform residual connections, convolution operations, and splicing operations on the features output by Modules 11, 14, and 17 in

[0117] to obtain the first fusion feature (i.e., Figure 2 the features output by Modules 18, 21, and 24 in Figure 2 the features output by Modules 4 and 6 in Figure 2 and the weighted second feature (i.e., Figure 2 the features output by Module 10 in

[0118] Step S5025: Based on the prediction layer, detect the input first fusion feature, output the first target detection box corresponding to the image data of the current frame, detect the input second fusion feature, and output the second target detection box corresponding to the image data of the next frame.

[0119] Optionally, in the embodiments of the present disclosure, the server first performs classification calculations on the first fusion feature using the prediction layer, maps the first fusion feature to the category space, outputs the scores corresponding to each water surface tracer category of the corresponding water surface tracer through a fully connected layer or a convolutional layer, and then uses the Softmax function to normalize the scores, converting the scores into a probability distribution such that the sum of the probabilities of all categories is 1, obtaining the probabilities corresponding to each category of the corresponding water surface tracer, and determining the category with the highest probability as the category of the corresponding water surface tracer.

[0120] Then, the server uses the prediction layer to perform regression calculation on the first fused feature, calculates the position and size of the corresponding water surface tracer according to the first fused feature, and obtains the information of multiple first candidate detection frames corresponding to the image data of the current frame, including the confidence of the candidate frame (i.e., the probability that the water surface tracer is included in the candidate frame), the central position coordinates of the candidate frame, the length and width of the candidate frame, etc.

[0121] After that, the server can use the non-maximum suppression method to determine the first target detection frame from multiple first candidate detection frames. First, all the first candidate detection frames are sorted in descending order of confidence, and the first candidate detection frame with the highest current confidence is selected as the retained frame, that is, the frame most likely to contain the water surface tracer. Then, the intersection-over-union ratio between the retained frame and the remaining first candidate detection frames is calculated to judge the overlap degree between these frames, and the first candidate detection frames with an intersection-over-union ratio greater than a preset threshold (such as 0.5) are discarded, and the above steps of selecting the retained frame, calculating the overlap degree, and removing the overlapping frames are repeated to obtain the finally remaining first candidate detection frame, that is, the first target detection frame.

[0122] Similarly, the server uses the prediction layer to perform classification calculation on the second fused feature to obtain the category information of the water surface tracer in the image data of the next frame, uses the prediction layer to perform regression calculation on the second fused feature to obtain the information of multiple second candidate detection frames corresponding to the image data of the next frame, and uses the non-maximum suppression method to screen and obtain the second target detection frame from multiple second candidate detection frames.

[0123] Step S503, use the object tracking algorithm to update the historical trajectories of all water surface tracers in the current frame based on the first target detection frame and the second target detection frame to obtain the target trajectories of all water surface tracers in the current frame. For details, please refer to Figure 1 Step S103 of the illustrated embodiment, which will not be elaborated here.

[0124] Step S504, obtain the water surface flow velocity of the image data of the current frame based on the target trajectory. For details, please refer to Figure 1 Step S104 of the illustrated embodiment, which will not be elaborated here.

[0125] In the embodiments of the present disclosure, by using the feature extraction layer, attention mechanism, feature fusion layer, and prediction layer in the object detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame and the image data of the next frame, the features of the water surface tracer in the image data at different scales can be accurately captured, accurate target detection frames can be obtained, and the detection accuracy of the water surface flow velocity can be improved.

[0126] In some alternative embodiments, the present embodiment provides a method for detecting the water surface flow velocity, as Figure 6 shown, Figure 6It is a schematic flowchart of another water surface flow velocity detection method according to an embodiment of the present disclosure. This process can be applied to a server and includes the following steps:

[0127] Step S601, obtain multiple frames of image data to be detected. The image data includes water surface tracers for water surface flow velocity detection. For details, please refer to Figure 5 Step S501 of the embodiment shown, which will not be elaborated here.

[0128] Step S602, use a target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame. Use the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame. Among them, the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer. The attention mechanism is located between the feature extraction layer and the feature fusion layer. For details, please refer to Figure 5 Step S502 of the embodiment shown, which will not be elaborated here.

[0129] Step S603, use a target tracking algorithm based on the first target detection box and the second target detection box to update the historical trajectories of all water surface tracers in the current frame to obtain the target trajectories of all water surface tracers in the current frame.

[0130] Specifically, the above step S603 includes:

[0131] Step S6031, obtain the status information of the first target detection box.

[0132] Optionally, in an embodiment of the present disclosure, the status information of the first target detection box is an 8-dimensional vector, including the center position coordinates (u, v) of the first target detection box, the aspect ratio y, the height h, the horizontal velocity of the center position abscissa (i.e., the moving velocity of the corresponding water surface tracer in the horizontal direction), the vertical velocity of the center position ordinate (i.e., the moving velocity of the corresponding water surface tracer in the vertical direction), the aspect ratio change velocity, and the height change velocity.

[0133] Step S6032, predict the trajectories of the next frame of all water surface tracers in the current frame based on the status information to obtain the tracking boxes of the next frame of all water surface tracers in the current frame.

[0134] Optionally, in an embodiment of the present disclosure, the server uses a Kalman filter. First, construct a state transition matrix and an observation matrix according to the status information of the first target detection box. Then, predict the trajectories of the next frame of all water surface tracers in the current frame based on the status information of the first target detection box and the state transition matrix to obtain the tracking boxes of the next frame of all water surface tracers in the current frame.

[0135] Step S6033: Match the second target detection box with the tracking box of the next frame to obtain a matching result.

[0136] Optionally, in the embodiments of the present disclosure, as Figure 2 shown, the server uses the Hungarian algorithm to match the second target detection box with the tracking box of the next frame to obtain a matching result.

[0137] In some alternative embodiments, the above step S6033 includes:

[0138] Step b1: Based on the historical unassociated frame numbers of all water surface tracers in the current frame, determine the states of the tracking boxes of the next frame, obtaining the first tracking boxes with determined states and the second tracking boxes with undetermined states.

[0139] Step b2: Perform a first match between the first tracking boxes and the second target detection boxes to obtain a first matching result, where the first matching result includes third tracking boxes and fourth tracking boxes. The third tracking boxes are the tracking boxes that are not matched with the second target detection boxes, and the fourth tracking boxes are the tracking boxes that are not matched with the second target detection boxes in the case of detecting new water surface tracers.

[0140] Step b3: Perform a second match between the second tracking boxes, the third tracking boxes, and the fourth tracking boxes to obtain a second matching result.

[0141] Optionally, in the embodiments of the present disclosure, the historical unassociated frame number refers to the number of frames since the last successful association of each water surface tracer. The tracking boxes of the next frame include the first tracking boxes with determined states and the second tracking boxes with undetermined states. The first matching result includes the third tracking boxes that are not matched with the second target detection boxes, and the fourth tracking boxes that are not matched with the second target detection boxes in the case of detecting new water surface tracers.

[0142] Specifically, when the server uses the Kalman filter to track all water surface tracers, for each water surface tracer, calculate the number of frames since the last successful association to obtain the historical unassociated frame number. If the current frame is successfully associated with the next frame (i.e., the second target detection box is successfully matched with the tracking box of the next frame), reset the historical unassociated frame number to 0; if the current frame is not successfully associated with the next frame (i.e., the second target detection box is not matched with the tracking box of the next frame), increment the historical unassociated frame number by 1.

[0143] First, the server determines the state of the tracking box for the next frame based on the historical unassociated frame numbers of all water surface tracers in the current frame. The state of the tracking box for the next frame includes a determined state, an undetermined state, and a deleted state. For each water surface tracer, if its historical unassociated frame number exceeds a preset frame number, the state of the corresponding tracking box for the next frame is the deleted state; if it is successfully associated in each frame before the current frame, the state of the corresponding tracking box for the next frame is the determined state, that is, the first tracking box; if its historical unassociated frame number does not exceed the preset frame number and there are frames that have not been successfully associated before the current frame, the state of the corresponding tracking box for the next frame is the undetermined state, that is, the second tracking box.

[0144] Then, the server performs a first matching (i.e., cascade matching) between the first tracking box in the determined state and the second target detection box to obtain a first matching result (i.e., cascade matching result). More specifically, the server classifies the first tracking boxes according to the occlusion time they have experienced, and determines the matching order according to the principle that the shorter the occlusion time, the higher the matching level (for example, start matching from the first tracking box with the shortest occlusion time). Then, the first tracking boxes are matched with the second target detection box in the matching order to obtain a third tracking box that is not matched with the second target detection box, and in the case of detecting a new water surface tracer, a fourth tracking box that is not matched with the second target detection box.

[0145] After that, the server performs a second matching (i.e., intersection over union matching) between the second tracking box, the third tracking box, and the fourth tracking box to obtain a second matching result (i.e., intersection over union matching result). More specifically, the server calculates the intersection over union between the second tracking box and the third tracking box and the fourth tracking box respectively to obtain the second matching result.

[0146] In the above embodiments, by performing the first matching and the second matching between the second target detection box and the tracking box for the next frame, the accuracy and stability of tracking water surface tracers can be improved, and the detection accuracy of water surface flow velocity can be improved.

[0147] Step S6034: Update the historical trajectories of all water surface tracers in the current frame based on the matching results to obtain the target trajectories.

[0148] Optionally, in the embodiments of the present disclosure, the matching results include a first matching result (i.e., cascade matching result) and a second matching result (i.e., intersection over union matching result). The server uses a Kalman filter to update the historical trajectories of all water surface tracers in the current frame according to the matching results to obtain the target trajectories.

[0149] In some alternative embodiments, the above step S6034 includes:

[0150] Step c1: Update the historical trajectories of all water surface tracers in the corresponding current frame using the fifth tracking box to obtain the first target trajectory.

[0151] Step c2: Determine the state of the third tracking box based on the historical non - associated frame numbers of all water surface tracers in the current frame corresponding to the third tracking box to obtain the first tracking box and the second tracking box.

[0152] Step c3: Compare the historical non - associated frame number of the first tracking box with a preset number of frames to obtain the sixth tracking box and the seventh tracking box, where the sixth tracking box is the tracking box with the historical non - associated frame number greater than the preset number of frames, and the seventh tracking box is the tracking box with the historical non - associated frame number less than the preset number of frames.

[0153] Step c4: Use the fourth tracking box and the seventh tracking box in the second matching result to create new trajectories of all water surface tracers in the corresponding current frame to obtain the second target trajectory.

[0154] Step c5: Delete the historical trajectories of all water surface tracers in the current frame corresponding to the second tracking box and the sixth tracking box.

[0155] Step c6: Obtain the target trajectory based on the first target trajectory and the second target trajectory.

[0156] Optionally, in the embodiments of the present disclosure, the first matching result includes, in addition to the third tracking box and the fourth tracking box mentioned in the above embodiments, a fifth tracking box that successfully matches the second target detection box. The second matching result includes the third tracking box, the fourth tracking box, and the fifth tracking box.

[0157] Among them, the third tracking box in the second matching result includes the first tracking box and the second tracking box. The first tracking box includes the sixth tracking box with the historical non - associated frame number greater than the preset number of frames and the seventh tracking box with the historical non - associated frame number less than the preset number of frames.

[0158] Specifically, the server updates the historical trajectories of all water surface tracers in the corresponding current frame using the fifth tracking box to obtain the first target trajectory, creates new trajectories of all water surface tracers in the corresponding current frame using the fourth tracking box and the seventh tracking box in the second matching result to obtain the second target trajectory, and then combines the first target trajectory and the second target trajectory to obtain the target trajectory.

[0159] In addition, the server also needs to delete the historical trajectories of all water surface tracers in the current frame corresponding to the second tracking box and the sixth tracking box.

[0160] In the above embodiments, by updating the historical trajectories of all water surface tracers in the current frame based on the first matching result and the second matching result, it is possible to continuously and stably track all water surface tracers, obtain accurate target trajectories, and improve the detection accuracy of water surface flow velocity.

[0161] Step S604, obtain the water surface flow velocity of the image data of the current frame based on the target trajectory.

[0162] Specifically, the above step S604 includes:

[0163] Step S6041, based on the target trajectory of each water surface tracer in the current frame, obtain the displacement information of each water surface tracer in the current frame.

[0164] Optionally, in the embodiments of the present disclosure, the server calculates the displacement information of each water surface tracer in the current frame according to the coordinate information of the first target detection box and the coordinate information of the second target detection box in the target trajectory of each water surface tracer in the current frame.

[0165] Step S6042, based on the displacement information and the time interval between the current frame and the next frame, obtain the flow velocity of each water surface tracer in the current frame.

[0166] Optionally, in the embodiments of the present disclosure, the server calculates the time interval between the current frame and the next frame according to the frame rate of the video data to be detected, and calculates the flow velocity of each water surface tracer in the current frame according to the displacement information of each water surface tracer in the current frame and the time interval between the current frame and the next frame.

[0167] Step S6043, fuse the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame.

[0168] Optionally, in the embodiments of the present disclosure, the server performs a fusion calculation (such as average calculation, median calculation, mode calculation, weighted calculation, etc.) on the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame.

[0169] In the embodiments of the present disclosure, by using the target tracking algorithm to update the historical trajectories of all water surface tracers in the current frame, it is possible to continuously and stably track all water surface tracers, obtain accurate target trajectories, and improve the detection accuracy of water surface flow velocity. By obtaining the flow velocity of each water surface tracer in the current frame based on the target trajectory of each water surface tracer in the current frame and fusing the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame, it is possible to reduce the error of the water surface flow velocity, ensure the reliability of the water surface flow velocity, and improve the detection accuracy of the water surface flow velocity.

[0170] In an alternative embodiment, before the server extracts features, fuses features, and makes predictions on the image data of the current frame using the target detection network, it can also train the initial detection network to obtain the target detection network.

[0171] Specifically, the server obtains the water surface tracer image data through methods such as web crawlers or on-site shooting, further expands the data set using data augmentation methods such as random rotation and random cropping. After unifying the image sizes, the server manually annotates the categories and bounding boxes of the water surface tracers in the images using annotation tools to obtain the water surface tracer data set, and divides the data set into a training set, a validation set, and a test set according to a certain ratio (such as 8:1:1). The training set is used to train the initial detection network, the validation set is used to adjust the hyperparameters of the initial detection network, and finally the test set is used to test the model effect of the initial detection network.

[0172] It should be noted that when the server trains the initial detection network, the selected loss function is the complete intersection over union loss function, and the optimization algorithm is the stochastic gradient descent algorithm, where the momentum is 0.9, the weight decay is 0.0001, and the initial learning rate is 0.0001. In the warm-up stage of training (i.e., the first 10 iterations), the server linearly increases the learning rate to 0.001, and then decays the learning rate with an exponential coefficient of 0.95. The number of iterations is 100. When the loss on the validation set no longer decreases (i.e., the detection box basically coincides with the annotated bounding box), the training of the initial detection network is stopped to obtain the target detection network.

[0173] This embodiment provides a water surface flow velocity detection device, as Figure 7 shown, including:

[0174] An acquisition module 701, configured to acquire multiple frames of image data to be detected, where the image data includes water surface tracers for water surface flow velocity detection;

[0175] A first obtaining module 702, configured to extract features, fuse features, and make predictions on the image data of the current frame using the target detection network to obtain a first target detection box corresponding to the image data of the current frame, and extract features, fuse features, and make predictions on the image data of the next frame using the target detection network to obtain a second target detection box corresponding to the image data of the next frame, where the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer;

[0176] A second obtaining module 703, configured to update the historical trajectories of all water surface tracers in the current frame based on the first target detection box and the second target detection box using a target tracking algorithm to obtain the target trajectories of all water surface tracers in the current frame;

[0177] A third obtaining module 704, configured to obtain the water surface flow velocity of the image data of the current frame based on the target trajectory.

[0178] In the embodiments of the present disclosure, by acquiring multiple frames of image data to be detected, the image data includes water surface tracers for water surface flow velocity detection; using a target detection network to perform feature extraction, feature fusion, and prediction on the image data of the current frame to obtain a first target detection box corresponding to the image data of the current frame, and using the target detection network to perform feature extraction, feature fusion, and prediction on the image data of the next frame to obtain a second target detection box corresponding to the image data of the next frame, where the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer, and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer; using a target tracking algorithm to update the historical trajectories of all water surface tracers in the current frame based on the first target detection box and the second target detection box to obtain the target trajectories of all water surface tracers in the current frame; obtaining the water surface flow velocity of the image data of the current frame based on the target trajectories. Since the embodiments of the present disclosure fuse the attention mechanism in the target detection network and use the target tracking algorithm to track the water surface tracers, the accuracy of water surface flow velocity detection is improved.

[0179] In some alternative embodiments, the first obtaining module 702 includes:

[0180] A first obtaining sub-module, configured to perform feature extraction on the image data of the current frame based on the feature extraction layer to obtain a first feature, and perform feature extraction on the image data of the next frame to obtain a second feature;

[0181] A second obtaining sub-module, configured to perform a convolution operation on the image data of the current frame based on a convolution layer to obtain a third feature, and select a first network feature from the third feature that comes from a preset layer, perform a convolution operation on the image data of the next frame to obtain a fourth feature, and select a second network feature from the fourth feature that comes from a preset layer;

[0182] A third obtaining sub-module, configured to weight the first feature based on the attention mechanism to obtain a weighted first feature, and weight the second feature to obtain a weighted second feature;

[0183] A fourth obtaining sub-module, configured to perform feature fusion on the first network feature and the weighted first feature based on the feature fusion layer to obtain a first fusion feature, and perform feature fusion on the second network feature and the weighted second feature to obtain a second fusion feature;

[0184] An output sub-module, configured to detect the input first fusion feature based on the prediction layer and output a first target detection box corresponding to the image data of the current frame, and detect the input second fusion feature and output a second target detection box corresponding to the image data of the next frame.

[0185] In some alternative embodiments, the third obtaining sub-module includes:

[0186] A first obtaining unit, configured to perform a pooling operation on the first feature based on an average pooling layer to obtain a first tensor, and perform a pooling operation on the second feature to obtain a second tensor;

[0187] A second obtaining unit, configured to perform a one-dimensional convolution on the first tensor based on a one-dimensional convolutional layer to obtain a third tensor, and perform a one-dimensional convolution on the second tensor to obtain a fourth tensor;

[0188] A third obtaining unit, configured to use an activation function to transform the third tensor into a preset interval to obtain a first weight, and transform the fourth tensor into a preset interval to obtain a second weight;

[0189] A fourth obtaining unit, configured to fuse the first feature and the first weight to obtain a weighted first feature, and fuse the second feature and the second weight to obtain a weighted second feature.

[0190] In some alternative embodiments, the second obtaining module 703 includes:

[0191] A first acquiring sub-module, configured to acquire status information of a first target detection box;

[0192] A fifth obtaining sub-module, configured to predict the trajectories of the next frames of all water surface tracers in the current frame based on the status information to obtain the tracking boxes of the next frames of all water surface tracers in the current frame;

[0193] A sixth obtaining sub-module, configured to match the second target detection box with the tracking boxes of the next frame to obtain a matching result;

[0194] A seventh obtaining sub-module, configured to update the historical trajectories of all water surface tracers in the current frame based on the matching result to obtain target trajectories.

[0195] In some alternative embodiments, the sixth obtaining sub-module includes:

[0196] A fifth obtaining unit, configured to determine the status of the tracking boxes of the next frame based on the historical unassociated frame numbers of all water surface tracers in the current frame to obtain a first tracking box with a determined status and a second tracking box with an undetermined status;

[0197] A sixth obtaining unit, configured to perform a first matching between the first tracking box and the second target detection box to obtain a first matching result, where the first matching result includes a third tracking box and a fourth tracking box, the third tracking box is a tracking box that is not matched with the second target detection box, and the fourth tracking box is a tracking box that is not matched with the second target detection box in the case of detecting a new water surface tracer;

[0198] A seventh obtaining unit, configured to perform a second matching on the second tracking frame, the third tracking frame, and the fourth tracking frame to obtain a second matching result.

[0199] In some alternative embodiments, the seventh obtaining sub-module includes:

[0200] An eighth obtaining unit, configured to update the historical trajectories of all water surface tracers in the corresponding current frame by using a fifth tracking frame to obtain a first target trajectory;

[0201] A ninth obtaining unit, configured to determine the states of the third tracking frame based on the historical non-association frames of all water surface tracers in the current frame corresponding to the third tracking frame, to obtain a first tracking frame and a second tracking frame;

[0202] A tenth obtaining unit, configured to compare the historical non-association frames of the first tracking frame with a preset number of frames to obtain a sixth tracking frame and a seventh tracking frame, where the sixth tracking frame is a tracking frame with historical non-association frames greater than the preset number of frames, and the seventh tracking frame is a tracking frame with historical non-association frames less than the preset number of frames;

[0203] An eleventh obtaining unit, configured to create new trajectories of all water surface tracers in the corresponding current frame by using the fourth tracking frame and the seventh tracking frame in the second matching result to obtain a second target trajectory;

[0204] A deletion unit, configured to delete the historical trajectories of all water surface tracers in the current frame corresponding to the second tracking frame and the sixth tracking frame;

[0205] A twelfth obtaining unit, configured to obtain a target trajectory based on the first target trajectory and the second target trajectory.

[0206] In some alternative embodiments, the third obtaining module 704 includes:

[0207] A second obtaining sub-module, configured to obtain the displacement information of each water surface tracer in the current frame based on the target trajectory of each water surface tracer in the current frame;

[0208] An eighth obtaining sub-module, configured to obtain the flow velocity of each water surface tracer in the current frame based on the displacement information and the time interval between the current frame and the next frame;

[0209] A ninth obtaining sub-module, configured to fuse the flow velocities of each water surface tracer in the current frame to obtain the water surface flow velocity of the image data of the current frame.

[0210] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding foregoing embodiments, and will not be elaborated herein.

[0211] The water surface flow velocity detection device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0212] This embodiment of the disclosure also provides a computer device having the above Figure 7 shown water surface flow velocity detection device.

[0213] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a computer device provided by an alternative embodiment of the disclosure. As Figure 8 shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 8 In

[0214] FIG. 14, one processor 10 is taken as an example.

[0215] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0216] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0217] The memory 20 may include a volatile memory, for example, a random access memory; the memory may also include a non-volatile memory, for example, a flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memories.

[0218] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0219] The embodiments of the present disclosure also provide a computer-readable storage medium. The methods according to the embodiments of the present disclosure can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and to be stored in a local storage medium, so that the methods described herein can be stored in such software processes on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0220] A part of the present disclosure can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present disclosure through the operations of the computer. Those skilled in the art should understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways for computer program instructions to be executed by a computer include, but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.

[0221] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting water surface flow velocity, characterized in that: The method comprises: Acquire multiple frames of image data to be detected, wherein the image data includes a water surface tracer for water surface flow velocity detection; Using the target detection network to perform feature extraction, feature fusion and prediction on the image data of the current frame to obtain a first target detection frame corresponding to the image data of the current frame, and using the target detection network to perform feature extraction, feature fusion and prediction on the image data of the next frame to obtain a second target detection frame corresponding to the image data of the next frame, wherein the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer; Using a target tracking algorithm, based on the first target detection frame and the second target detection frame, the historical tracks of all the water surface tracers in the current frame are updated to obtain the target tracks of all the water surface tracers in the current frame; The water surface flow velocity of the image data of the current frame is obtained based on the target trajectory.

2. The method according to claim 1, characterized in that The feature extraction layer includes a plurality of convolutional layers, and the target detection network is used to perform feature extraction, feature fusion and prediction on the image data of the current frame to obtain a first target detection frame corresponding to the image data of the current frame, and the target detection network is used to perform feature extraction, feature fusion and prediction on the image data of the next frame to obtain a second target detection frame corresponding to the image data of the next frame, including: Performing feature extraction on the image data of the current frame based on the feature extraction layer to obtain a first feature, and performing feature extraction on the image data of the next frame to obtain a second feature; Performing a convolution operation on the image data of the current frame based on the convolution layer to obtain a third feature, and selecting a first network feature from a preset layer from the third feature, performing a convolution operation on the image data of the next frame to obtain a fourth feature, and selecting a second network feature from the preset layer from the fourth feature; Weighting the first feature based on the attention mechanism to obtain a weighted first feature, and weighting the second feature to obtain a weighted second feature; Based on the feature fusion layer, the first network feature and the weighted first feature are subjected to feature fusion to obtain a first fused feature, and the second network feature and the weighted second feature are subjected to feature fusion to obtain a second fused feature; Based on the prediction layer, the first input fusion feature is detected, and a first target detection frame corresponding to the image data of the current frame is output; the second input fusion feature is detected, and a second target detection frame corresponding to the image data of the next frame is output.

3. The method according to claim 2, characterized in that The step of weighting the first feature based on the attention mechanism to obtain the weighted first feature, and weighting the second feature to obtain the weighted second feature includes: Performing a pooling operation on the first feature based on an average pooling layer to obtain a first tensor, and performing a pooling operation on the second feature to obtain a second tensor; Performing a one-dimensional convolution on the first tensor based on a one-dimensional convolution layer to obtain a third tensor, and performing a one-dimensional convolution on the second tensor to obtain a fourth tensor; The third tensor is transformed into a preset interval by using an activation function to obtain a first weight, and the fourth tensor is transformed into the preset interval to obtain a second weight; The first feature and the first weight are fused to obtain the weighted first feature, and the second feature and the second weight are fused to obtain the weighted second feature.

4. The method according to claim 1, characterized in that: The method of updating the historical tracks of all water surface tracers in the current frame based on the first target detection frame and the second target detection frame by using the target tracking algorithm to obtain the target tracks of all water surface tracers in the current frame includes: Obtaining status information of the first object detection frame; Predicting the trajectories of the next frame of all the water surface tracers in the current frame based on the state information to obtain the tracking frames of the next frame of all the water surface tracers in the current frame; Matching the second target detection frame with the tracking frame of the next frame to obtain a matching result; The historical tracks of all water surface tracers in the current frame are updated based on the matching result to obtain the target track.

5. The method according to claim 4, characterized in that The matching the second target detection frame with the tracking frame of the next frame to obtain a matching result includes: Based on the number of historical unrelated frames of all water surface tracers in the current frame, determining the state of the tracking frame of the next frame, and obtaining a first tracking frame in a determined state and a second tracking frame in an undetermined state; Performing a first matching on the first tracking frame and the second target detection frame to obtain a first matching result, wherein the first matching result includes a third tracking frame and a fourth tracking frame, the third tracking frame is a tracking frame that does not match the second target detection frame, and the fourth tracking frame is a tracking frame that does not match the second target detection frame when a new water surface tracer is detected; The second tracking frame, the third tracking frame and the fourth tracking frame are subjected to a second matching to obtain a second matching result.

6. The method according to claim 5, characterized in that The first matching result also includes a fifth tracking frame, the fifth tracking frame is a tracking frame that successfully matches the second target detection frame, the second matching result includes the third tracking frame, the fourth tracking frame, and the fifth tracking frame, and the updating of the historical tracks of all water surface tracers in the current frame based on the matching result to obtain the target track includes: Using the fifth tracking frame to update the historical tracks of all water surface tracers in the corresponding current frame, to obtain a first target track; Determine the state of the third tracking frame based on the number of historical unrelated frames of all water surface tracers in the current frame corresponding to the third tracking frame, and obtain the first tracking frame and the second tracking frame; Compare the number of historical unrelated frames of the first tracking frame with a preset number of frames to obtain a sixth tracking frame and a seventh tracking frame, wherein the sixth tracking frame is a tracking frame whose number of historical unrelated frames is greater than the preset number of frames, and the seventh tracking frame is a tracking frame whose number of historical unrelated frames is less than the preset number of frames; Using the fourth tracking frame and the seventh tracking frame in the second matching result to create new tracks of all water surface tracers in the corresponding current frame, to obtain a second target track; Deleting the historical tracks of all water surface tracers in the current frame corresponding to the second tracking frame and the sixth tracking frame; The target trajectory is obtained based on the first target trajectory and the second target trajectory.

7. The method according to claim 1, characterized in that The step of obtaining the water surface velocity of the image data of the current frame based on the target trajectory includes: Based on the target track of each water surface tracer in the current frame, obtaining displacement information of each water surface tracer in the current frame; Based on the displacement information and the time interval between the current frame and the next frame, the flow velocity of each water surface tracer in the current frame is obtained; The flow velocity of each water surface tracer in the current frame is fused to obtain the water surface flow velocity of the image data of the current frame.

8. A water surface flow velocity detection device, characterized in that: The device comprises: An acquisition module, used for acquiring multiple frames of image data to be detected, wherein the image data includes a water surface tracer for water surface flow velocity detection; A first obtaining module is used to perform feature extraction, feature fusion and prediction on the image data of the current frame using the target detection network to obtain a first target detection frame corresponding to the image data of the current frame, and to perform feature extraction, feature fusion and prediction on the image data of the next frame using the target detection network to obtain a second target detection frame corresponding to the image data of the next frame, wherein the target detection network includes a feature extraction layer, an attention mechanism, a feature fusion layer and a prediction layer, and the attention mechanism is located between the feature extraction layer and the feature fusion layer; A second obtaining module is used to update the historical tracks of all water surface tracers in the current frame based on the first target detection frame and the second target detection frame by using a target tracking algorithm to obtain the target tracks of all water surface tracers in the current frame; The third obtaining module is used to obtain the water surface flow velocity of the image data of the current frame based on the target trajectory.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the water surface flow velocity detection method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the water surface flow velocity detection method according to any one of claims 1 to 7.

Images