Water surface floating object detection method based on complementary space-time attention mechanism

Abstract

A water surface floating object detection method based on a complementary space-time attention mechanism comprises the following steps: S1, carrying out optical flow calculation on RGB frames shot by a fixed camera position to generate optical flow frames; s2, the RGB frames and the optical flow frames are sent into a feature extraction module which is constructed based on a convolutional neural network and shares two parameters frame by frame, and spatial feature information is extracted; s3, the extracted RGB and optical flow modal features are respectively sent to a cross-modal feature fusion module to generate fusion features, the fusion features are sent to a feature pyramid, and features fused with multi-scale information are output; and S4, sending the obtained features with lower levels into a short-range global Transform module, extracting global features containing advanced semantic information layer by layer, carrying out up-sampling on the global features and the features of the last layer of the optical flow mode, sending the extracted features and features of the last layer of the optical flow mode into a long-range local Transform module with lower processing levels, outputting feature mapping considering global and local modeling, sending the feature mapping into a full-connection layer, and then sending the feature mapping into a full-connection layer. And outputting the frame coefficient and the classification required by the overwater floating object target frame. According to the method and the device, the water surface floating objects in the video picture can be detected more efficiently.

Description

Technical Field

[0001] The present invention belongs to the technical field of hydrological forecasting, and in particular relates to a method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism. Background Art

[0002] With the rapid development of society, ecological and environmental problems have become increasingly prominent, and water pollution has become one of the major problems that need to be solved urgently. Floating objects on the water surface, as a significant manifestation of water pollution, not only affect the health of water landscapes and water ecosystems, but also pose a potential threat to human health and economic development. Therefore, carrying out the task of detecting floating objects on the water surface has important practical significance and academic value.

[0003] Traditional detection of floating objects on the water surface often slices the video frame by frame and feeds it into the model for target prediction. Compared with direct prediction of the video, converting the video task into independent frame prediction reduces the amount of calculation and is conducive to the rapid popularization of the detection task. However, this method ignores the timing information between frames in the video, which may lead to discontinuity and decreased accuracy of the detection results. In contrast, direct prediction of the video utilizes the temporal correlation between video frames, which helps to improve the consistency and accuracy of the detection. By capturing the motion features in continuous frames, the video prediction method can more effectively identify and track the dynamic changes of floating objects on the water surface and reduce the probability of false detection and missed detection. Therefore, efficient modeling of spatiotemporal features is a key issue to ensure the performance of the task of floating object detection on the water surface.

[0004] With the development of deep learning, the introduction of the attention mechanism provides a new idea for solving the above problems. In recent years, Transformer has been widely used in the field of computer graphics due to its excellent global modeling capabilities. Sending two inputs from different sources as query vectors and key / value vectors into the mutual attention module can automatically learn the correlation between different inputs and dynamically adjust the weight distribution of features according to their importance, so that the model can selectively focus on the information that is most helpful for the current task. However, as the input sequence increases, the computational complexity of Transformer will also increase significantly, which seriously limits the application of the attention mechanism in the task of modeling features in the spatiotemporal dimension.

[0005] Most of the current research on using attention mechanisms to model spatiotemporal features simply decouples the spatiotemporal Transformer into two combinations that model temporal and spatial features separately, taking into account the computational cost of the non-local attention mechanism. Although simple decoupling is efficient, it loses the inherent potential relationship between the two dimensions of time and space. Summary of the invention

[0006] In order to solve the problems raised in the background technology, the present invention provides a method for detecting floating objects on the water surface based on a complementary spatiotemporal attention mechanism.

[0007] A method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism comprises the following steps: S1, performing optical flow calculation on the RGB frame shot by a fixed camera to generate an optical flow frame; S2, the RGB frame and the optical flow frame are fed into two parameter-sharing feature extraction modules based on convolutional neural networks one by one to extract spatial feature information; S3, extract the features of each layer of RGB and optical flow modalities They are sent to the cross-modal feature fusion module to generate fusion features , send the fused features into the feature pyramid, and output the features that fuse multi-scale information ; S4, will get the lower level features The short-range global Transformer module is used to extract global features containing high-level semantic information layer by layer, and then integrate them with the last layer of features of the optical flow modality. Together, after upsampling, it is sent to a relatively low-level processing The long-range local Transformer module outputs a feature map that takes into account both global and local modeling. After being sent to the fully connected layer, it outputs the box coefficients and classification required for the target box of floating objects on the water.

[0008] Furthermore, in step S2, the feature extraction module used to extract spatial feature information is composed of ResNet50 pre-trained by ImageNet. During the training and testing phases of floating object detection, the weights of the feature extraction modules of the two branches are frozen.

[0009] Furthermore, in step S3, the cross-modal feature fusion module is composed of a 3×3 convolution block connected in series with a ReLU activation function.

[0010] Furthermore, in step S4, the short-range global Transformer module and the long-range local Transformer module are respectively cascaded by N short-range global Transformer blocks and N long-range local Transformer blocks; the output of each layer of the short-range global Transformer block is upsampled and, with the help of its global semantic advantage, is sent to the long-range local Transformer block of the same level as a guide.

[0011] Furthermore, the short-range global Transformer block includes a multi-dimensional attention module and a multi-layer perceptron module connected in series. Both the multi-dimensional attention module and the multi-layer perceptron module need to undergo layer normalization operations before processing, and after processing, they need to be added pixel by pixel with the features of the jump connection.

[0012] Furthermore, the query vector, key vector, and value vector of the multidimensional attention module are all projected from the matrix-transposed .

[0013] Furthermore, the long-range local Transformer block includes an optical flow guided deformable attention module and a multi-layer perceptron module, and there are layer normalization operations and residual links between the optical flow guided deformable attention module and the multi-layer perceptron module.

[0014] Furthermore, the optical flow guided deformable attention module needs to accept the upsampled optical flow modality high-level features The input, after residual connection, needs to be concatenated pixel by pixel with the output of the short-range global Transformer block at the same level before being sent to the multi-layer perceptron module.

[0015] Furthermore, the query vector of the optical flow-guided deformable attention module is projected from , the key vector and the value vector are projected from .

[0016] Furthermore, the deformable mechanism is as follows: when the value vector participates in the attention calculation, only 9 elements interact with the key vector.

[0017] The present invention can achieve the following beneficial effects: 1. This method can efficiently integrate effective information in the time dimension and the space dimension, perform attention modeling in both the short-range global and long-range local directions, provide useful features for the task of detecting floating objects on the water surface, and thus ensure the stability of the visual algorithm.

[0018] 2. The convolutional projection layer can efficiently fuse the optical flow modality and RGB modality features to generate fused features that reflect both motion trends and detailed textures; the fused features are sent to the feature pyramid to achieve cross-fusion of multi-scale features; at the same time, combined with the expression characteristics of features at different levels, the short-range global Transformer module and the long-range local Transformer module are implemented respectively, which reduces the amount of calculation while allowing the model to capture global and local beneficial features in the spatiotemporal dimensions at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 This is a flow chart of a method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to the present invention. DETAILED DESCRIPTION

[0020] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0021] like Figure 1 As shown, a method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism comprises the following steps: Step 1, perform optical flow calculation on the RGB frame shot by a fixed camera to generate an optical flow frame; Step 2: Send the RGB frame and the optical flow frame frame by frame to two parameter-sharing feature extraction modules based on convolutional neural networks to extract spatial feature information. Specifically, the feature extraction module used to extract spatial feature information is ResNet50 pre-trained with ImageNet. During the training and testing phases of floating object detection, the weights of the feature extraction modules of the two branches are frozen.

[0022] Step 3: Extract the features of each layer of RGB and optical flow modalities They are sent to the cross-modal feature fusion module respectively, where Refers to the extracted RGB layer features, Refers to the extracted features of each layer of the optical flow modality to generate fusion features , send the fused features into the feature pyramid, and output the features that fuse multi-scale information It should be noted that the cross-modal feature fusion module is composed of a 3×3 convolution block and a ReLU activation function in series.

[0023] Step 4: Get lower-level features The short-range global Transformer module is used to extract global features containing high-level semantic information layer by layer, and then integrate them with the last layer of features of the optical flow modality. Together, after upsampling, it is sent to a relatively low-level processing The long-range local Transformer module outputs a feature map that takes into account both global and local modeling. After being sent to the fully connected layer, it outputs the box coefficients and classification required for the target box of floating objects on the water.

[0024] Among them, the short-range global Transformer module and the long-range local Transformer module are respectively composed of N short-range global Transformer blocks and N long-range local Transformer blocks cascaded. After upsampling, the output of each layer of the short-range global Transformer block is sent to the long-range local Transformer block of the same level as a guide with its global semantic advantage.

[0025] The specific composition and processing flow of the short-range global Transformer block and the long-range local Transformer block are as follows: The short-range global Transformer block includes a multi-dimensional attention module and a multi-layer perceptron module connected in series. Both the multi-dimensional attention module and the multi-layer perceptron module need to undergo layer normalization operations before processing, and after processing, they need to be added pixel by pixel with the features of the jump connection.

[0026] The query vector, key vector, and value vector of the multidimensional attention module are projected from the matrix-transposed Specifically, yes The matrix is ​​sampled by a sliding window with a width of 3. The matrix transpose operation is to concatenate the time dimension and batch dimension of the input features, and the dense attention calculation is performed on both time and space as a three-dimensional input.

[0027] The long-range local Transformer block includes an optical flow guided deformable attention module and a multi-layer perceptron module. Similar to the short-range global Transformer block, there is a layer normalization operation and a residual link between the optical flow guided deformable attention module and the multi-layer perceptron module. The optical flow guided deformable attention module needs to accept the upsampled optical flow modality high-level features The input, after residual connection, needs to be concatenated pixel by pixel with the output of the short-range global Transformer block at the same level before being sent to the multi-layer perceptron module.

[0028] The query vector of the optical flow-guided deformable attention module is projected from , the key vector and the value vector are projected from .

[0029] The specific deformable mechanism is that when the value vector participates in the attention calculation, only 9 elements interact with the key vector, so as to ensure long-range modeling while significantly reducing the amount of calculation.

[0030] The implementation principle of the method for detecting floating objects on the water surface based on the complementary spatiotemporal attention mechanism in this application is as follows: in the temporal dimension, short-range information can provide clear and accurate guidance for detection, while long-range information helps to deal with challenging situations such as occlusion and motion blur. In the spatial dimension, local context can provide fine details, while global context is conducive to contrastive learning to locate salient objects. Therefore, a more effective solution is to combine short-range and long-range temporal cues with global and local spatial information at the same time and take advantage of their complementarity.

[0031] In the face of videos captured from fixed positions, global and local attention modeling is performed in the time and space dimensions to efficiently detect floating objects on the water surface in the video. In this method, the convolutional projection layer can efficiently fuse the optical flow modality and RGB modality features to generate fused features that reflect both motion trends and detailed textures; the fused features are sent to the feature pyramid to achieve cross-fusion of multi-scale features.

[0032] At the same time, combined with the expression characteristics of features at different levels, short-range global Transformer modules and long-range local Transformer modules are implemented respectively to reduce the amount of calculation while allowing the model to capture global and local beneficial features in the spatiotemporal dimension at the same time. Specifically, relatively high-level features have richer semantic information and relatively less spatial information. Implementing dense attention modeling on their adjacent frames can generate high-level features containing global semantic information under a relatively low computational burden; relatively low-level features have richer texture details, and direct attention calculation resources are unbearable. Using the motion expression of the optical flow frame as a guide, deformable attention modeling is performed on the local moving parts that are more worthy of attention, so that the model can pay more attention to the beneficial detail features more efficiently.

[0033] Therefore, this method can efficiently integrate effective information in the time dimension and the space dimension, perform attention modeling in both the short-range global and long-range local directions, provide useful features for the task of floating object detection on the water surface, and thus ensure the stability of the visual algorithm.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism, characterized in that: The following steps are involved: S1, performing optical flow calculation on the RGB frame shot by a fixed camera to generate an optical flow frame; S2, the RGB frame and the optical flow frame are fed into two parameter-sharing feature extraction modules based on convolutional neural networks one by one to extract spatial feature information; S3, extract the features of each layer of RGB and optical flow modalities They are sent to the cross-modal feature fusion module to generate fusion features , send the fused features into the feature pyramid, and output the features that fuse multi-scale information ; S4, will get the lower level features The short-range global Transformer module is used to extract global features containing high-level semantic information layer by layer, and then integrate them with the last layer of features of the optical flow modality. Together, after upsampling, it is sent to a relatively low-level processing The long-range local Transformer module outputs a feature map that takes into account both global and local modeling. After being sent to the fully connected layer, it outputs the box coefficients and classification required for the target box of floating objects on the water.

2. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 1, characterized in that: In step S2, the feature extraction module used to extract spatial feature information is ResNet50 pre-trained by ImageNet. During the training and testing phases of floating object detection, the weights of the feature extraction modules of the two branches are frozen.

3. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 1, characterized in that: In step S3, the cross-modal feature fusion module is composed of a 3×3 convolution block connected in series with a ReLU activation function.

4. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 1, characterized in that: In step S4, the short-range global Transformer module and the long-range local Transformer module are respectively cascaded by N short-range global Transformer blocks and N long-range local Transformer blocks; the output of each layer of the short-range global Transformer block is upsampled and, with its global semantic advantage, is sent to the long-range local Transformer block of the same level as a guide.

5. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 4, characterized in that: The short-range global Transformer block includes a multi-dimensional attention module and a multi-layer perceptron module connected in series. Both the multi-dimensional attention module and the multi-layer perceptron module need to undergo layer normalization operations before processing, and after processing, they need to be added pixel by pixel with the features of the jump connection.

6. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 5, characterized in that: The query vector, key vector, and value vector of the multidimensional attention module are projected from the matrix-transposed .

7. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 1, characterized in that: The long-range local Transformer block includes an optical flow-guided deformable attention module and a multi-layer perceptron module, between which there are layer normalization operations and residual links.

8. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 7, characterized in that: The optical flow guided deformable attention module needs to accept the upsampled optical flow modality high-level features The input, after residual connection, needs to be concatenated pixel by pixel with the output of the short-range global Transformer block at the same level before being sent to the multi-layer perceptron module.

9. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 8, characterized in that: The query vector of the optical flow-guided deformable attention module is projected from , the key vector and the value vector are projected from .

10. The method for detecting floating objects on a water surface based on a complementary spatiotemporal attention mechanism according to claim 9, characterized in that: The specific deformable mechanism is that when the value vector participates in the attention calculation, only 9 elements interact with the key vector.

Images