Marine litter identification method fusing global scene and local target features

Abstract

The invention discloses a marine litter identification method fusing global scene and local target features, and belongs to the technical field of remote sensing video recognizing.The method comprises the steps that marine remote sensing video data are collected, a data set is constructed, and then preprocessing is conducted; the method comprises the following steps: constructing a marine litter identification model which comprises a spatial-temporal feature extraction module, a high-low altitude multi-scale target feature extraction module, a global scene feature extraction module and a global scene and local target feature adaptive fusion module; a loss function is adopted to train the marine litter identification model, and parameters of the marine litter identification model are updated through a back propagation algorithm; and inputting the marine remote sensing video data into the trained marine litter identification model, and outputting related litter category or state information. According to the method, multiple modules cooperatively extract features and adaptively fuse the features, the marine litter recognition accuracy is improved, the method adapts to remote sensing video complex scenes, marine litter monitoring is assisted, and technical support is provided for marine ecological protection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing video recognition, and particularly relates to a marine garbage object recognition method fusing global scene and local target features. BACKGROUND

[0002] In recent years, with the pollution of the marine environment, the detection and recognition of marine garbage have become one of the key tasks in marine ecological monitoring. With the development of remote sensing technology, artificial intelligence and deep learning, marine garbage video recognition based on unmanned aerial vehicle or satellite platforms has gradually become an important research direction of efficient monitoring means. In this process, how to efficiently and accurately identify key targets such as floating garbage and accumulated pollutants from marine remote sensing videos is the key to improving the intelligent level of marine environmental governance.

[0003] The rapid development of deep learning has greatly promoted the application of computer vision in the field of remote sensing image and video analysis, especially in basic tasks such as image classification, target detection and semantic segmentation. However, marine garbage object recognition, as a complex task that fuses spatial, temporal and semantic modeling, still faces many challenges.

[0004] Specifically, on the one hand, the marine environment has the characteristics of wide open, complex background change, significant weather and light interference, etc., which makes the garbage target in the remote sensing image present the characteristics of large scale change, strong fusion of color and background, weak and small texture blur, etc., making it difficult for the recognition model to accurately locate the target; on the other hand, marine garbage has the characteristics of drift and dynamic aggregation, and the appearance of the target in the video sequence has temporal sparseness and uncertainty, combined with the change of the shooting angle of the platform (such as unmanned aerial vehicle) and the interference of its own movement, making it difficult to accurately recognize the garbage target by relying only on single frame or single dimension feature.

[0005] Existing marine garbage recognition methods are mostly based on single frame image or static scene assumption, lacking the ability to dynamically model the target and understand the global background. Such methods are difficult to effectively capture the temporal motion features and spatial context association of garbage targets, resulting in insufficient recognition accuracy and robustness in actual complex marine environments.

[0006] Therefore, in the process of marine garbage object recognition, the current technology still has problems such as the inability to effectively model the global semantic information of the marine scene and the local garbage detail features, the weak recognition ability of multi-scale floating garbage, and the insufficient spatio-temporal modeling ability, and therefore it is urgent to propose a new marine garbage recognition method that has both local target focusing ability and global scene understanding ability to improve the adaptability and generalization performance of the model in complex open sea environments. SUMMARY

[0007] In view of the prior art defects in the background art, the present application proposes a marine garbage object recognition method fusing global scene and local target features, acquires a marine remote sensing video dataset; constructs a garbage object recognition model, extracts spatiotemporal features of the video by a spatiotemporal feature extraction module, uses a three-dimensional convolutional neural network (3D CNN) as a processing unit, simultaneously models spatial structure and temporal dynamic information of the video, and extracts spatiotemporal features of the video; extracts multi-scale spatial details by a multi-scale target feature extraction module, and acquires local target features by using convolution kernels of different receptive fields; extracts global scene features by a global marine scene feature extraction module, combines long sequence modeling capability of a state space model and context modeling advantages of a self-attention mechanism, and extracts global scene features; automatically learns attention weights of local target features and global environment features by a global scene and local target feature adaptive fusion module, and realizes dynamic fusion of local structural features and global scene features; and performs marine garbage recognition based on the trained garbage object recognition model, and outputs a recognition result.

[0008] The technical scheme adopted by the present application is as follows: A marine garbage object recognition method fusing global scene and local target features, characterized in that it comprises the following steps: Step S1, collecting marine remote sensing video data and constructing a dataset, and then performing preprocessing; Step S2, constructing a marine garbage object recognition model to extract features, comprising a spatiotemporal feature extraction module, a high-low multi-scale target feature extraction module, a global scene feature extraction module, and a global scene and local target feature adaptive fusion module; Step S3, training the marine garbage object recognition model by using a loss function, and updating parameters of the marine garbage object recognition model by a back propagation algorithm; Step S4, inputting the marine remote sensing video data to be recognized after preprocessing into the trained marine garbage object recognition model, and outputting garbage categories or state information related thereto.

[0009] In a possible implementation manner, in the step S1, the specific process of collecting the marine remote sensing video dataset and performing preprocessing is as follows: The marine remote sensing video data is collected and a dataset is constructed, and the marine remote sensing video dataset is preprocessed, comprising: dividing the marine remote sensing video data into n equal-length segments by using a sparse sampling strategy, and randomly sampling one frame from each segment to obtain n video frames of an RGB type, performing random size scaling, multi-scale cropping and horizontal flip data enhancement operations on the video frames, uniformly adjusting pixel resolution, and preparing for subsequent extraction of marine remote sensing video features.

[0010] In a possible implementation manner, in the step S2, the specific process of extracting spatiotemporal features by the spatiotemporal feature extraction module is as follows: After frame sampling and data enhancement of ocean remote sensing video data, a frame set F in RGB format is obtained, which is then processed using 3D CNN to obtain a two-dimensional image; the two-dimensional image is divided into blocks and expanded to obtain a one-dimensional feature map, and the expanded one-dimensional feature map is positionally embedded and normalized to obtain spatiotemporal features. , the specific formula is:

[0011]

[0012]

[0013] In the formula It indicates the frame set in RGB format after frame sampling and data enhancement, 3 indicates RGB three channels, Indicates the number of sampling frames, and Indicates the width and height of the image; Represents 3D convolution; Represents the two-dimensional image obtained after 3D CNN processing, which is equivalent to the image being processed according to Size is divided into image blocks; Function represents the dimension reshaping function, which is used to expand the two-dimensional spatial features into one-dimensional features; The function represents the position embedding function, which embeds the one-dimensional feature into a position to retain the spatial information; Represents the position feature of the expanded position embedding, where ; Representation layer normalization operation, Represents the spatiotemporal characteristics of the output; Among them, the specific formula of position embedding is expressed as:

[0014] In the formula represents the features after the input dimension is reshaped, represents the learnable projection matrix, Linear projection to size vector of Represents the classification head, is an image No. piece, represents the position embedding vector using a normal distribution.

[0015] In a possible implementation, in step S2, the specific process of extracting local target features by the high- and low-altitude multi-scale target feature extraction module is as follows: pass The transformation operation transforms spatiotemporal features transformed into two-dimensional spatial features, and then one-dimensional time convolution is used in the time dimension to model the time sequence to obtain intermediate features , and then four different convolution branches are used to obtain branch features Finally, the branch features of the four branches are spliced in the channel dimension through operation, and the branch features are unfolded in the spatial dimension to one dimension through operation to meet the input requirements of the subsequent module, and finally the local target features are obtained , which is expressed by the formula:

[0016] , wherein represents the input spatiotemporal features, wherein represents the number of frames, represents the number of feature channels, represents the length of one-dimensional features; function represents a dimension reshaping function; represents one-dimensional time convolution; represents the intermediate features after and operations; , represents the first branch feature obtained by four different convolution branches, wherein and are the width and height of the feature, respectively; represents two-dimensional convolution with a convolution kernel size of 1x1, and represent depth separable convolution with a convolution kernel size of 3x3 and 5x5, respectively; represents the maximum pooling operation with a convolution kernel size of 3x3; represents the channel dimension splicing operation, which splices the features of all four branches in the channel dimension; represents the local target feature, wherein .

[0017] In one possible implementation, the global scene feature extraction module in step S2 includes a visual Mamba module and a multi-layer perception module, and the specific process of extracting the global scene feature is as follows: (1) Visual Mamba module First, the local target features are normalized to obtain a sequence, and then they are linearly projected into vector and vector ; then ​​Process: for each direction, first one-dimensional convolution is performed, and then forward and backward SSMs are used to obtain forward long-distance features and backward long-distance features ; finally, the is obtained after passing through the activation function, and after being multiplied by and , the forward features and the backward features are obtained, respectively, and then and are added and passed through the linear projection layer and the residual to obtain the residual features , the specific formula is as follows:

[0018] wherein, represents the input sequence; represents the layer normalization operation; represents the linear projection layer for projecting the feature dimension from C to D; represents the intermediate feature in the branch after linear projection, which is used for subsequent forward and backward one-dimensional convolution and SSM processing; represents the intermediate feature in the branch after linear projection, which is used for the gating activation operation; and represent the forward and backward SSM state space models, respectively; and represent the forward and backward one-dimensional convolution in the feature dimension, respectively; represents the forward long-distance feature output by the SSM, represents the backward long-distance feature output by the SSM; represents the linear unit activation function; represents the element-wise multiplication operation; represents the processed forward feature and represents the processed backward feature; represents the residual feature obtained after passing through the linear projection layer and the residual; (2) Multi-layer perceptron module The inner product is obtained by multiplying the query and the transpose of the key, and then divided by the square root of the number of attention heads , and then weighted summed by the Softmax function and the value V to obtain the output result of the single-head attention mechanism, the calculation formula is as follows: ​​​

[0019] Where, 、 、 Represents query, key and value respectively, represents the number of attention heads, represents the normalization function, Represents the calculation function of the single-head attention mechanism; Compared with the single-head attention mechanism, the multi-head attention mechanism enables the model to focus on information from different feature subspaces at different locations, thereby obtaining more diverse and rich feature representations; The residual features Input into the multi-head self-attention mechanism for processing to obtain global scene features The specific formula is expressed as:

[0020]

[0021] Where, , , Respectively represent the projection of input features into queries, keys, and values ​​in the attention mechanism; Indicates the Output of single-head attention; represents the projection matrix; Indicates the number of attention heads; Represents a splicing operation; Represents the calculation function of the multi-head self-attention mechanism.

[0022] In a possible implementation, in step S2, the global target feature adaptive fusion module fuses the local target features and global scene features The specific process is as follows: Use softmax attention to selectively fuse local target features from high- and low-altitude multi-scale target feature extraction module branches and the global scene features from the global scene feature extraction module branch ;use Convolution and layer normalization achieve feature channel alignment, the formula is expressed as:

[0023] In the formula is a 1×1 convolution, is the layer normalization operation, is the local feature after alignment, is the global feature after alignment, The fusion feature after splicing the two features; Use global average pooling to generate channel-dimensional statistical features, and then learn the adaptive selection weights through the linear layer. The specific formula is:

[0024] In the formula is the global average pooling, is a linear layer with ReLU activation function, is the attention weight vector, where is the vector dimension; Use on channel dimension The function generates selection factors and adaptively selects appropriate features from the fusion features; use The channel attention mechanism of the module is adaptively calibrated to eliminate convolution modules and The module's features are misplaced; finally, a The layer enables local features and global features to fully interact, complement each other, and enhance each other, thereby improving feature modeling capabilities. The specific formula is as follows:

[0025] In the formula is the fully connected layer, To pass The fused features after attention selection, is the channel attention mechanism, is a linear layer with ReLU activation function, is the final output feature.

[0026] In a possible implementation, in step S3, the specific calculation formula of the loss function is as follows:

[0027] In the formula is the true category distribution, is the category distribution predicted by the model, is the sample index, is the category index, is the total number of categories, is the number of samples input for training, is the loss value.

[0028] To solve the above problems, the present invention also provides an electronic device comprising a memory and a processor, wherein: the memory is used to store a program; the processor is coupled to the memory and is used to execute the program to implement the steps of the above-mentioned marine debris identification method.

[0029] In addition, the application also provides a computer readable storage medium for storing a set of instructions, which can be executed by a processor to enable a device to perform the operations in any of the above-mentioned marine garbage object identification methods.

[0030] Compared with the prior art, the application provides a marine garbage object identification method that fuses global scene and local target features, and has the following beneficial effects: The application proposes an identification method that fuses global scene and local target features for the complex detection scene of garbage target in the marine environment, significantly improves the model identification performance and scene adaptability. On the one hand, a visual Mamba global modeling module is designed, which ingeniously combines the characteristics of state space model (SSM) for efficiently processing long sequence dependence and the advantages of attention mechanism for accurately capturing key information, breaks through the limitations of traditional convolution for capturing long-distance feature association, and effectively extracts long-term dependence information in a complex marine background. On the other hand, a high-low multi-scale feature extraction module is introduced, which captures the detailed features of garbage targets of different scales through 4 differentiated convolution branches, solves the problem of "multi-scale target missed detection and false detection" of traditional single-scale extraction, and enhances the detailed capture ability of multi-scale garbage targets. Finally, through the feature fusion module, the attention mechanism is used, 1x1 convolution and layer normalization are used to realize feature channel alignment, and then global average pooling and a linear layer are used to learn dynamic weights, dynamically adjust the contribution of local and global features, improve the identification accuracy and robustness of the model in a complex open sea, and provide more reliable technical support for marine garbage monitoring and ecological protection. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of the marine garbage object identification method proposed by the application that fuses global scene and local target features is shown in the figure. Figure 2 A framework diagram of the marine garbage object identification model provided by the application that fuses global scene and local target features is shown in the figure. Figure 3 A structure diagram of the spatio-temporal feature extraction module of the identification model provided by the application is shown in the figure. Figure 4 A structure diagram of the high-low multi-scale target feature extraction module of the identification model provided by the application is shown in the figure. Figure 5 A structure diagram of the global scene and local target feature adaptive fusion module (GSLTAF module) of the identification model provided by the application is shown in the figure. DETAILED DESCRIPTION

[0032] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be further described in detail. It should be explained that the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of the present application.

[0033] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings. In order to make the above-mentioned purposes, features and advantages of the present application clearer, the advantages of the present application will be further described by comparing embodiments with reference to the accompanying drawings and specific embodiments.

[0034] In the first embodiment of the present application, the present application proposes an ocean garbage object recognition method fusing global scene and local target features, the flow chart of the method is as shown in Figure 1 The steps of the method will be described in detail as follows: Step S1, collect ocean remote sensing video data and construct a data set, and then perform preprocessing; In the present embodiment, in the step S1, the specific process of collecting ocean remote sensing video data and constructing a data set, and then performing preprocessing is as follows: (1) Collect ocean remote sensing video data set The ocean remote sensing video data set is collected, including multiple categories of different marine environments and multiple dynamic garbage events under different perspectives, i.e., the ocean environment garbage recognition data set SeaDronesSee and MARVID, and is divided into a training set, a validation set and a test set according to a ratio of 6:1:3, which are respectively used for model training, validation and performance evaluation; (2) Preprocess the ocean remote sensing video data set The ocean remote sensing video data set is preprocessed, including: using the sparse sampling strategy of TSN (Temporal Segment Network) to divide the ocean remote sensing video data set into n equal-length segments, and randomly sampling one frame from each segment to obtain n video frames of the RGB type; in the experiment, n=8, the original image resolution is 640x640 pixels, after random size scaling, multi-scale cropping and horizontal flipping and other data enhancement operations, it is uniformly adjusted to 224x224 pixels, which is used for subsequent extraction of ocean remote sensing video features.

[0035] Step S2, construct an ocean garbage object recognition model for feature extraction, the model structure is as shown in Figure 2As shown, it comprises: a spatio-temporal feature extraction module, a high-low multi-scale target feature extraction module, a global scene feature extraction module, and a global scene and local target feature adaptive fusion module. In this embodiment, in step S2, the spatio-temporal feature extraction module extracts spatio-temporal features in the following specific process: The structure of the spatio-temporal feature extraction module is as shown in Figure 3 Based on the spatio-temporal feature extraction module, a 3D convolutional neural network (3D CNN) is used as a processing unit to model the spatial structure and temporal dynamic information of the marine remote sensing video dataset and extract the spatio-temporal features of the garbage target in the marine remote sensing video dataset. Compared with static images, marine remote sensing video datasets have both time and space dimensions, and contain key dynamic change information, which is particularly important for detecting marine garbage, which is floating and time-varying. Although some static garbage targets can be detected by image models, due to factors such as shooting angle, light reflection, and wave obstruction, single-frame images are prone to information loss. However, continuous video sequences can provide more stable and complete spatial background and target motion clues, thereby improving the accuracy and robustness of recognition. A single-layer 3D CNN is used to perform preliminary feature extraction on the marine remote sensing video dataset, and the feature dimension is effectively compressed through step control to provide a unified feature input basis for the subsequent modules. After frame sampling and data enhancement on the marine remote sensing video dataset V, a frame set in RGB format is obtained Then, a 3D CNN is used for processing to obtain a two-dimensional image. To meet the input requirements of the subsequent model, the two-dimensional image is divided and expanded to obtain a one-dimensional feature map, i.e., the two-dimensional spatial features are expanded into a one-dimensional feature. The expanded feature map is position-embedded and normalized to obtain spatio-temporal features The specific formula is:

[0036]

[0037]

[0038] In the formula, represents the frame set in RGB format obtained after frame sampling and data enhancement, 3 represents the RGB three channels, represents the number of sampled frames, and represent the width and height of the picture; represents a 3D convolution used to extract spatio-temporal features of the video frame set, and the convolution kernel size is , represents the multiple of image downsampling; Represents 3D CNN features, which is equivalent to converting the image into The size is divided into image blocks, where is the number of feature channels; Function represents the dimension reshaping function, which is used to expand the two-dimensional spatial features into one-dimensional features; The function represents the position embedding function, which embeds the one-dimensional feature into a position to retain the spatial information; Represents the position feature of the expanded position embedding, where ; Representation layer normalization operation, Represents the spatiotemporal characteristics of the output; Among them, the specific formula of position embedding is expressed as:

[0039] In the formula represents the features after the input dimension is reshaped, represents the learnable projection matrix, Linear projection to size vector of Represents the classification head, is an image No. piece, represents the position embedding vector using a normal distribution.

[0040] In this embodiment, the specific process of extracting local target features by the high-low altitude multi-scale target feature extraction module in step S2 is as follows: The structure of the high- and low-altitude multi-scale target feature extraction module is as follows: Figure 4 As shown in the figure, first, the one-dimensional features are restored to two-dimensional features in the spatial dimension to adapt to the two-dimensional spatial convolution, and then the one-dimensional temporal convolution is used in the time dimension to extract the temporal features; depthwise separable convolution is used, combined with convolution and pooling operations of different sizes such as 1×1 convolution, 3×3 convolution, 5×5 convolution, and 3×3 pooling, to extract multi-scale local detail information from the input features while retaining the locality and translation invariance of the features. In addition, the maximum pooling operation is used to improve the robustness of the model to transformations such as translation, scaling, and rotation, and the 1×1 convolution reduces the feature dimension, thereby improving the accuracy and efficiency of the model; pass The transformation operation transforms spatiotemporal features Converted into two-dimensional spatial features through transformation operation, and then used in the time dimension One-dimensional time convolution (convolution kernel size is 3) is used to perform time series modeling , and then obtain through 4 different convolution branches , , and finally through The operation splices the features of the four branches in the channel dimension, and the operation The operation expands the feature map in the spatial dimension to one dimension to meet the input requirements of the subsequent fusion module, and finally obtains the local target feature , which is expressed by the formula:

[0041] wherein, represents the input spatio-temporal feature, wherein represents the number of frames, represents the number of feature channels, represents the one-dimensional feature length; the function represents a dimension reshaping function; represents one-dimensional time convolution; represents the intermediate feature after the operation and operation; , represents the first branch feature obtained by the four different convolution branches, wherein and are the width and height of the feature, respectively; represents two-dimensional convolution with a convolution kernel size of 1x1, and represent depth separable convolution with a convolution kernel size of 3x3 and 5x5, respectively; represents the maximum pooling operation with a convolution kernel size of 3x3; represents the channel dimension splicing operation, which splices the features of all the four branches in the channel dimension; represents the local target feature, wherein .

[0042] In this embodiment, in the step S2, the global scene feature extraction module includes a visual Mamba module and a multilayer perception module, and the specific process of extracting the global scene feature is as follows: The global scene feature extraction module (VMAGM module) includes a visual Mamba module and a multilayer perception (MLP) module, which is used to model the global scene semantic information in the unmanned aerial vehicle aerial video, effectively extracts the long-distance dependent global features in the video sequence by combining the state space modeling capability and the self-attention mechanism, and enhances the representation ability of the model to event features in complex scenes; specifically including: (1) Visual Mamba module First, normalize the local target feature to obtain a sequence, and then linearly project the normalized sequence into a vector and a vector ; then the two directions of forward and backward are processed for each direction, first one-dimensional convolution is performed on , and then and are obtained by forward and backward SSM After activation function, the gating vector is obtained, which is multiplied by and respectively to obtain and , and are added, and then linear projection layer and residual are used to obtain residual feature , the specific formula is as follows:

[0043] wherein, represents the input sequence; represents the layer normalization operation; represents the linear projection layer for projecting the feature dimension from C to D; represents the intermediate feature in the branch after linear projection, which is used for subsequent forward and backward one-dimensional convolution and SSM processing; represents the intermediate feature in the branch after linear projection, which is used for gating activation operation; and represent the forward and backward SSM state space model respectively; and represent the forward and backward one-dimensional convolution in the feature dimension respectively; represents the forward long-distance feature output by SSM, represents the backward long-distance feature output by SSM; represents the linear unit activation function; represents the element-wise multiplication operation; represents the processed forward feature and represents the processed backward feature; represents the residual feature obtained after linear projection layer and residual; (2) Multi-layer perception module The multilayer perception (MLP) module is used for nonlinear activation and dimension transformation of the input sequence feature. Through at least one layer of linear transformation and nonlinear activation function, the feature space is reconstructed and the semantic is enhanced to improve the modeling ability of the model for global scene and time relationship. The multi-head attention module structure is similar to the structure of the Transformer encoder, and the multi-head self-attention mechanism is used. whereby and the inner product is obtained by multiplying the transpose of , divided by the square root of the number of attention heads (to avoid the inner product result being too large, causing the gradient to disappear after Softmax), and then weighted summation is performed with the help of the Softmax function and V, to obtain the result of the single-head attention mechanism output, the calculation formula is:

[0044] In the formula, , , represent query, key and value respectively, represent the number of attention heads, represent the normalization function, represent the single-head attention mechanism calculation function; Compared with the single-head attention mechanism, the multi-head attention mechanism makes the model pay attention to information from different positions and different feature subspaces, so as to obtain more diverse and rich feature representation; The residual feature is input into the multi-head self-attention mechanism for processing, to obtain the global scene feature The specific formula representation is:

[0045]

[0046] In the formula , , represent the projection of the input feature into the query, key and value in the attention mechanism respectively; represent the output of the single-head attention; represent the projection matrix; represent the number of attention heads; represent the concatenation operation; represent the multi-head self-attention mechanism calculation function.

[0047] In the step S2 in the embodiment, the specific process of the global target feature adaptive fusion module fusing the local target feature and the global scene feature is as follows: The structure of the global target feature adaptive fusion module is shown in Figure 5 , which selectively fuses the local target feature from the high-low altitude multi-scale target feature extraction module branch and the global scene feature from the global scene feature extraction module branch using the softmax attention. ;use Convolution and layer normalization achieve feature channel alignment, the formula is expressed as:

[0048] In the formula is a 1×1 convolution, is the layer normalization operation, is the local feature after alignment, is the global feature after alignment, The fusion feature after splicing the two features; To adaptively select local and global features, global average pooling is used to generate channel-dimensional statistical features. Then, the weights of adaptive selection are learned through a linear layer while reducing the channel dimension to achieve higher efficiency. The specific formula is:

[0049] In the formula is the global average pooling, is a linear layer with ReLU activation function, is the attention weight vector, where is the vector dimension; because The function introduces competition between attention weights, thereby obtaining more selective attention weights; therefore, using The function generates selection factors and adaptively selects appropriate features from the fusion features; in order to make full use of context information, use The channel attention mechanism of the module is adaptively calibrated to eliminate convolution modules and The module's features are misplaced; finally, a The layer enables local and global features to fully interact, complement each other, and enhance each other, thereby improving the feature modeling capability. The specific formula is as follows:

[0050] In the formula is the fully connected layer, To pass The fused features after attention selection, is the channel attention mechanism, is a linear layer with ReLU activation function, is the final output feature.

[0051] Step S3, using a loss function to train the marine debris recognition model, and updating the parameters of the marine debris recognition model through a back propagation algorithm; In this embodiment, in step S3, the loss function is as follows: The model is trained by using a loss function, the model parameters are continuously updated by using a back propagation algorithm, the recognition accuracy is optimized, and finally the marine garbage object recognition model is obtained after training; The loss function is a cross-entropy loss function, and the loss function is used for calculating the loss between the real class distribution of the marine remote sensing video and the marine remote sensing video class distribution predicted by the model, and the calculation formula is as follows:

[0052] In the formula, is the real class distribution, is the class distribution predicted by the model, is the sample index, is the class index, is the total number of classes, is the number of training input samples, is the loss value.

[0053] Step S4, input the preprocessed marine remote sensing video data to be recognized into the trained marine garbage object recognition model, output the garbage class or state information related thereto, and realize efficient and accurate marine garbage recognition.

[0054] In the second embodiment of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned unmanned aerial vehicle event recognition method fusing global scene and local target features.

[0055] The present application provides a kind of marine garbage object recognition method fusing global scene and local target features, belong to remote sensing video recognition technical field.Its method includes: obtaining marine remote sensing video dataset;Recognition model is built;Marine remote sensing video data is input into identification model, and video feature is obtained after pre-processing;According to space-time alignment module, use 3D CNN as processing unit, to simultaneously model the spatial structure and time dynamic information of video, effectively extract the space-time feature of garbage target;According to high-low air multi-scale target feature extraction module, different size convolution kernel is used to extract spatial features of different scales, to obtain the local detail information of marine garbage under different scales;Based on global scene feature extraction module, the efficient long sequence processing capacity of state space model and the modeling capacity of self-attention mechanism are combined, to realize the global modeling of background semantics in marine environment;Based on global scene and local garbage target feature adaptive fusion module, the importance of local and global features is automatically weighed according to different marine environment scenes, and the corresponding attention weight is learned, and then the feature information of key garbage area is highlighted, and local and global features are dynamically fused;Finally, based on the fused features, recognition is carried out, to realize efficient and accurate recognition of garbage in marine remote sensing video.

[0056] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application.

[0057] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A marine debris identification method that integrates global scene and local target features, characterized by: include: Step S1, collect ocean remote sensing video data and construct a data set, and then perform preprocessing; Step S2: constructing a marine debris recognition model for feature extraction, including: a spatiotemporal feature extraction module, a high- and low-altitude multi-scale target feature extraction module, a global scene feature extraction module, and a global scene and local target feature adaptive fusion module; Step S3, using a loss function to train the marine debris recognition model, and updating the parameters of the marine debris recognition model through a back propagation algorithm; In step S4, the pre-processed ocean remote sensing video data to be identified is input into the trained ocean garbage identification model, and the garbage category or status information related thereto is output.

2. The marine debris identification method integrating global scene and local target features according to claim 1 is characterized in that: In step S1, the specific process of collecting and preprocessing the ocean remote sensing video dataset is as follows: Marine remote sensing video data is collected and a dataset is constructed. The marine remote sensing video dataset is preprocessed, including: using a sparse sampling strategy to divide the marine remote sensing video data into n segments of equal length, and randomly sampling one frame from each segment to obtain n RGB type video frames. After performing random size scaling, multi-scale cropping and horizontal flipping data enhancement operations, the pixel resolution is uniformly adjusted to prepare for the subsequent extraction of marine remote sensing video features.

3. The method for identifying marine debris by integrating global scene and local target features according to claim 1 is characterized in that: In step S2, the specific process of the spatiotemporal feature extraction module extracting spatiotemporal features is as follows: After frame sampling and data enhancement of the ocean remote sensing video data, a frame set F in RGB format is obtained, which is then processed using 3DCNN to obtain a two-dimensional image; the two-dimensional image is divided into blocks and expanded to obtain a one-dimensional feature map, and the expanded one-dimensional feature map is positionally embedded and normalized to obtain spatiotemporal features. , the specific formula is: ; ； ； In the formula It represents a set of frames in RGB format after frame sampling and data enhancement. 3 represents three RGB channels, T represents the number of sampling frames, and H and W represent the width and height of the image. Represents 3D convolution; Represents the two-dimensional image obtained after 3D CNN processing, which is equivalent to dividing the image into image blocks of size P×P; Function represents the dimension reshaping function, which is used to expand the two-dimensional spatial features into one-dimensional features; The function represents the position embedding function, which embeds the one-dimensional feature into a position to retain the spatial information; Represents the position feature of the expanded position embedding, where ; Representation layer normalization operation, Represents the spatiotemporal characteristics of the output; Among them, the specific formula of position embedding is expressed as: ； In the formula represents the features after the input dimension is reshaped, represents the learnable projection matrix, Linear projection to a vector of size D; Represents the classification head, is an image No. piece, represents the position embedding vector using a normal distribution.

4. The method for identifying marine debris by integrating global scene and local target features according to claim 1 is characterized in that: In step S2, the specific process of extracting local target features by the high-low altitude multi-scale target feature extraction module is as follows: pass The transformation operation transforms spatiotemporal features Converted into two-dimensional spatial features through transformation operation, and then used in the time dimension One-dimensional time convolution performs time series modeling to obtain intermediate features , and then obtain branch features through 4 different convolution branches , , and finally through The operation concatenates the branch features of the four branches in the channel dimension and passes The operation expands the branch features into one dimension in the spatial dimension to meet the input requirements of the subsequent modules, and finally obtains the local target features , the formula is: ; in, Represents the spatiotemporal features of the input, where T represents the number of frames, C represents the number of feature channels, and N represents the one-dimensional feature length; Function represents the dimension reshaping function; represents one-dimensional temporal convolution; Indicates passing and Intermediate features after operation; , Represents the first Branch features, where H and W are the width and height of the feature respectively; represents a two-dimensional convolution with a convolution kernel size of 1×1, and Represents depthwise separable convolution with kernel sizes of 3×3 and 5×5 respectively; Indicates the maximum pooling operation with a convolution kernel size of 3×3; Represents the channel dimension splicing operation, which splices the features of all four branches in the channel dimension; Represents the local target feature, where N=H×W.

5. The method for identifying marine debris by integrating global scene and local target features according to claim 1 is characterized in that: In step S2, the global scene feature extraction module includes a visual Mamba module and a multi-layer perceptron module. The specific process of extracting global scene features is as follows: (1) Visual Mamba Module First, the local target features are normalized to obtain a sequence, and then linearly projected to vectors and vector ; Then from the forward and backward directions Processing: For each direction, first Perform one-dimensional convolution, and then obtain forward long-distance features through forward and backward SSM and backward long-range features ; Finally, pass After the activation function, the gate vector is obtained, which is respectively and After multiplication, the forward features are obtained respectively and backward features , and then and After addition, the residual features are obtained after the linear projection layer and the residual , the specific formula is as follows: ; in, represents the input sequence; Representation layer normalization operation; represents the linear projection layer that projects the feature dimension from C to D; After linear projection Branch intermediate features for subsequent forward and backward one-dimensional convolution and deal with; After linear projection Branch intermediate features, used for gated activation operations; and Represents forward and backward State space models; and Represents the forward and backward one-dimensional convolution in the feature dimension respectively; express The output forward long-distance features, express Output long-distance backward features; represents the linear unit activation function; Represents element-wise multiplication operation; Represents the processed forward features and represents the processed backward features; Represents the residual features obtained after the linear projection layer and the residual; (2) Multilayer Perceptron Module By query and key The inner product is obtained by multiplying the transpose of , divided by the square root of the number of attention heads , and then use the Softmax function and value V to perform weighted summation to obtain the output of the single-head attention mechanism. The calculation formula is: ; Where, 、 、 Represents query, key and value respectively, represents the number of attention heads, represents the normalization function, Represents the calculation function of the single-head attention mechanism; Compared with the single-head attention mechanism, the multi-head attention mechanism enables the model to focus on information from different feature subspaces at different locations, thereby obtaining more diverse and rich feature representations; The residual features Input into the multi-head self-attention mechanism for processing to obtain global scene features The specific formula is expressed as: ; ； Where, , , Respectively represent the projection of input features into queries, keys, and values ​​in the attention mechanism; Indicates the Output of single-head attention; represents the projection matrix; Indicates the number of attention heads; Represents a splicing operation; Represents the calculation function of the multi-head self-attention mechanism.

6. The method for identifying marine debris by integrating global scene and local target features according to claim 1 is characterized in that: In step S2, the global target feature adaptive fusion module fuses the local target features and global scene features The specific process is as follows: Use softmax attention to selectively fuse local target features from high- and low-altitude multi-scale target feature extraction module branches and the global scene features from the global scene feature extraction module branch ;use Convolution and layer normalization achieve feature channel alignment, the formula is expressed as: ; In the formula is a 1×1 convolution, is the layer normalization operation, is the local feature after alignment, is the global feature after alignment, The fusion feature after splicing the two features; Use global average pooling to generate channel-dimensional statistical features, and then learn the adaptive selection weights through the linear layer. The specific formula is: ; In the formula is the global average pooling, is a linear layer with ReLU activation function, is the attention weight vector, where is the vector dimension; Use on channel dimension The function generates selection factors and adaptively selects appropriate features from the fusion features; use The channel attention mechanism of the module is adaptively calibrated to eliminate convolution modules and The module's features are misplaced; finally, a The layer enables local features and global features to fully interact, complement each other, and enhance each other, thereby improving feature modeling capabilities. The specific formula is as follows: ； In the formula is the fully connected layer, To pass The fused features after attention selection, is the channel attention mechanism, is a linear layer with ReLU activation function, is the final output feature.

7. The method for identifying marine debris by integrating global scene and local target features according to claim 1 is characterized in that: In step S3, the specific calculation formula of the loss function is as follows: ； In the formula is the true category distribution, is the category distribution predicted by the model, is the sample index, is the category index, is the total number of categories, is the number of samples input for training, is the loss value.

8. The present invention provides a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned marine debris identification method that integrates global scene and local target features.

Images