An underwater video multi-label classification processing method and system

Abstract

This invention discloses an underwater video multi-label classification processing method and system, relating to the field of video processing technology. The method includes: processing underwater video to obtain original image features, clear image features, and text semantic features; encoding based on a multi-head attention encoder, fusing the original visual feature representation and text feature representation through a cross-modal attention mechanism to generate a multi-modal complete feature representation; using a generative adversarial network, utilizing the multi-modal complete feature representation, original visual feature representation, clear visual feature representation, and text feature representation as self-supervised signals to generate enhanced and completed multi-modal features; constructing a composite objective function, and performing multi-label classification of the underwater video based on the optimized model and the enhanced and completed multi-modal features. The multi-head attention encoder analyzes and processes the original video data, and constructs a self-supervised signal based on clear visual features, improving the accuracy of underwater video feature fusion.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a method and system for underwater video multi-tag classification processing. Background Technology

[0002] This section provides only background information related to this application to enable those skilled in the art to understand this application more thoroughly and accurately, and it is not necessarily prior art.

[0003] Underwater video analysis is one of the main technical means for marine resource exploration and ecological monitoring. However, underwater videos are easily affected by the changes in light caused by water molecules, resulting in poor video quality, such as blurriness, color distortion, and reduced visibility.

[0004] Existing underwater video processing methods mainly improve the robustness of underwater video classification through multimodal information fusion technology. However, current multimodal information fusion technology suffers from insufficient cross-modal semantic correlation, making it difficult to combine semantic operations between various video modalities. In cases of severe distortion of underwater video modalities, the feature reconstruction accuracy of existing underwater video processing methods is insufficient, making it difficult to meet the accuracy requirements of underwater video multi-label classification processing. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art, and proposes an underwater video multi-label classification processing method and system. It analyzes and processes the raw video data through a multi-head attention encoder, and constructs a self-supervised signal based on clear visual features to improve the accuracy of underwater video feature fusion.

[0006] This invention proposes a multi-label underwater video classification method based on multimodal generative adversarial networks, comprising the following steps:

[0007] Step S1: Process the input underwater video to obtain raw image data, clear image data and text data; extract features from the raw image data, clear image data and text data to obtain raw image features, clear image features and text semantic features.

[0008] Step S2: Encode the original image features, clear image features, and text semantic features based on a multi-head attention encoder to generate original visual feature representation, clear visual feature representation, and text feature representation. Then, fuse the original visual feature representation and text feature representation through a cross-modal attention mechanism to generate a multimodal complete feature representation.

[0009] Step S3: Based on the generative adversarial network, the visual features are reconstructed using the multimodal complete feature representation, the original visual feature representation, the clear visual feature representation, and the text feature representation as self-supervised signals to enhance their clarity, and the missing modal features are reconstructed to generate enhanced and completed multimodal features.

[0010] Step S4: Construct a composite objective function that includes classification loss and at least one feature reconstruction-related loss. Use the composite objective function to jointly optimize the model parameters. Based on the optimized model and the enhanced and completed multimodal features, perform multi-label classification on the underwater video.

[0011] Furthermore, step S1 includes:

[0012] A pre-trained ResNet50 neural network is used as a visual core feature extractor. The input underwater video frame sequence is processed by the visual core feature extractor to extract its features and generate original image features and clear image features.

[0013] A pre-trained BERT network model is used to perform semantic analysis on the text description information corresponding to the underwater video, extract the deep semantic information of the text description information and generate text semantic features.

[0014] Furthermore, step S1 also includes:

[0015] The original visual features are extracted based on the original blurred video frame sequence, and the sharp visual features are extracted based on the enhanced sharp video frame sequence after image enhancement processing.

[0016] Furthermore, step S1 also includes:

[0017] The spatiotemporal feature extraction module is used to process continuous video frame sequences to extract features in both spatial and temporal dimensions, thereby obtaining spatiotemporal visual features.

[0018] Image enhancement processing is performed on the original image data by combining spatiotemporal visual features with the original visual features to obtain clear image data.

[0019] Furthermore, step S2 includes:

[0020] The original visual feature representation is generated based on the original visual attention encoder to process the original image features and the feature dependencies within the video frames.

[0021] The text semantic features are processed by a text modality attention encoder, and a text feature representation is generated based on the semantic association of text information.

[0022] A multimodal complete feature representation is constructed by performing cross-attention operations on the text feature representation and the original visual feature representation through a cross-modal attention encoder.

[0023] The sharp image features are processed using a visual self-supervised attention encoder to generate a sharp visual feature representation.

[0024] Furthermore, step S2 also includes:

[0025] In the cross-modal attention encoder, textual features are used as query values, and the original visual features are used as key and output values. Through cross-attention calculation, a modal complementary feature representation dominated by the visual modality is constructed as a multimodal complete feature representation.

[0026] Furthermore, step S2 also includes:

[0027] Based on the cross-modal attention mechanism, textual features are used as query values, and corresponding spatiotemporal regions are queried in spatiotemporal visual features based on the query values;

[0028] The original visual feature representation obtained from the query within the corresponding spatiotemporal region is set as the semantic weight of the text feature representation.

[0029] Furthermore, step S3 includes:

[0030] Based on generative adversarial networks, the multimodal complete feature representation, the original visual feature representation, the clear visual feature representation, and the text feature representation are used as self-supervised signals to reconstruct visual features to enhance their clarity, and to reconstruct missing modal features, thereby generating enhanced and completed multimodal features.

[0031] Furthermore, step S3 also includes:

[0032] A discriminator is used to distinguish between the unimodal feature representation and the multimodal complete feature representation, and adversarial training is used to force the unimodal feature representation to approximate the multimodal complete feature representation.

[0033] And / or, the features generated by the reconstruction or rebuilding are input again into the corresponding encoder for secondary encoding, and the secondary encoded features are cyclically regressed to the multimodal complete feature expression through cyclic consistency loss.

[0034] The present invention also provides an underwater video multi-label classification processing system, the processing system comprising:

[0035] Data preprocessing module: used to process the input underwater video, obtain raw image data, clear image data and text data, and extract features from the raw image data, clear image data and text data to obtain raw image features, clear image features and text semantic features;

[0036] The data fusion module is used to encode the original image features, clear image features, and text semantic features based on a multi-head attention encoder to generate original visual feature representation, clear visual feature representation, and text feature representation. The module then fuses the original visual feature representation, clear visual feature representation, and text feature representation through a cross-modal attention mechanism to generate a multimodal complete feature representation.

[0037] Data analysis module: Based on generative adversarial networks, it uses the multimodal complete feature representation and specific unimodal feature representation as self-supervised signals to reconstruct visual features to enhance their clarity, and to reconstruct missing modal features to generate enhanced and completed multimodal features.

[0038] Data classification module: Construct a composite objective function that includes classification loss and at least one feature reconstruction-related loss, use the composite objective function to jointly optimize the model parameters, and perform multi-label classification on the underwater video based on the optimized model and the enhanced and completed multimodal features.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] By setting up multiple multi-head attention encoding networks, the complex relationships within and between modalities are accurately captured to capture key classification attributes: Based on an attention mechanism with text features as queries and visual features as keys and values, visual information is deeply fused to capture their complementarity and generate a complete attribute feature expression that integrates the essence of both modalities. This can more accurately identify and associate the key attributes corresponding to multiple tags in underwater videos.

[0041] By introducing a cross-modal generative adversarial network, the visual self-supervised feature representation generated by the visual self-supervised attention encoder is used as a self-supervised signal to drive the generator to reconstruct blurred underwater visual features, effectively solving the problem of underwater visual modal distortion and blurring, thereby improving the accuracy of underwater video multi-label classification processing. Attached Figure Description

[0042] Figure 1 This is a flowchart of the underwater video multi-tag classification processing method in an embodiment of the present invention;

[0043] Figure 2 This is a logic block diagram of the underwater video multi-tag classification processing method in this embodiment of the invention;

[0044] Figure 3 This is a schematic diagram of the cascaded residual device structure in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the underwater video multi-tag classification and processing system in an embodiment of the present invention. Detailed Implementation

[0046] To further illustrate the technical means and effects adopted by this application to achieve its intended purpose, the specific implementation methods, structures, features, and effects according to this application are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0047] Example 1:

[0048] Please refer to Figures 1 to 3 This invention provides a method for multi-label classification processing of underwater videos, the method comprising the following steps:

[0049] Step S1: Process the input underwater video to obtain raw image data, clear image data, and text data. Extract features from these data to obtain raw image features, clear image features, and text semantic features. Specifically, the input underwater video can be decomposed into a series of video frames, which constitute the raw image data. To obtain clear image data, various image enhancement techniques can be used to process the raw image data, such as physics-based underwater image restoration algorithms, deep learning-driven image deblurring networks, or color correction methods. Text data can originate from video metadata, manual annotation, or narration extracted from the video through speech recognition. For feature extraction, various pre-trained convolutional neural network (CNN) models can be used to process the image data to obtain raw image features and clear image features. For example, network structures such as VGG, Inception, or EfficientNet can be used. For text data, word embedding models (such as Word2Vec, GloVe) or sequence models such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) can be used to extract text semantic features. These feature extractors transform raw, clear visual and textual semantic information into high-dimensional vector representations, laying the foundation for subsequent modality fusion and classification tasks.

[0050] Step S2: Encode the original image features, sharp image features, and text semantic features using a multi-head attention encoder to generate original visual feature representations, sharp visual feature representations, and text feature representations. Then, fuse the original visual feature representations and text feature representations using a cross-modal attention mechanism to generate a multimodal complete feature representation. Specifically, the multi-head attention encoder can independently process the features of each modality to capture their internal dependencies and important information. For example, for original image features, a dedicated visual attention encoder can be designed to focus on the visual information of different regions in the image to generate original visual feature representations. Similarly, attention encoders can be designed for sharp image features and text semantic features respectively to generate corresponding sharp visual feature representations and text feature representations. Subsequently, to achieve effective fusion between modalities, a cross-modal attention mechanism can be used. For example, the text feature representation can be used as the query, and the original visual feature representation as the key and value. By calculating attention weights to measure the degree of correlation between text information and visual information, and then weighting and fusing the visual features accordingly, a multimodal complete feature representation that can simultaneously reflect visual content and text semantics can be generated. This fusion approach aims to compensate for the deficiencies caused by the degradation of underwater visual information by using textual information to semantically supplement and calibrate the visual information.

[0051] Multimodal complementary attention encoding networks aim to further model the visual and textual features extracted by multimodal feature extraction networks, fully exploring the internal structural information of each modality and the complementary relationships between modalities. To this end, this embodiment constructs four parallel multi-head attention encoding networks, including a primary visual attention encoder, a textual modal attention encoder, a cross-modal attention encoder, and a visual self-supervised attention encoder. The parallel design of the visual and textual modal attention encoders preserves the feature representation of each modality, while the cross-modal attention encoder uses a cross-attention mechanism to achieve multimodal fusion of visual and textual features, forming a more complete and robust feature representation. The calculation formula for the multi-head attention encoding network is as follows:

[0052] ;

[0053] in, () represents the activation function, where Q, K, and V represent the query matrix, value matrix, and key matrix, respectively, and a scaling factor is introduced. Prevent gradient vanishing.

[0054] In the visual modality attention encoder part of the multimodal complementary attention coding network The calculation formula is as follows:

[0055] ;

[0056] Among them, LN This is a layer normalization operation.

[0057] ;

[0058] use Generate the key matrix, value matrix, and query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix, a scaling factor is introduced to prevent gradient vanishing. Among them, the clear visual modality attention encoder The calculation steps are the same as described above, and the final output is a clear visual feature representation. .

[0059] Visual modality attention encoder in cross-modal generative adversarial networks This refers to the reuse of the visual modality attention encoder at different stages in the multimodal complementary attention coding network. The calculation formula is as follows:

[0060] ;

[0061] Among them, LN This is a layer normalization operation.

[0062] ;

[0063] use Generate the key matrix and value matrix. Generate a query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix, a scaling factor is introduced to prevent gradient vanishing. .

[0064] Text Modal Attention Encoder The calculation formula is as follows:

[0065] ;

[0066] Among them, LN This is a layer normalization operation.

[0067] ;

[0068] use Generate the key matrix, value matrix, and query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix, a scaling factor is introduced to prevent gradient vanishing. .

[0069] Cross-modal attention encoder The calculation formula is as follows:

[0070] ;

[0071] Among them, LN This is a layer normalization operation.

[0072] ;

[0073] use Generate the key matrix and value matrix. Generate a query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix, a scaling factor is introduced to prevent gradient vanishing. .

[0074] To generate the final representation of each modality group, a multi-head attention output aggregation strategy is adopted. The computation results of all attention heads are integrated through a concatenation operation to form a unified feature representation.

[0075] ;

[0076] in, Describe a linear projection matrix. Let represent the output of the j-th head of the i-th modality group, and h be the number of heads. The core value of this multi-head architecture lies in integrating multiple attention heads. Its spatial projection mechanism projects the original high-dimensional input data in parallel to multiple independent low-dimensional subspaces. In each subspace, specific types of feature dependencies are handled, capturing the diversity of data from different perspectives and further enhancing the model's learning ability and expressive power.

[0077] Step S3: Based on the Generative Adversarial Network (GAN), using the aforementioned multimodal complete feature representation, original visual feature representation, clear visual feature representation, and text feature representation as self-supervised signals, the visual features are reconstructed to enhance their clarity, and missing modal features are reconstructed to generate enhanced and completed multimodal features. Specifically, the GAN can contain a generator and a discriminator. The generator is designed to receive degraded visual features or incomplete modal features and attempt to generate enhanced visual features or reconstructed missing modal features. The discriminator is trained to distinguish the features output by the generator from real, high-quality features. Through this adversarial training process, the generator is forced to learn how to generate more realistic and clearer visual features, and how to reconstruct missing modal features based on existing modal information. For example, when the original visual features are blurry, the generator can use the multimodal complete feature representation, clear visual feature representation, etc., as references to learn how to reconstruct the original visual features into clearer visual features. When a modality (such as text) is missing, the generator can use the visual feature representation to infer and reconstruct the corresponding text semantic features. Therefore, this step can effectively solve the problems of underwater visual degradation and modality loss, and provide high-quality enhanced and completed multimodal features.

[0078] Cross-modal generative adversarial networks aim to improve multimodal complementary attention encoding networks. The output feature representation is optimized and enhanced. To ensure that the visual features generated by the generator closely approximate the real modality in structure and distribution, a feature inpainting mechanism based on generative adversarial networks is introduced, and a cascaded residual generator is designed accordingly. It is jointly optimized with the feature gradient loss function and the perceptual loss function.

[0079] To address the problem of underwater visual blur, a feature gradient loss function is introduced. To guide the cascaded residual generator Optimizations were made to make the generator repair visual features more focused on high-frequency edge information and local details, improving the contour and texture sharpness of the generated image features, and the feature gradient loss function. The formula is calculated as follows:

[0080] ;

[0081] Where d is the feature dimension. The 1024 features extracted from the real, clear image by a preprocessing network. For cascaded residual generator The generated restored image features, The difference between adjacent dimensions of the feature vector is used to simulate the gradient. Difference between adjacent dimensions of the target feature, The L2 norm is used as a regularization term for the objective function to prevent the model from becoming too complex to fit the training set and thus causing overfitting, thereby improving the model's generalization ability.

[0082] To address the issues of low contrast and color degradation in underwater vision, a perceptual loss function is introduced. To guide the cascaded residual generator Optimization is achieved by making the generator's repaired visual features focus more on high-level semantic features, correcting the true color and structure of the generated image features, and using a perceptual loss function. The formula is calculated as follows:

[0083] ;

[0084] Where d is the feature dimension. For cascaded residual generator The generated restored image features, For target features, It is an L2 norm.

[0085] We introduce the adversarial learning mechanism of generative adversarial networks (GANs), using a discriminator from the GAN to measure the distance between two data distributions. Here, we will... and The discriminator fed into the generative adversarial network and In the identification, with complete modalities As a self-supervised signal, discrimination loss is introduced in this process. The formula for calculating the constraint guidance can be expressed as follows:

[0086] ;

[0087] The goal is to enable the modal features output by the visual and text modal attention encoders. and complete modal features The difference between them is the smallest.

[0088] To address the issue that traditional GAN ​​generators map arbitrary noise into the target space, leading to loss failure, this paper utilizes the original short video modal features generated by three generators, and then encodes the resulting features using a multi-head attention network. , Complete feature representation Adversarial training is conducted to ultimately achieve secondary encoding features. , With complete modal features The approximation alignment makes the modal feature representation after secondary encoding and the complete modal feature representation... The process minimizes the distance between features. It introduces a cycle consistency loss on top of the traditional GAN ​​loss to ensure that the features generated by the generator are consistent with the feature space of the real complete features.

[0089] Step S4: Construct a composite objective function that includes classification loss and at least one feature reconstruction-related loss. Use this composite objective function to jointly optimize the model parameters, and based on the optimized model and the enhanced and completed multimodal features, perform multi-label classification on the underwater video. Specifically, the classification loss can be binary cross-entropy loss or focus loss, etc., to measure the model's performance on the multi-label classification task. The feature reconstruction-related loss can include L1 loss, L2 loss, or perceptual loss, etc., to measure the quality of the generator's reconstructed or rebuilt features. By weighting and combining these loss functions to form a composite objective function, the model can achieve a balance between classification accuracy and feature quality. For example, during training, the model parameters are iteratively adjusted to minimize the composite objective function. This joint optimization strategy ensures that the model not only performs accurate multi-label classification, but also that its internal feature representation is effectively enhanced and completed. Finally, using the optimized model, combined with the enhanced and completed multimodal features generated in step S3, accurate multi-label classification of the input underwater video can be achieved.

[0090] Specifically, in step S2, this application employs a multi-head attention encoder to finely encode the features of each modality, and fuses the original visual feature representation and textual feature representation through a cross-modal attention mechanism to generate a complete multimodal feature representation. This mechanism enables textual semantics to effectively guide the understanding and weighting of visual features, and even when visual information is ambiguous, the discriminative power of features can be strengthened through semantic supplementation of text. Compared with simple feature concatenation or early fusion methods, the cross-modal attention fusion method of this application can more deeply bridge the semantic gap between modalities and effectively solve the problem of inefficiency in cross-modal fusion.

[0091] Furthermore, this application introduces a generative adversarial network (GAN) in step S3, utilizing multimodal complete feature representations as self-supervised signals to reconstruct visual features to enhance clarity and reconstruct missing modal features. In the example above, this means that even if the clownfish in the original video is blurry, the generator can reconstruct it into clearer visual features using clear image features and semantic information from the text description. Simultaneously, when the text description is temporarily missing, the system can also reconstruct the corresponding text semantics based on the visual information. This adversarial reconstruction mechanism significantly improves the model's ability to recover distorted features and enhances its robustness in modal missing situations, effectively solving problems such as the difficulty of reconstructing distorted features and the vulnerability to modal missing features.

[0092] Finally, in step S4, by constructing a composite objective function that includes classification loss and feature reconstruction-related loss for joint optimization, this application ensures a dual improvement in model classification accuracy and feature quality. This end-to-end optimization strategy enables the model to learn more robust and discriminative feature representations as a whole, thereby achieving more accurate multi-label classification in complex underwater degradation scenarios. In summary, this application systematically addresses the challenges of underwater visual degradation through an innovative combination of multimodal feature collaborative processing and adversarial reconstruction mechanisms, significantly improving the performance and robustness of underwater video multi-label classification.

[0093] Specifically, step S1 includes: using a pre-trained ResNet50 neural network as a visual core feature extractor, processing the input underwater video frame sequence through the visual core feature extractor, extracting its features and generating original image features and clear image features; using a pre-trained Bert network model to perform semantic analysis on the text description information corresponding to the underwater video, extracting the deep semantic information of the text description information and generating text semantic features.

[0094] This core visual feature extractor aims to extract discriminative and robust visual features from underwater video frames to address the complexity and degradation of underwater environments. ResNet50, a deep residual network, effectively solves the vanishing gradient problem in deep network training through residual connections, possessing powerful feature extraction capabilities. Pre-training means that the network has learned rich, general visual features on large image datasets (such as ImageNet), enabling it to better adapt to new image recognition tasks. One implementation approach is to directly load the ImageNet pre-trained ResNet50 model and fine-tune it according to the characteristics of underwater videos to better adapt it to the feature distribution of underwater images. Alternatively, other pre-trained convolutional neural network (CNN) architectures, such as VGG, Inception, or EfficientNet, can be chosen. These networks also perform well in image recognition tasks and can serve as foundational models for visual feature extraction.

[0095] The system processes the input underwater video frame sequence, extracts its features, and generates original image features and sharp image features. It acquires visual information from consecutive underwater video frames and distinguishes between original (which may be blurred or degraded) and sharp (which may have undergone some preprocessing or latent) image features, providing a foundation for subsequent multimodal fusion and enhancement.

[0096] Furthermore, frames can be selected from the video frame sequence using a sliding window or fixed-interval sampling method, and then these frames can be input into a visual core feature extractor. Raw image features can be directly extracted from these frames, while sharp image features can be more discriminative feature representations learned from the raw image features during training through some self-supervised or semi-supervised method.

[0097] A pre-trained BERT network model performs semantic analysis on the text description information corresponding to underwater videos, extracting deep semantic information from the text description information and generating text semantic features. This BERT network model is used to transform unstructured text description information into structured, semantically rich feature vectors for fusion with visual features. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture, capable of bidirectionally understanding text context, thereby extracting high-quality deep semantic information and capturing word meaning, syntax, and contextual relationships in the text.

[0098] Furthermore, a pre-trained BERT model is loaded, taking textual descriptions of underwater videos (e.g., scene descriptions, target lists, etc.) as input. The encoder layer of BERT obtains the embedding vectors for each word or the entire sentence, and then generates fixed-length textual semantic features through pooling (e.g., average pooling or max pooling) or using the output of [CLS] tokens. By introducing a pre-trained ResNet50 neural network as the core visual feature extractor and a pre-trained BERT network model for textual semantic analysis, the aim is to improve the quality and robustness of modal features in underwater video multi-label classification tasks from the source.

[0099] Specifically, in this embodiment, when processing the input underwater video, the video frame sequence can first be sampled, for example, extracting 5 frames per second. For these images, a ResNet50 model pre-trained on the ImageNet dataset is used as the visual core feature extractor. Each video frame is scaled to 224x224 pixels and normalized before being input into the ResNet50. The output feature vector of the penultimate layer of the ResNet50 (e.g., before the global average pooling layer) can be used as the original image features. If there are clear video frames processed by image enhancement algorithms (e.g., underwater image dehazing or color correction algorithms), their features are also extracted using the ResNet50 model as clear image features.

[0100] Meanwhile, for textual descriptions of underwater videos, such as "coral reefs, tropical fish, jellyfish," a pre-trained `bert-base-uncased` model can be used for semantic analysis. First, the text description is segmented and encoded using BERT's built-in tokenizer. Then, the encoded text is input into the BERT model, and the embedding vectors corresponding to the `[CLS]` tokens in its output layer are extracted. These embedding vectors serve as the deep semantic features of the textual description, typically with 768 dimensions. This approach ensures that features acquired from both visual and textual modalities possess rich semantic information and strong robustness, providing high-quality input for subsequent multimodal fusion and classification tasks.

[0101] Specifically, step S1 further includes: extracting original visual features based on the original blurred video frame sequence, and extracting clear visual features based on the enhanced clear video frame sequence after image enhancement processing. The original blurred video frame sequence refers to a set of continuous image frames obtained directly from underwater video without any image processing, exhibiting degradation phenomena such as blurriness, low contrast, and color distortion due to the underwater environment. Its function is to directly represent the original visual information of the underwater video, capturing the true degradation characteristics of the underwater environment. This sequence can be obtained by sampling directly from the video stream or by decoding the video file.

[0102] The original visual features are numerical representations extracted from the original blurred video frame sequence, reflecting its visual content and degradation characteristics, providing a foundation for subsequent feature enhancement, reconstruction, and multimodal fusion. These features can be extracted by forward propagation of the original blurred video frame sequence using a convolutional neural network (e.g., a pre-trained ResNet50 neural network); or by extracting texture features using traditional image processing methods such as Local Binary Pattern (LBP) or Gabor filters. The enhanced sharp video frame sequence refers to a set of consecutive image frames whose visual quality is significantly improved after image enhancement processing of the original blurred video frame sequence, with improvements in sharpness, contrast, and color fidelity.

[0103] Specifically, in step S1, the input underwater video is first sampled at a rate of 5 frames per second to obtain an original blurred video frame sequence. Next, a deep learning-based underwater image enhancement model (e.g., a trained Water-GAN model) is applied to each frame of this original blurred video frame sequence to generate a corresponding enhanced sharp video frame sequence. Subsequently, a pre-trained ResNet50 neural network is used as the visual core feature extractor. The original blurred video frame sequence and the enhanced sharp video frame sequence are input into the ResNet50 network respectively to extract the original visual features and the sharp visual features.

[0104] Through the above technical solution, this application can ensure more accurate and robust extraction of visual features in underwater video multi-label classification. By explicitly extracting features based on the original blurred video frame sequence and the enhanced clear video frame sequence, the feature distortion problem caused by underwater visual degradation is effectively solved, providing high-quality and highly discriminative visual information for subsequent multimodal fusion and classification, and significantly improving the accuracy of underwater video multi-label classification and the robustness of the model in complex underwater environments.

[0105] Furthermore, step S1 also includes: using a spatiotemporal feature extraction module to process a continuous video frame sequence to simultaneously extract features in both spatial and temporal dimensions to obtain spatiotemporal visual features; and performing image enhancement processing on the original image data based on the combination of the spatiotemporal visual features and the original visual features to obtain clear image data.

[0106] The spatiotemporal feature extraction module is a processing unit capable of simultaneously capturing spatial (e.g., object shape, texture, and color distribution in an image) and temporal (e.g., object trajectory, speed, deformation, and illumination changes) information from a continuous sequence of video frames. Its function is to overcome the shortcomings of traditional single-frame processing in video analysis by neglecting temporal dynamics, providing more comprehensive contextual information for subsequent image enhancement. As one implementation, this module can employ a 3D convolutional neural network (3D-CNN) to directly learn spatiotemporal features from the video cube by performing convolution operations simultaneously in both spatial and temporal dimensions. Alternatively, a two-stream network structure can be used, with one stream processing spatial information (e.g., using a 2D-CNN) and the other processing temporal information (e.g., using an optical flow network or a recurrent neural network (RNN / LSTM), and then the features from the two streams are fused.

[0107] The spatiotemporal feature extraction module can be specifically a feature extraction network based on 3D-ResNet. This network learns multi-scale spatiotemporal features from a continuous sequence of video frames by stacking multiple 3D convolutional layers and pooling layers. When processing a continuous sequence of video frames, 5 frames per second can be sampled from underwater video, and 16 frames can be selected continuously as an input sequence to form a spatiotemporal cube. The 3D convolutional kernels of 3D-ResNet capture information such as texture and edges of the image in the spatial dimension, and capture dynamic information such as object motion and deformation in the temporal dimension, thereby obtaining a tensor of dimension CxDxHxW as the spatiotemporal visual feature, where C is the number of channels, D is the temporal depth, and H and W are the spatial dimensions. During image enhancement processing, the spatiotemporal visual features extracted by the above 3D-ResNet can be fused with the original visual features extracted from a single frame of the original image by a pre-trained ResNet50 (as the original visual feature extractor) through a feature fusion layer (e.g., an attention module or a simple concatenation followed by a convolutional layer). The fused features are input into an image enhancement generator based on a U-Net structure, which outputs enhanced, sharp image data with the same size as the original image.

[0108] A pre-trained ResNet50 network is used as the backbone for feature extraction to extract features from the input underwater RGB image sequence. Unlike the original structure, this invention replaces the fully connected layer used for classification with a 1024-dimensional feature output layer to output visual features. and Let the input original video frame sequence be... ,in Indicates the first The visual feature extraction process for a frame of RGB image is as follows:

[0109] ;

[0110] ;

[0111] ;

[0112] in, To remove the original classification layer of the ResNet50 convolutional backbone network, It is an eigenvector. It is the output of global average pooling of features. It is a weight matrix. It is a bias vector. It is a 1024-dimensional original visual feature.

[0113] The text branch is used to process scene information descriptions corresponding to the video. It uses a pre-trained BERT network as the feature extraction backbone to extract 1024-dimensional semantic features from the input text. Let the input text be described as ,in For the token, the text feature extraction process is as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] in, For word embedding, position embedding, and paragraph embedding, Here, H is the BERT encoder, H is the context-aware token vector, and s is the average pooled semantic vector. It is the dimensional projection weight. For bias vectors, It is a 1024-dimensional text feature.

[0119] Through the above technical solutions, the underwater video processing method proposed in this invention can effectively overcome the problem of insufficient enhancement effect caused by the neglect of temporal continuity and dynamic changes in underwater videos. By utilizing the temporal context information of the video, the model can more accurately identify and correct degradation phenomena such as blurring and distortion caused by the underwater environment, thereby generating higher-quality, clearer image data. This significantly improves the robustness and effectiveness of underwater video image enhancement, provides more reliable visual input for subsequent multi-label classification, and thus improves the overall classification accuracy.

[0120] Specifically, step S2 includes: processing the original image features based on the original visual attention encoder, generating the original visual feature representation based on the feature dependencies within the video frame; processing the text semantic features through the text modality attention encoder, generating the text feature representation based on the semantic association of the text information; performing cross-attention operations on the text feature representation and the original visual feature representation through the cross-modality attention encoder to construct a multimodal complete feature representation; and processing the clear image features based on the visual self-supervised attention encoder to generate a clear visual feature representation.

[0121] The original visual attention encoder is a neural network module specifically designed to process visual modal features. Its core function is to capture the internal correlations of input visual features through an attention mechanism. This encoder can be a Transformer encoder layer based on a self-attention mechanism, extracting long-distance dependencies by calculating the correlation between each element in the feature sequence and all other elements; or it can be a hybrid structure combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The CNN is used to extract local spatial features, while the RNN or its variants (such as LSTM and GRU) are used to capture temporal dependencies, further enhanced by an attention mechanism to strengthen the features of keyframes or regions. Its role is to focus on the intrinsic structural relationships of visual features under underwater visual degradation conditions, avoiding feature distortion caused by external interference, thereby enhancing the stability and discriminative power of the original visual features.

[0122] In step S2, the original visual attention encoder can be a network structure composed of multiple stacked Transformer encoder layers, where each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. This encoder receives original image features as input and captures spatial and temporal dependencies within video frames through a self-attention mechanism. For example, a 3D self-attention mechanism can be used to simultaneously process feature associations in both spatial and temporal dimensions, thereby generating the original visual feature representation. The text modality attention encoder can employ the encoder part of a pre-trained BERT model, receiving text semantic features as input. Through its internal multi-head self-attention mechanism and feedforward neural network, it deeply analyzes word associations and contextual information in the text to generate text feature representations. The cross-modal attention encoder can be a cross-attention layer of a Transformer architecture. In this layer, text feature representations serve as queries, while original visual feature representations serve as both keys and values. By calculating the similarity between the query and the key, attention weights are generated. These weights are then applied to the value, thereby aggregating visual information highly relevant to the text semantics into the text features, or integrating text information into visual features, ultimately constructing a multimodal complete feature representation. A visual self-supervised attention encoder can be a Transformer encoder layer with a structure similar to the original visual attention encoder, but its training objective may be related to sharpness restoration or contrastive learning. This encoder receives sharp image features as input, extracts their internal features through a self-attention mechanism, and generates sharp visual feature representations, providing a high-quality reference for subsequent self-supervised learning and feature enhancement.

[0123] Furthermore, step S2 also includes: using textual feature representation as query value and original visual feature representation as key value and output value in the cross-modal attention encoder, and constructing a modal complementary feature representation dominated by visual modality as a multimodal complete feature representation through cross-attention calculation.

[0124] A cross-modal attention encoder is a neural network module used to fuse information from different modalities (e.g., visual and textual). Its role is to generate a feature representation that integrates multimodal information by calculating the correlations between features from different modalities. This encoder can be implemented based on the cross-attention mechanism in the Transformer architecture, or constructed using gated attention networks, bidirectional attention networks, and other methods to effectively capture complementary and correlated information between modalities. In the attention mechanism, the query value represents the intent or context for the information that needs to be focused on or extracted. By refining the settings of query values, key values, and output values ​​in the cross-modal attention encoder, multimodal feature fusion with visual modality as the primary driver is achieved. This significantly improves the discriminative power and robustness of multimodal complete feature representations in complex underwater environments, providing a more stable and reliable feature foundation for subsequent underwater video multi-label classification.

[0125] Specifically, in a cross-modal attention encoder, a Transformer-based cross-attention layer can be used to implement the above mechanism. First, the textual feature representation is passed through a linear transformation layer to generate a query vector Q. Simultaneously, the original visual feature representation is passed through two independent linear transformation layers to generate a key vector K and an output vector V. Then, the dot product of the query vector Q and the key vector K is calculated and scaled (e.g., divided by the square root of the key vector dimension), and then the Softmax function is applied to obtain the attention weights. Finally, these attention weights are weighted and summed with the output vector V to obtain a visual modality-dominant complementary feature representation. For example, a multi-head attention mechanism can be used, where the above calculations are performed in parallel multiple times, and the outputs of different heads are concatenated and integrated through a linear transformation layer to capture richer intermodal relationships. This implementation ensures that textual semantic information effectively guides the extraction and fusion of visual features while maintaining the dominance of visual information, thereby generating a more discriminative multimodal complete feature representation.

[0126] Specifically, step S2 further includes: using the text feature expression as the query value based on the cross-modal attention mechanism, and querying the corresponding spatiotemporal region in the spatiotemporal visual features based on the query value; setting the original visual feature expression obtained from the query in the corresponding spatiotemporal region as the semantic weight of the text feature expression.

[0127] Cross-modal attention mechanisms, which use textual features as query values, are a mechanism for establishing associations between different modalities (text and vision), where textual features are used as active query signals. Cross-modal attention mechanisms can dynamically capture dependencies between data from different modalities, thereby achieving effective information fusion. Using textual features as query values ​​means that the model, guided by the semantic information of the text, actively "searches" for or "focuses on" related content in the visual modality.

[0128] Furthermore, the process of querying the corresponding spatiotemporal region in the spatiotemporal visual features based on the query value describes the process of accurately locating and extracting information in the visual modality using the text query value. Spatiotemporal visual features contain rich information about the video frame sequence in both time and space dimensions, reflecting the dynamic changes and local details of the video content. Querying the corresponding spatiotemporal region in the spatiotemporal visual features based on the text query value aims to identify the specific time period and spatial location in the video that is most relevant to the semantic description of the text.

[0129] Based on this, the original visual feature expressions obtained from the query within the corresponding spatiotemporal region are set as the semantic weights of the text feature expressions. This aims to dynamically adjust or strengthen the semantic contribution of the text feature expressions through visual information. After querying the spatiotemporal regions corresponding to the text semantics, the original visual feature expressions extracted from these regions are used as the semantic weights of the text feature expressions, enabling the text feature expressions to better reflect their importance and relevance in the visual context, thereby enhancing the accuracy and robustness of cross-modal fusion.

[0130] Specifically, in a cross-modal attention encoder, a multi-head self-attention mechanism can be used to implement the above process. First, the textual feature representation is used as the query vector Q. Then, spatiotemporal visual features (e.g., features extracted from a video frame sequence via a 3D convolutional network or a spatiotemporal Transformer encoder) are used as the source of the key vector K and value vector V. By calculating the similarity (e.g., dot product) between the query vector Q and the key vector K, and normalizing it using the Softmax function, an attention weight matrix is ​​obtained. This matrix indicates the distribution of textual semantics in the spatiotemporal visual features, i.e., the corresponding spatiotemporal region has been queried. Next, based on this attention weight matrix, the original visual feature representation is weighted and summed to obtain a visual context feature highly correlated with the textual semantics. Finally, this visual context feature is fused with the original textual feature representation, for example, through a gating mechanism where the visual context feature acts as a gating signal to dynamically adjust the various dimensions of the textual feature representation, thereby setting the original visual feature representation obtained from the query as the semantic weight of the textual feature representation. This fusion method allows the textual feature representation to be guided and strengthened by visual information, thus more accurately reflecting the video content.

[0131] Specifically, step S3 includes: based on the generative adversarial network, using the multimodal complete feature representation, the original visual feature representation, the clear visual feature representation, and the text feature representation as self-supervised signals, reconstructing the visual features to enhance their clarity, and reconstructing the missing modal features to generate enhanced and completed multimodal features.

[0132] Specifically, a generator based on the U-Net architecture can be used to perform the reconstruction of visual features and the reconstruction of missing modal features. This generator can take the original visual feature representation as input and combine it with multimodal complete feature representation and text feature representation as conditional information.

[0133] Furthermore, the generator can contain an encoder-decoder structure, where the encoder progressively extracts abstract representations of the input features, and the decoder uses these abstract representations and skip connections to generate high-resolution reconstructed features.

[0134] The discriminator can employ a PatchGAN structure, which can discriminate local regions of features, thereby better capturing the local realism of features. During training, various loss functions can be designed to guide the generator and discriminator. For visual feature reconstruction, pixel-level L1 or L2 loss can be introduced to ensure that the reconstructed visual features are as numerically close as possible to the clear visual feature representation.

[0135] Furthermore, perceptual loss can be introduced, which uses features extracted by pre-trained deep neural networks (such as VGG networks) to measure the semantic similarity between reconstructed features and clear features, thereby making the reconstruction results more visually natural and clearer.

[0136] For the reconstruction of missing modal features, cross-entropy loss or mean squared error loss can be used to ensure that the reconstructed features are semantically and distributionally consistent with the features of the true missing modalities. Furthermore, adversarial loss is essential; it forces the generator to generate true features that are difficult for the discriminator to distinguish through an adversarial game between the generator and the discriminator. For example, the generator can be trained to minimize `log(1-D(G(z)))`, while the discriminator is trained to minimize `log(D(x))+log(1-D(G(z)))`, where `D` is the discriminator, `G` is the generator, `x` is the true feature, and `z` is the generator's input. Through the joint optimization of these loss functions, the model can effectively learn the complex mapping relationships of underwater features, thereby generating high-quality augmented and completed multimodal features.

[0137] Specifically, step S3 further includes: using a discriminator to distinguish between unimodal feature representations and multimodal complete feature representations, and using adversarial training to force unimodal feature representations to approximate multimodal complete feature representations; and / or, inputting the reconstructed or rebuilt features back into the corresponding encoder for secondary encoding, and using cyclic consistency loss to make the secondary encoded features cyclically regress to multimodal complete feature representations.

[0138] In the process of feature reconstruction and rebuilding using generative adversarial networks (GANs), the following strategy can be adopted: First, a discriminator is constructed. This discriminator can be a neural network containing three fully connected (FC) layers with the LeakyReLU activation function. The last layer uses the Sigmoid function to output a probability value between 0 and 1. The discriminator receives unimodal feature representations (e.g., original visual feature representations) and multimodal complete feature representations as inputs and attempts to distinguish them. During the adversarial training phase, the generator (i.e., the module responsible for reconstructing or rebuilding features) aims to generate unimodal features that can deceive the discriminator, causing it to be misclassified as a multimodal complete feature representation. The discriminator's loss function can be a binary cross-entropy loss, while the generator's loss function can be designed as the negative log-likelihood of the discriminator's output. Second, to further ensure the cycle consistency of features, the features reconstructed or rebuilt in step S3 (e.g., enhanced visual features) can be input again into their corresponding encoder (e.g., the original visual attention encoder) for secondary encoding. Then, the L1 distance between the secondary encoded features and the multimodal complete feature representation is calculated as the cycle consistency loss. By minimizing this loss, the features after secondary encoding can be forced to maintain a high degree of consistency with the multimodal complete feature representation. In actual training, adversarial loss and cycle consistency loss can be optimized simultaneously to achieve feature enhancement, completion, and consistency assurance.

[0139] Furthermore, the technical solution provided in this invention effectively solves the problems of inefficient cross-modal fusion, difficulty in reconstructing distorted features, and vulnerability to modality loss caused by visual modality degradation in underwater video multi-label classification. Specifically, the adversarial training mechanism forces the alignment of single-modal feature representations with multimodal complete feature representations, significantly improving the accuracy and robustness of feature reconstruction. This allows the model to generate high-quality enhanced and completed multimodal features even in complex and degraded underwater environments. Simultaneously, the introduction of cycle consistency loss ensures that the reconstructed or rebuilt features remain highly consistent with the multimodal complete feature representation after secondary encoding, effectively preventing feature drift and enhancing the model's adaptability to modality loss or degradation. These improvements work together to make the final feature representation used for classification more stable, accurate, and discriminative, thereby significantly improving the accuracy and robustness of underwater video multi-label classification.

[0140] Example 2:

[0141] Please refer to Figure 4 The present invention also provides an underwater video multi-label classification and processing system, which effectively addresses the challenges posed by complex and degraded underwater environments by integrating multimodal feature processing and adversarial reconstruction mechanisms.

[0142] The processing system includes:

[0143] Data preprocessing module 10: This module processes the input underwater video to obtain raw image data, clear image data, and text data. It then extracts features from these data to obtain raw image features, clear image features, and text semantic features. Specifically, raw image data directly reflects the underwater degradation state, clear image data provides enhanced reference, and text data supplements semantic information. These three elements work together to ensure the comprehensiveness of feature extraction under visual distortion conditions, thereby mitigating the vulnerability to modality loss.

[0144] Data fusion module 20: This module encodes the original image features, clear image features, and text semantic features based on a multi-head attention encoder, generating original visual feature representations, clear visual feature representations, and text feature representations. Further, this module fuses these feature representations through a cross-modal attention mechanism to generate a multimodal complete feature representation. The solution in this application uses a multi-head attention encoder to handle dependencies within different modalities and combines a cross-modal attention mechanism to bridge the semantic gap between visual and textual information. This allows textual semantics to effectively guide the understanding and weighting of visual features, enhancing feature discriminative power even when visual information is ambiguous, thereby solving the problem of inefficient cross-modal fusion.

[0145] Data Analysis Module 30: Based on Generative Adversarial Networks (GANs), this module uses multimodal complete feature representations and specific unimodal feature representations as self-supervised signals to reconstruct visual features to enhance their clarity and reconstruct missing modal features, generating enhanced and completed multimodal features. Specifically, the generator of the GAN receives degraded features and attempts to generate enhanced features. The discriminator distinguishes between real and generated features. Adversarial training forces unimodal features to approximate the complete representation, while a cycle consistency mechanism ensures the stability of feature reconstruction. For example, when the original visual features are blurry, the generator uses multimodal complete feature representations and clear visual feature representations as references to learn to reconstruct degraded features into clearer visual features. When text data is missing, the generator reconstructs the corresponding text semantic features based on the visual feature representation, thus effectively addressing the difficulties in reconstructing distorted features and the vulnerability to modal missing features.

[0146] Data Classification Module 40: Constructs a composite objective function comprising classification loss and at least one feature reconstruction-related loss. This composite objective function is used to jointly optimize the model parameters. Based on the optimized model and the enhanced and completed multimodal features, it performs multi-label classification of underwater videos. The classification loss measures the difference between the predicted and true labels, while the feature reconstruction-related loss supervises the feature quality in the generative adversarial network. The weighted combination forms the composite objective function, ensuring a balance between classification accuracy and feature quality. After joint optimization, the model can perform accurate multi-label classification of underwater videos based on the enhanced and completed multimodal features.

[0147] By combining a multimodal feature fusion mechanism with generative adversarial network (GAN) reconstruction technology in a joint optimization manner, this approach effectively bridges the semantic gap between modalities caused by underwater visual degradation and achieves adaptive feature enhancement and completion, significantly improving the accuracy and robustness of underwater video multi-label classification. Compared to basic solutions, this application offers advantages such as enhanced cross-modal semantic association, improved degradation feature reconstruction capabilities, and improved robustness against modal loss. For example, in coral reef monitoring scenarios, even if the original video is blurred due to the underwater environment, the system can still reconstruct visual features through semantic supplementation of textual descriptions, accurately identifying multiple labels such as "coral reef" and "fish school," providing more reliable technical support for efficient marine resource exploration and precise ecological environment monitoring.

[0148] Example 3:

[0149] This invention was tested on the standard underwater video multi-label dataset BCE300, which contains 40,000 underwater videos, over 100,000 original images extracted at equal intervals from the videos and their corresponding clear images, scene text descriptions for each video segment, and 1-4 labels assigned to each video, covering 30 categories of underwater organisms and environments, such as "coral reef," "fish school," and "shipwreck." 80% of the dataset was used as the training set, and 20% as the test set. The experimental results are shown in Table 1, which compares the experimental results of this invention with several classic classification models to highlight the effectiveness of this invention. To ensure the consistency and realism of the experimental results, the same training and test sets were used in all experiments, and the input modalities and parameters of all comparison methods were fine-tuned to match the underwater video dataset used in this paper. The experimental results were evaluated on five metrics. The experimental results show that this invention outperforms representative methods on multiple classification metrics, demonstrating its superiority in solving the underwater video multi-label classification problem.

[0150] Table 1: Comparison of multi-label classification performance of the classification processing method of the present invention and conventional underwater video processing methods on the BCE300 dataset.

[0151] method CO↓ RL↓ OE↓ HL↓ mAP↑ GoogleNet 5.0438 0.4631 0.4810 0.0215 0.6032 C3D 4.5836 0.3982 0.4012 0.0203 0.6961 RMSL 5.0115 0.1032 0.3367 0.0241 0.7763 C2AE 4.3093 0.1211 0.3571 0.0196 0.7948 CGL 9.4514 0.2964 0.2230 0.0183 0.7019 GLOCAL 4.7490 0.1391 0.3022 0.0844 0.7426 MDFS 3.4361 0.0968 0.3760 0.0766 0.7443 DRML 2.9840 0.0542 0.2141 0.0329 0.8047 MG-Net 2.7830 0.0469 0.2033 0.0312 0.8591

[0152] To verify the effectiveness of multimodal fusion, the visual and textual modal classification results were compared between single-modal and visual / textual multimodal classifications in this experiment. The results are shown in Table 2. The results demonstrate the superiority of multimodal over single-modal classification in multi-label classification, thus verifying the effectiveness of the multimodal fusion method of this invention.

[0153] Table 2: Comparison of the effectiveness of multimodal fusion

[0154] Modal CO↓ RL↓ OE↓ HL↓ mAP↑ Visual 4.3430 0.1844 0.3087 0.0870 0.8001 text 8.7696 0.4421 0.4623 0.1466 0.6141 Visual + Text 2.7830 0.0469 0.2033 0.0312 0.8591

[0155] Visual modalities are usually not missing in underwater videos and contain the richest information. However, targets may be lost during the process of extracting images from videos at equal intervals. Therefore, in this experiment, the training set was not processed before training the model. Then, 10%, 20%, 30%, 40%, and 50% of the video text modalities in the test set were randomly set to 0 to verify the anti-modal loss ability of the present invention. The experimental results are shown in Table 3. The results show that the present invention can effectively resist modal loss.

[0156] Table 3: Comparison of model performance under different proportions of missing modalities in the training set

[0157] Modal missing ratio CO↓ RL↓ OE↓ HL↓ mAP↑ 10% 2.984 0.0501 0.2541 0.0337 0.8543 20% 3.183 0.0507 0.2591 0.0340 0.8501 30% 3.221 0.0516 0.2611 0.0355 0.8436 40% 3.380 0.0519 0.2654 0.0364 0.8341 50% 3.411 0.0524 0.2699 0.0370 0.8283

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-label underwater video classification method based on multimodal generative adversarial networks, characterized in that, Includes the following steps: Step S1: Process the input underwater video to obtain raw image data, clear image data and text data; extract features from the raw image data, clear image data and text data to obtain raw image features, clear image features and text semantic features. Step S2: Encode the original image features, clear image features, and text semantic features based on a multi-head attention encoder to generate original visual feature representation, clear visual feature representation, and text feature representation. Then, fuse the original visual feature representation and text feature representation through a cross-modal attention mechanism to generate a multimodal complete feature representation. Step S3: Based on the generative adversarial network, the visual features are reconstructed using the multimodal complete feature representation, the original visual feature representation, the clear visual feature representation, and the text feature representation as self-supervised signals to enhance their clarity, and the missing modal features are reconstructed to generate enhanced and completed multimodal features. Step S4: Construct a composite objective function that includes classification loss and at least one feature reconstruction-related loss. Use the composite objective function to jointly optimize the model parameters. Based on the optimized model and the enhanced and completed multimodal features, perform multi-label classification on the underwater video.

2. The underwater video multi-label classification method according to claim 1, characterized in that, Step S1 includes: A pre-trained ResNet50 neural network is used as a visual core feature extractor. The input underwater video frame sequence is processed by the visual core feature extractor to extract its features and generate original image features and clear image features. A pre-trained BERT network model is used to perform semantic analysis on the text description information corresponding to the underwater video, extract the deep semantic information of the text description information and generate text semantic features.

3. The underwater video multi-label classification method according to claim 2, characterized in that, Step S1 further includes: The original visual features are extracted based on the original blurred video frame sequence, and the sharp visual features are extracted based on the enhanced sharp video frame sequence after image enhancement processing.

4. The underwater video multi-label classification method according to claim 3, characterized in that, Step S1 further includes: The spatiotemporal feature extraction module is used to process continuous video frame sequences to extract features in both spatial and temporal dimensions, thereby obtaining spatiotemporal visual features. Image enhancement processing is performed on the original image data by combining spatiotemporal visual features with the original visual features to obtain clear image data.

5. The underwater video multi-label classification method according to claim 1, characterized in that, Step S2 includes: The original visual feature representation is generated based on the original visual attention encoder to process the original image features and the feature dependencies within the video frames. The text semantic features are processed by a text modality attention encoder, and a text feature representation is generated based on the semantic association of text information. A multimodal complete feature representation is constructed by performing cross-attention operations on the text feature representation and the original visual feature representation through a cross-modal attention encoder. The sharp image features are processed using a visual self-supervised attention encoder to generate a sharp visual feature representation.

6. The underwater video multi-label classification method according to claim 5, characterized in that, Step S2 further includes: In the cross-modal attention encoder, textual features are used as query values, and the original visual features are used as key and output values. Through cross-attention calculation, a modal complementary feature representation dominated by the visual modality is constructed as a multimodal complete feature representation.

7. The underwater video multi-label classification method according to claim 6, characterized in that, Step S2 further includes: Based on the cross-modal attention mechanism, textual features are used as query values, and corresponding spatiotemporal regions are queried in spatiotemporal visual features based on the query values; The original visual feature representation obtained from the query within the corresponding spatiotemporal region is set as the semantic weight of the text feature representation.

8. The underwater video multi-label classification method according to claim 1, characterized in that, Step S3 includes: Based on generative adversarial networks, the multimodal complete feature representation, the original visual feature representation, the clear visual feature representation, and the text feature representation are used as self-supervised signals to reconstruct visual features to enhance their clarity, and to reconstruct missing modal features, thereby generating enhanced and completed multimodal features.

9. The underwater video multi-label classification method according to claim 8, characterized in that, Step S3 further includes: A discriminator is used to distinguish between the unimodal feature representation and the multimodal complete feature representation, and adversarial training is used to force the unimodal feature representation to approximate the multimodal complete feature representation. And / or, the features generated by the reconstruction or rebuilding are input again into the corresponding encoder for secondary encoding, and the secondary encoded features are cyclically regressed to the multimodal complete feature expression through cyclic consistency loss.

10. An underwater video multi-tag classification and processing system, characterized in that, The processing system includes: Data preprocessing module: used to process the input underwater video, obtain raw image data, clear image data and text data, and extract features from the raw image data, clear image data and text data to obtain raw image features, clear image features and text semantic features; The data fusion module is used to encode the original image features, clear image features, and text semantic features based on a multi-head attention encoder to generate original visual feature representation, clear visual feature representation, and text feature representation. The module then fuses the original visual feature representation, clear visual feature representation, and text feature representation through a cross-modal attention mechanism to generate a multimodal complete feature representation. Data analysis module: Based on generative adversarial networks, it uses the multimodal complete feature representation and specific unimodal feature representation as self-supervised signals to reconstruct visual features to enhance their clarity, and to reconstruct missing modal features to generate enhanced and completed multimodal features. Data classification module: Construct a composite objective function that includes classification loss and at least one feature reconstruction-related loss, use the composite objective function to jointly optimize the model parameters, and perform multi-label classification on the underwater video based on the optimized model and the enhanced and completed multimodal features.

Images