Underwater image caption generation method and system based on multimodal information fusion

Abstract

The present invention discloses a method and system for generating underwater image subtitles based on multimodal information fusion. First, Faster R-CNN is used to extract multi-scale image features of underwater images, including full-image features and regional features, to capture the scene and salient target information of the underwater image. Subsequently, the CLIP model is used to generate text word embedding codes related to the underwater image content, and multi-level clustering is performed through K-means to extract hierarchical text features, and the hierarchical structure of the text information is further analyzed. Next, a fusion method based on a multi-head attention mechanism is adopted to effectively fuse image features and text features, thereby enhancing the model's ability to understand underwater images. Finally, the fused multimodal features are input into a Transformer-based image subtitle generator to generate underwater image subtitles related to the image content and context. This method can effectively improve the accuracy and robustness of underwater image subtitle generation and has strong practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater image subtitle generation, and particularly relates to a method and system for generating underwater image subtitles based on multimodal information fusion. Background Art

[0002] The underwater space is a vast and mysterious environment, home to a diverse array of organisms and abundant natural resources. Currently, many advanced technologies are being employed in underwater photography and surveying, including optical imaging, laser imaging, and acoustic sensing. Underwater optical imaging is the most widely used due to its high resolution and ability to capture detailed image information. Underwater cameras aboard underwater robots (AUVs) or unmanned vessels (UAVs) can capture vast quantities of underwater optical images. However, interpreting and analyzing these images of ocean scenes requires extensive expertise and resources, particularly in specialized fields such as marine resource exploration, marine ecological protection, and coral reef monitoring. To address these needs, underwater image captioning technology can be used to describe and analyze underwater images. This technology automatically generates accurate and detailed descriptions of ocean scenes and objects using artificial intelligence algorithms, assisting professionals in marine environmental monitoring and resource management. This method for automatically generating descriptive captions can be embedded in underwater exploration equipment or applied to batch image analysis, improving efficiency while reducing reliance on extensive manual analysis. Therefore, it provides a more efficient and scalable solution for underwater exploration and facilitates the transition from visual perception to semantic understanding in underwater image analysis.

[0003] Underwater image captioning systems take underwater optical images as input and output a set of grammatically correct and fluent sentences describing the image scene and content. This is a typical example of a cross-modal information processing system. Currently, most systems employ an image encoder and a language generator. The image encoder extracts features that effectively represent the information in the image, while the language generator integrates the information from the image encoder and outputs a sentence describing the image content based on the contextual semantics. Specific implementation methods fall into the following categories, with research focusing on how to more accurately and comprehensively represent image features and generate semantically complete sentences.

[0004] The limitations of existing underwater image captioning techniques are, on the one hand, insufficient attention to the effective representation and feature extraction of text, and on the other hand, insufficient fusion of text features and image features.

[0005] (1) Insufficient consideration of text features

[0006] Underwater image captioning is a typical cross-modal task, involving generating textual descriptions from images. However, traditional models often focus excessively on optimizing image feature extraction, such as global and regional features based on convolutional neural networks, which can improve image representation, while neglecting the role of textual features in enhancing model performance. This limitation reduces the quality of generated descriptions and compromises the expressiveness of the text in the generated captions. Therefore, effectively extracting textual features to improve underwater image captioning methods presents a significant challenge.

[0007] (2) Insufficient fusion of image and text features

[0008] After extracting text features, achieving deep fusion with image features to enhance description capabilities remains a key challenge. Currently, most underwater image captioning models rely on sequence generators such as the Long Short-Term Memory (LSTM) model for text generation; however, they exhibit significant limitations in processing multimodal information and struggle to fully capture the deep correlations between text and image features. Therefore, developing an efficient cross-modal alignment mechanism and designing a flexible feature fusion module are crucial for improving underwater image captioning performance. Summary of the Invention

[0009] In order to solve the technical problems existing in the background technology, the present invention aims to provide a method and system for generating underwater image subtitles based on multimodal information fusion.

[0010] In order to solve the technical problem, the technical solution of the present invention is:

[0011] A method for generating underwater image captions based on multimodal information fusion, the method comprising:

[0012] Multi-scale image feature extraction: Underwater images are acquired and a faster region-based convolutional neural network (Faster R-CNN) is used to extract multi-scale image features, including full-image features and regional features.

[0013] Hierarchical text feature extraction: Utilize the Contrastive Language-Image Pre-Training (CLIP) model based on contrastive learning to perform text word embedding encoding, obtain semantic information associated with image content, and use K-means clustering for hierarchical text feature extraction to analyze data distribution and extract text features at different levels.

[0014] Multimodal information fusion: Construct a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features and text features of different scales to obtain fused features;

[0015] Caption generation: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant.

[0016] Furthermore, in the image feature extraction stage, Faster R-CNN is used for image feature extraction, with the pre-trained Residual Network (ResNet101) as the backbone network; multi-scale image features expand the global and local information expression of underwater images from both full-image features and regional features;

[0017] Assume that a three-channel true color underwater image I is input. ResNet101 is a typical convolutional neural network containing a residual module. It uses different types of convolution kernels to perform a series of convolution operations. It outputs a full-image feature map F, which represents the scene information of the underwater image.

[0018] F=ResNet101(I). (1)

[0019] Faster R-CNN uses the Region Proposal Network (RPN) to generate N proposed target regions, represented as {r1, r2, ..., r N}, it predicts whether each position contains an object and calculates the offset of the bounding box of the object. Region of Interest Pooling (RoI Pooling) is used to process regions of interest of inputs of different sizes. It outputs N regional feature vectors labeled R, corresponding to N proposed objects, as shown below:

[0020] R=RoIpooling(r1,r2,…,r N ). (2)

[0021] Finally, a fully connected network is used to combine bounding box regression and softmax function to predict the bounding box and category of the candidate object. The multi-scale image information provides support for word sequence prediction in the underwater image caption generator.

[0022] Furthermore, the text feature extraction stage specifically includes:

[0023] CLIP-based word embedding encoding:

[0024] The text annotated in the underwater image caption dataset is segmented and counted to construct a corresponding vocabulary. The words in the vocabulary are then embedded using the pre-trained CLIP model to generate word vectors related to the image content. The shape of the word vectors is [N×512], where N represents the number of words in the vocabulary and 512 is the dimension of the word embedding vector output by the CLIP model.

[0025] Generate text clustering features:

[0026] Use the K-means algorithm to cluster the word embedding vectors generated by CLIP. First, perform low-level clustering to cluster the word vectors of all words to obtain low-level cluster centers. The shape of the clustering result is [N1×512], where N1 represents the number of low-level cluster centers.

[0027] Generate high-level clusters:

[0028] Based on the vector set of low-level cluster centers, high-level clustering is performed; the low-level cluster center vectors are further clustered to obtain high-level cluster centers. The shape of the clustering result is [N2×512], where N2 represents the number of high-level cluster centers;

[0029] Multi-level clustering is called "low-level clustering" and "high-level clustering", and "hierarchical text features" are obtained.

[0030] Text feature representation of hierarchical cluster centers:

[0031] The cluster centers represent samples similar to them, so each cluster center serves as a classification feature of these words, enhancing the expression of text semantics. Low-level cluster centers represent more detailed semantic categories, while high-level cluster centers represent more abstract and broad categories.

[0032] Information Fusion:

[0033] The text features of low-level clusters and high-level clusters are gradually fused with the features of underwater images to provide rich semantic support for the subsequent image caption generation task; through the above steps, the word embedding generated by CLIP and the two-level clustering method based on the K-means algorithm effectively extract the hierarchical text features of underwater image captions, enhancing the multimodal information association between text and image.

[0034] Furthermore, the multimodal information fusion stage specifically includes:

[0035] Assume that at time step t of the underwater image caption generation model, the full image feature F t and regional characteristics R t =[r1, r2, ..., r n], extracted by target detection, and then concatenated to form a combined image representation, which is represented as V t , where V t Aggregates multi-scale image information and is defined as:

[0036] V t =[F t , R t ]. (3)

[0037] In the multi-head attention module, by t And the output of the j-th level text clustering module, representing the extracted text feature vector C j , the attention mechanism is applied to realize feature fusion, and the attention weight is calculated as:

[0038]

[0039] Where W q and W k It is the query and key value matrix that realizes linear dimensional transformation in the self-attention mechanism, and the parameter d k Yes (W k C j ) dimension, parameter α t Represents the attention weight, which quantifies the V of the combined full-image features and regional features t and text clustering feature C j the correlation between

[0040] Next, the weighted text clustering features are calculated using the following formula:

[0041]

[0042] In the formula Represents weighted text clustering features, W v represents the linear transformation matrix;

[0043] Then, and V t Perform residual connection and then perform layer normalization. The formula is:

[0044]

[0045] The function LayerNorm(·) represents the layer normalization operation, which is based on the learnable parameters of all samples to perform scaling and offset, X t It is the output of the layer normalization operation. Layer normalization stabilizes the output distribution by normalizing the activation function output values ​​within each batch, thereby accelerating the training process and enhancing the generalization ability of the model.

[0046] X tIt is fed into a feedforward neural network, then added to itself and normalized by performing layer normalization;

[0047] Fusion vt =LayerNorm(X t +FFN(X t )), (7)

[0048] Fusion vt The FFN(·) function represents the two-layer feedforward network operation, which is a supplement to the self-attention mechanism and enhances the expression ability of the model. Finally, the output fusion feature Fusion vt Sent to a Transformer-based underwater image caption generator to predict the next word in the sentence sequence;

[0049] Through two-level clustering, we obtained the text feature vectors of two groups of cluster centers. In the process of information fusion, we combined them step by step according to the global attention mechanism fusion mode to obtain a vector that integrates more feature information. We adopted a hierarchical information fusion strategy. The higher-level text clustering features were fused with the image features of the previous layer, mainly focusing on identifying the main categories of words. The lower-level text clustering features were fused in the later layers to express more concrete information, and finally the fusion features of image and text were obtained.

[0050] Furthermore, the model training in the subtitle generation stage specifically includes:

[0051] Assume that the underwater image caption generation model generates the description y of image I as:

[0052] y=(y1,y2,...,y T ) (8)

[0053] where y i is the i-th word generated, T is the length of the description;

[0054] The training process of the underwater image caption generator includes two stages: cross-entropy optimization and optimization of CIDEr, an evaluation index for the similarity between image description text and reference values. In the cross-entropy optimization stage, the cross-entropy loss function is used to quantify the model's ability to predict word sequences, which is defined as:

[0055]

[0056] where y t is the output of the generator at time t, y 1：t-1 is the output of the generator before time t, given the underwater image caption generation model parameter θ, symbol p θ Indicates that the output of the generator is yt , and the previous word is y 1：t-1 The probability distribution of

[0057] The generator is trained by minimizing the cross-entropy loss to generate sentences similar to the reference sentences. When the cross-entropy optimization reaches a certain level and overfitting or training stagnation occurs, reinforcement learning technology is further applied to directly optimize the value of the evaluation parameter CIDEr of the generated text;

[0058] The Self-Critical Sequence Training (SCST) method is used to optimize the model based on the expected reward. The loss function of reinforcement learning is defined as:

[0059]

[0060] The reward function r(·) is the CIDEr parameter value, which is used to measure the consistency between the description generated by the model and the reference text. By maximizing the mathematical expectation of the CIDEr parameter, the model's ability to generate high-quality underwater image captions is further improved.

[0061] A system for generating underwater image captions based on multimodal information fusion, the system being applied to any of the above methods, comprising:

[0062] Image feature extraction module: Acquire underwater images and use Faster R-CNN to extract multi-scale image features, including full-image features and regional features;

[0063] Text feature extraction module: Use CLIP to perform text word embedding encoding to obtain semantic information associated with image content, use K-means clustering to perform hierarchical text feature extraction, analyze data distribution, and extract text features at different levels;

[0064] Multimodal information fusion module: Constructs a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features of different scales and hierarchical text features to obtain fused features;

[0065] Caption generation module: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant.

[0066] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned methods for generating underwater image subtitles based on multimodal information fusion is implemented.

[0067] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any one of the above-mentioned methods for generating underwater image subtitles based on multimodal information fusion.

[0068] Compared with the prior art, the advantages of the present invention are:

[0069] (1) A novel method is developed to extract hierarchical text features to enhance the ability of underwater image caption generation. The CLIP model pre-trained with language images based on contrastive learning is used to obtain word embedding encodings whose semantic context is consistent with the underwater image content. This method helps to generate captions that closely correspond to the image content. In addition, a hierarchical clustering method is implemented to extract hierarchical text features that capture typical underwater objects, scenes, and activities in the text. This achieves more accurate text prediction by leveraging semantic similarity and promotes a deeper understanding of the contextual relationships between underwater image descriptions.

[0070] (2) A multi-head attention-based fusion method for multimodal information (i.e., image and text features) is developed, which effectively integrates full-image and regional features with hierarchical text clustering features. By adopting an attention-based weight distribution in the underwater image captioning task, it focuses on salient objects and their related semantic text features. As a result, this method can generate more accurate and grammatically correct underwater image captions. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 , the technical roadmap of the underwater image caption generation method based on multimodal information fusion of the present invention;

[0072] Figure 2 ,Multi-scale image feature extraction diagram based on Faster RCNN;

[0073] Figure 3 , CLIP word embedding encoding and hierarchical text clustering graph of the text in the corresponding dictionary of underwater image captions;

[0074] Figure 4 ,Multimodal information fusion diagram of image and text features based on multi-head attention mechanism;

[0075] Figure 5 ,Multi-level image-text feature fusion graph based on attention mechanism;

[0076] Figure 6 ,Underwater image detection visualization example based on Faster R-CNN model;

[0077] Figure 7,Content words and two-level cluster center graph of the underwater image caption dataset vocabulary;

[0078] Figure 8 , a graph showing the impact of the number of high-level text clustering centers on model performance. DETAILED DESCRIPTION

[0079] The specific implementation of the present invention is described below in conjunction with examples:

[0080] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0081] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0082] Example 1:

[0083] like Figure 1 As shown, this embodiment provides an overall framework of an underwater image caption generation method based on multimodal information fusion;

[0084] First, in the image feature extraction stage, Faster R-CNN is used to extract multi-scale image features of underwater images, including global and regional features. These features effectively capture information about the scene and salient objects in the image. Simultaneously, in the text feature extraction stage, the CLIP model based on contrastive learning is used to obtain word embeddings associated with the image content. Subsequently, a K-means clustering algorithm is used to analyze the data distribution and extract hierarchical text features related to the text. A method based on a multi-head attention mechanism is then developed to effectively fuse the multimodal information of the extracted image and text features. Finally, these features are input into a Transformer-based language generator to generate contextually relevant image captions for underwater images.

[0085] 1. Multi-scale image feature extraction

[0086] Faster R-CNN has become one of the popular solutions for real-time object detection tasks due to its balance of high accuracy and reasonable speed. For the underwater image captioning task, Faster R-CNN is used for image feature extraction, with the pre-trained ResNet101 as the backbone network. The parameters in the first few layers show good generalization, enabling it to process the basic features of underwater images, while the latter layers are fine-tuned on the underwater image captioning dataset to capture more detailed, task-specific features. These extracted features provide rich image information to support the underwater image captioning task. Multi-scale image features expand the global and local information expression of underwater images from both full-image features and regional features. The specific process is as follows Figure 2 As shown:

[0087] Assume that a three-channel true color underwater image I is input. ResNet101 is a typical CNN with a residual module, which uses different types of convolution kernels to perform a series of convolution operations. It outputs a full-image feature map F, which represents the scene information of the underwater image, as shown in the figure:

[0088] F=ResNet101(I). (1)

[0089] Faster R-CNN uses the Region Proposal Network (RPN) to generate N proposed target regions, represented as {r1, r2, ..., r N}. It predicts whether each location contains an object and calculates the offset of the bounding box of the object. Region of Interest (RoI) pooling is used to process regions of interest of inputs of different sizes. It outputs N region feature vectors labeled R, corresponding to N proposed objects, as shown below

[0090] R=RoIpooling(r1,r2,...,r N ). (2)

[0091] The final fully connected network, accompanied by bounding box regression and softmax function, predicts bounding box candidate objects and object categories. All of this multi-scale image information can be used in the underwater image caption generation task, enabling it to be fused and used to predict word sequences in the underwater image caption generator.

[0092] In summary, this module not only extracts full-scale features from underwater images to represent scene information, but also obtains corresponding regional features through a region proposal network. These multi-scale image features provide a rich image representation for subsequent multimodal information fusion, thereby enhancing the underwater image caption generation model's ability to understand complex underwater environments.

[0093] 2. Hierarchical text feature extraction

[0094] Word embedding text feature representation based on CLIP:

[0095] Encoding textual information is crucial for underwater image captioning. Word embedding technology maps words into a continuous, high-dimensional space, generating dense vector representations that facilitate semantic understanding. To more effectively extract textual features and establish relationships between text and images, the CLIP model was developed. Trained on a large dataset of images and their corresponding textual descriptions, CLIP effectively captures the semantic connections between text and images, significantly enhancing image understanding and text generation capabilities.

[0096] Underwater image captioning faces complex visual environments and unique imaging conditions, placing higher demands on the model's ability to understand multimodal information. Given CLIP's strengths in semantic extraction and cross-modal features, the model performs remarkably well in addressing the challenges of underwater image captioning. Therefore, using word embeddings generated by CLIP as text features can effectively improve the model's performance in underwater image captioning tasks.

[0097] Hierarchical text feature extraction based on two-level clustering:

[0098] Regional features derived from underwater imagery correspond to objects and categories present in the image content. When integrating information from text, it is important to not only consider contextual encoding but also to add category-specific information, which reflects higher-level abstract information in the text data. In the absence of predefined text classification labels, unsupervised clustering methods can be used to generate a sample set of word vectors for content words, with cluster centers representing the categorical features of semantically similar words. A K-means-based algorithm is implemented to perform this process.

[0099] During the clustering process, it is not easy to accurately determine the number of cluster centers. An inappropriate number may result in unclear boundaries between clusters. Single-level clustering may not capture the complexity of the data. In order to achieve more reasonable sample clustering, multi-level clustering can be used to obtain better text clustering features, further reduce the differences within the clusters, thereby improving the final clustering quality, and revealing complex patterns in the data set, such as nested clusters, sub-cluster structures, etc. Lower-level clustering may produce broader categories, while higher-level clustering can help find more representative subgroups in each cluster. This can make the final clustering results easier to interpret.

[0100] Use Figure 3In the two-level K-means clustering operation shown in Figure 1, the first level clusters the word embedding vectors of all words in the vocabulary, obtaining N1 cluster centers, called low-level cluster centers, denoted as C1. The second level clusters the vector set of the first-level cluster centers, obtaining N2 cluster centers, called high-level cluster centers, denoted as C2.

[0101] The cluster center represents samples similar to it, so it can be used as a classification feature for these words. When the present invention uses two-level clustering, it enhances the hierarchical relationship in semantic expression and makes the text information more comprehensive. They will be used for the fusion of image and text information.

[0102] 3. Multimodal Information Fusion Based on Multi-Head Attention Mechanism

[0103] The multi-head attention mechanism performs well in modeling long-term dependencies and capturing subtle contextual changes. It can focus on multiple aspects of the input of things at the same time, thereby enhancing the ability to model complex dependencies within the data, thereby achieving better performance in generation tasks. Therefore, in underwater image caption generation, the Transformer model is used to replace the traditional LSTM to improve the efficiency and accuracy of caption generation. However, relying solely on image features may not be enough to fully understand the complex underwater situation. However, relying solely on visual features may not be enough to fully understand the complex underwater scenes. Therefore, a multimodal information fusion method based on a multi-head attention mechanism is proposed, which effectively fuses full-image features, regional features, and hierarchically clustered text features. This method enhances the model's ability to understand and describe underwater images by making full use of the complementarity of different modalities. The overall fusion process is as follows Figure 4 As shown in Figure 2, it demonstrates how to fuse textual and visual features using a multi-head attention mechanism.

[0104] Assume that at time step t of the underwater image caption generation model, the full image feature is F t , regional characteristics R t =[r1, r2, ..., r n ], extract regional features through target detection, and then connect them to form a unified image representation. This combination is represented by V t , where V tt Aggregates multi-scale image information and is defined as:

[0105] V t =[F t , R t ]. (3)

[0106] In the multi-head attention module, by t and the output of the j-th layer word clustering module (representing the extracted text feature vector C j) Apply the attention mechanism to achieve feature fusion. The attention weight is calculated as

[0107]

[0108] Where W q and W k d is the query and key matrix that implements linear dimensionality transformation in the self-attention mechanism. k Yes (W k C j ) dimension. Parameter α t Represents the attention weight, which quantifies the combined full image and regional features V t and text clustering feature C j The correlation between them.

[0109] Next, the weighted text clustering features are calculated using the following formula

[0110]

[0111] In the formula Represents weighted text clustering features, W v Represents a linear transformation matrix

[0112] Then, and V t Perform residual connection and then perform layer normalization, the formula is

[0113]

[0114] The function LayerNorm(·) represents the layer normalization operation, which is based on the learnable parameters of all samples to perform scaling and offset, X t is the output of the layer normalization operation. Layer normalization stabilizes the output distribution by normalizing the activation function output values ​​within each batch, thereby accelerating the training process and enhancing the generalization ability of the model.

[0115] X t It is fed into a feedforward neural network, then added to itself and normalized by performing layer normalization, as shown in the figure

[0116] Fusion vt =LayerNorm(X t +FFN(X t )), (7)

[0117] Fusion vt The FFN(·) function represents the two-layer feedforward network operation, which is a supplement to the self-attention mechanism and can enhance the expression ability of the model. vtinto a Transformer-based generator to predict the next word in a sentence sequence.

[0118] Two-level clustering is performed to obtain the text feature vectors of two groups of cluster centers. In the process of information fusion, they can be sequentially integrated according to the global attention mechanism fusion mode to obtain a vector that integrates more feature information, such as Figure 5 shown

[0119] Inspired by the human cognitive process of moving from general to specific reasoning, the proposed method employs a multi-level information fusion strategy. Higher-level text clustering features are fused with image features from the previous layer to focus on identifying the main categories of words, while lower-level text clustering features are fused in subsequent layers to convey more concrete information.

[0120] 4. Training based on reinforcement learning

[0121] Assume that the underwater image caption generation model generates the description y of image I as:

[0122] y=(y1,y2,...,y T ) (8)

[0123] where y i is the i-th word generated, and T is the length of the description.

[0124] The training process of the underwater image caption generator includes two stages: cross-entropy optimization and optimization of CIDEr, an evaluation index for the similarity between image description text and reference values. In the cross-entropy optimization stage, the cross-entropy loss function is used to quantify the model's ability to predict word sequences, which is defined as:

[0125]

[0126] where y t is the output of the generator at time t, y 1：t-1 is the output of the generator before time t. Given the underwater image caption generation model parameter θ, symbol p θ Indicates that the output of the generator is y t , and the previous word is y 1：t-1 The probability distribution of .

[0127] The generator is trained by minimizing the cross-entropy loss to generate sentences similar to the reference sentences. When the cross-entropy optimization reaches a certain level and overfitting or training stagnation occurs, reinforcement learning technology is further applied to directly optimize the value of the evaluation parameter CIDEr of the generated text; CIDEr is an evaluation parameter widely used to evaluate image descriptions, which is mainly used to evaluate the similarity between generated sentences and annotations.

[0128] The Self-Critical Sequence Training (SCST) method is used to optimize the model based on the expected reward. The loss function of reinforcement learning is defined as:

[0129]

[0130] The reward function r(·) is the CIDEr parameter value, which is used to measure the consistency between the description generated by the model and the annotation. By maximizing the mathematical expectation of the CIDEr parameter, the model's ability to generate high-quality underwater image captions is further improved.

[0131] Example 2:

[0132] The second embodiment is applied to the first embodiment. The implementation process of the technical solution of this embodiment specifically includes:

[0133] Dataset

[0134] Experiments were conducted on a newly annotated underwater image captioning dataset based on the Underwater Image Enhancement Benchmark (UIEB). Each image was manually annotated with five different English sentences. The 915 underwater images in the dataset are divided into 765 training samples, 90 validation samples, and 60 test samples. The images in the UIEB dataset contain a variety of real underwater scenes with a rich variety of target categories, including divers, different types of fish and corals, sea turtles, coral reefs, shipwrecks, antiques, and stone statues. This diversity makes it particularly valuable, ensuring the diversity and variability of training samples.

[0135] The underwater image captioning dataset contains a total of 49,278 words, of which 1,469 words are unique. These unique words constitute the candidate vocabulary for caption generation. Figure 7 The statistical distribution of the 20 most frequently occurring words is presented. Notably, these high-frequency words are primarily related to the marine environment and underwater activities, including terms such as "sea," "fish," "coral," "diver," "coral reef," and "rock." Furthermore, a part-of-speech analysis was performed on the words in the dataset, and the approximate percentage distribution of each part-of-speech category is shown in Table 1. The most frequently occurring categories include nouns (NOUN), proper nouns (PROPN), verbs (VERB), adjectives (ADJ), and adverbs (ADV), which form the core lexical components of sentence structure. Furthermore, function words such as determiners (DET) and adverbs (ADP) are also present, contributing to the grammatical structure of the text.

[0136] Table 1 - Percentage of Parts of Speech Categories

[0137]

[0138] Experimental setup

[0139] The experiments were conducted on a system equipped with an Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz, 128GB RAM, and an NVIDIA GeForce RTX 2080Ti GPU. The operating system used was Ubuntu 20.04, PyTorch 1.0.4 was used as the deep learning framework, and CUDA 10.1 was used for GPU acceleration.

[0140] During the training process of the developed underwater image caption generation model, the model parameters were set as follows: the dimension of the global and regional feature vectors was 2048, while the dimension of the word embedding vector based on contrastive learning was 512. In the two-level text clustering, the lower level contained 100 cluster centers, and the upper level contained 10 cluster centers. Text and image features were fused using a three-layer structure, with the text cluster centers set to 10, 10, and 100, respectively. The Transformer-based generator had a dimension of 512, 8 attention heads, and the internal dimension of the feedforward network module was 2048. The training parameters were set to a batch size of 25 samples per batch, an initial learning rate of 1e-4 for cross-entropy training, and 5e-6 for reinforcement learning. The training tolerance was set to 5, meaning that if the cross-entropy loss gradient was less than a threshold for five consecutive epochs, the training process would switch to reinforcement learning mode, or if the CIDEr value decreased for five consecutive epochs, the training process would terminate.

[0141] Evaluation Metrics

[0142] To ensure objective evaluation, this paper uses four widely adopted standard metrics: BLEU-N, METEOR, ROUGE-L, CIDEr, and S* to comprehensively evaluate the quality of automatically generated underwater image captions. For any given underwater image, a labeled reference caption is used to compare with the candidate captions generated by the model. The definitions of these five metrics are as follows:

[0143] (1) BLEU-N is used to evaluate the degree of match between candidate subtitles and reference subtitles. It is a precision-based metric that measures the overlap of N-grams between a candidate subtitle and one or more reference subtitles.

[0144] (2) METEOR calculates the harmonic mean of precision and recall between candidate points and reference points. It evaluates the generated subtitles by comparing them with reference subtitles based on multiple linguistic factors, including synonym matching, stemming, word order, and paraphrase.

[0145] (3) ROUGE-L measures the similarity between candidate subtitles and reference subtitles by calculating the f-measure based on the longest common subsequence, effectively capturing overlapping information.

[0146] (4) CIDEr treats each subtitle as a document, calculates the cosine similarity of the inverse word frequency document frequency vector, and aggregates the similarities of n graphs of different lengths to obtain the final score.

[0147] (5) S* is defined as the average of four evaluation metrics: BLEU-4, METEOR, ROUGE-L, and CIDEr. It provides a comprehensive evaluation of the performance of the generated candidate subtitles.

[0148] By using multiple evaluation metrics, this study comprehensively evaluates the underwater image captioning model developed from multiple perspectives, thereby more accurately reflecting its overall effectiveness. For consistency and clarity, the results of subsequent experiments are presented in decimal form, with higher scores indicating better performance.

[0149] Image feature extraction

[0150] During the image feature extraction stage, the input underwater image is uniformly resized to 224×224 pixels. A pretrained ResNet101 backbone is then used to extract full-image features, resulting in a 2048-dimensional feature vector. Next, the present invention uses Faster R-CNN to extract target region features from the underwater image. Each region is represented by a 2048-dimensional feature vector, and a region proposal network, RoI pooling, and fully connected layers are used to obtain bounding box coordinates and object category information.

[0151] Figure 6 The detection results of two underwater sample images are shown, in which target areas such as shipwrecks and rocks are effectively detected. However, due to the limited number of underwater scene samples in the pre-training dataset, there may be differences between the labeled categories and the actual categories. For example, Figure 6 In (a), the shipwreck is mistaken for an elephant, while in Figure 6 In (b), corals are mistaken for trees. However, this inconsistency does not affect Figure 6 The effectiveness of mid-region feature vectors in underwater image caption generation. Moreover, the attribute information of salient objects, such as “brown”, “dark”, “large”, and “rocky”, is generally accurate, providing valuable support for underwater image caption generation.

[0152] Text feature extraction

[0153] In the candidate vocabulary, 317 unique words were identified, including adjectives, comparative forms of adjectives, superlative forms of adjectives, nouns, plural forms of nouns, proper nouns, and verbs. Since there are no predefined text classification labels for these words, the present invention applies clustering to extract text features of these words. The word embeddings at the cluster centers represent underwater objects and activities, which are then fused with visual features to enhance the generation of underwater image captions.

[0154] Figure 7 A hierarchical text clustering center is given, which consists of nouns, verbs and adjectives related to the ocean, marine life, diving, underwater exploration and related fields. The lexical cluster centers have the following characteristics, among which the main ones refer to marine life (such as sharks, eels, fish) and marine environments (such as corals, seaweed, shipwrecks), while other nouns and verbs are related to diving activities, underwater exploration and human equipment (such as gloves, clothes, boxes, equipment). Adjectives and adverbs mainly describe the properties of objects, such as cylindrical, thin, huge, colorful and golden. Terms describe the shape and properties of objects. These text features show the salient features of typical underwater objects, scenes and activities. When these clustering features are fused with visual features to produce output, the predicted words will also contain underwater information corresponding to the text-visual fusion features.

[0155] Comparison with typical image caption generation models

[0156] The underwater image captioning model developed in this paper is compared with some typical image captioning models, including the Neural Image Captioning (NIC) model (Vinyals et al., 2015), the soft attention captioning model (Xu et al., 2015), the bidirectional LSTM captioning model (Bi-LSTM) (Wang et al., 2016), the adaptive attention captioning model (Lu et al., 2017), the spatial attention (Lu et al., 2017), the bottom-up and top-down attention captioning model (Anderson et al., 2018), the Attention on Attention Network (AoANet) captioning model (Li et al., 2018), and the Meshed-Memory Transformer (M2 Transformer) captioning model (Cornia et al., 2016). al., 2020). As shown in Table 2, the experimental results of the underwater image captioning model proposed in the present invention and some typical image captioning models on the underwater image captioning dataset are shown. The results with the best evaluation parameters are presented in bold. The arrow "↑" to the right of each evaluation parameter in the first row of the table indicates that the "(RL)" model of the present invention is improved over other models in this value.

[0157] Table 2 - Experimental results of image caption generation model on underwater image caption dataset

[0158]

[0159] The NIC image captioning model, as a baseline, exhibits relatively low performance, with scores lagging behind subsequently improved models on all evaluation metrics. After the introduction of the attention mechanism, the soft-attention captioning model shows improvements in metrics such as BLEU and CIDEr, however, its overall performance remains limited. The Bi-LSTM captioning model enhances its ability to model contextual information, resulting in slight improvements in BLEU and METEOR values. The adaptive attention model and the spatial attention model further improve the evaluation parameter values ​​by refining the attention mechanism, however, their generalization capabilities in complex underwater scenes remain limited. Bottom-up and top-down attention captioning models significantly improve by fusing regional and global features, producing strong results in BLEU, METEOR, and CIDEr metrics. The AoANet captioning model enhances attention to image details through a multi-layer attention mechanism, and the M2 Transformer is a Transformer-based image captioning model that utilizes multi-scale feature fusion to achieve near-optimal BLEU and CIDEr scores.

[0160] Building on these studies, the underwater image captioning model developed in this paper, trained with cross-entropy (XE) and reinforcement learning (RL), achieves state-of-the-art performance across most evaluation metrics. After optimization using reinforcement learning, it further improves the quality of underwater image captioning, leading in BLEU, METEOR, ROUGE-L, CIDEr, and S* scores. This demonstrates that the captions generated by our model are not only more closely aligned with the reference captions, but also exhibit greater semantic accuracy and diversity, making it more effective in capturing the fine-grained details of complex underwater scenes.

[0161] Ablation experiments

[0162] This paper conducts a series of ablation experiments to further verify the effectiveness of the developed underwater image captioning model based on multi-model information fusion. Specifically:

[0163] (a) The impact of text feature fusion

[0164] The present invention uses Transformer as a generator and studies the role of text features in the underwater image caption generation model developed by the present invention. The experimental results are shown in Table 3. Analysis of the evaluation metrics shows that the evaluation parameters have relatively low performance when using only regional features as model input, especially in BLEU-4 (0.1994) and CIDEr (0.2753). This shows that the caption generation model using only regional features has obvious limitations in generating high-quality underwater image captions. The model of the present invention performs multimodal information fusion of image and text and is trained using cross-entropy loss. Table 3 shows the improvement in various evaluation metrics, especially in BLEU-2, BLEU-3, BLEU-4, and CIDEr, where its performance is significantly better. Notably, CIDEr increased from 0.2753 to 0.5214, indicating a substantial improvement in the quality of caption generation. In addition, the model trained by the present invention using reinforcement learning outperforms the cross-entropy trained model and the baseline in all metrics, achieving the best performance under various evaluation criteria. Overall, these results verify the effectiveness of multimodal information fusion, which combines multi-scale image features and text clustering features to enhance the performance of underwater image captioning.

[0165] Table 3 - Experimental results of text feature fusion effect on underwater image caption dataset

[0166]

[0167] "Transformer (Reg.)" refers to a model that uses only regional features. "Invention (XE)" and "Invention (RL)" refer to models of the present invention that incorporate text features and are trained using cross entropy loss and reinforcement learning, respectively.

[0168] (b) The impact of using different types of image features

[0169] The present invention conducted two groups of experiments to evaluate the performance of the model of the present invention in fusing different image features and text features. The first group only used the regional features extracted by Faster R-CNN as input, while the second group combined the combination of full-image features and regional features. The model of the present invention adopts a multimodal information fusion mode to fuse image and text information through a two-level text clustering mechanism. Specifically, one group of experimental configurations consists of 100 low-level cluster centers and 10 high-level cluster centers, while the other group of experimental configurations consists of 100 low-level cluster centers and 30 high-level cluster centers. Their image features and text clustering features are fused in a three-layer fusion mode: two layers of fusion of high-level cluster centers and one layer of fusion of low-level cluster centers. The experimental results are shown in Table 4. When the input image features consist of a fusion of full-image features and regional features, the image caption generation model shows significant improvements in multiple evaluation indicators on the dataset. For example, in two comparative experiments, the comprehensive evaluation index S * The values ​​of 0.4282 and 0.4196 are achieved, which shows a significant performance improvement compared with 0.4149 and 0.4029 obtained when only regional features are used. These experimental results strongly verify the effectiveness of multi-scale image, scene and target feature fusion in improving the quality of underwater image caption generation.

[0170] Table 4 - Experimental results of different visual features. RF means using only regional features, while FI-RF combines full image features and regional features.

[0171]

[0172] (c) The impact of different training modes on the model

[0173] In the training process of the underwater image caption generation model of the present invention, both cross entropy loss optimization and reinforcement learning with improved evaluation parameter CIDEr can be used.

[0174] Table 5 presents the results under different training modes. Based on the evaluation metrics, the reinforcement learning-based model consistently outperforms the model trained using the cross-entropy loss across all metrics. These results highlight that incorporating reinforcement learning strategies can significantly improve the performance of underwater image captioning models, particularly in achieving a better balance between text fluency, quality, and generation diversity.

[0175] Table 5 - Experimental results of different training procedures

[0176]

[0177] (d) The impact of the number of high-level text clustering centers in the model

[0178] During the fusion process, the results of the five different combinations of the two-level cluster centers are as follows: Figure 8 This figure illustrates the impact of fusing different configurations of high-level and low-level text cluster centers on evaluation metrics. The two-level text cluster features are still combined with image features through a three-layer fusion model. This three-layer fusion configuration, consisting of a multi-level text feature set consisting of 100 low-level and different high-level cluster centers, captures variations in text features at different levels and enhances the model's ability to extract and utilize rich semantic information.

[0179] Figure 8 (a) shows the comparison of the METEOR and CIDEr evaluation parameters for different numbers of high-order cluster centers. The METEOR value is highest when the number of high-level text cluster centers is 30, while the CIDEr value reaches a peak when the number of high-level text cluster centers is 10. This shows that in the latter case, the model achieves higher semantic similarity and sentence quality. Figure 8 (b) Comparison of ROUGE_L and S* scores is shown. While 30 high-level text clusters yield the highest ROUGE_L score, a combination of 10 high-level text clusters achieves the highest S* score, demonstrating the overall performance of the system and providing the most balanced results. This configuration performs well in both N-gram matching and natural language relevance.

[0180] (e) The impact of the number of fusion layers in the model

[0181] The results are shown in Table 6 when different numbers of cluster centers or fusion layers are used. In previous experiments, when two-level text clustering features were fused with image features across a three-layer structure with multi-head attention, the 10-10-100 cluster center configuration achieved the highest evaluation index. To compare the performance when there are fewer fusion layers, the present invention designed a two-layer structure using a combination of 10-100 cluster features, and another structure using 100 cluster centers to perform a single fusion layer. Different levels of fusion operations enhance features at different depths or strengthen features at specific levels through multiple fusions. From the perspective of various indicators, the three-layer model outperforms the single-layer and two-layer models on multiple evaluation matrices, especially BLEU-4 (0.3084) and CIDEr (0.5989). The three-layer fusion in the present invention's model also showed a significant improvement in the average value S* of the four evaluation parameters, indicating that the overall performance of the model is much superior. In general, increasing the number of layers helps improve the performance of the model in the underwater image caption generation task, especially in terms of expression quality and generation ability.

[0182] Table 6 Experimental results of different fusion layers in the model

[0183]

[0184] Example 3:

[0185] This embodiment provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a method for generating underwater image subtitles based on multimodal information fusion, including the following steps:

[0186] Image feature extraction: Acquire underwater images and use Faster R-CNN to extract multi-scale image features, including full-image features and regional features;

[0187] Text feature extraction: CLIP is used to perform text word embedding encoding to obtain semantic information associated with image content. K-means clustering is used to perform hierarchical text feature extraction, analyze data distribution, and extract text features at different levels.

[0188] Multimodal information fusion: Construct a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features and text features of different scales to obtain fused features;

[0189] Caption generation: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant.

[0190] Example 4:

[0191] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0192] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for generating underwater image subtitles based on multimodal information fusion in the above embodiment. The processor may load and execute the following steps:

[0193] Image feature extraction: Acquire underwater images and use Faster R-CNN to extract multi-scale image features, including full-image features and regional features;

[0194] Text feature extraction: CLIP is used to perform text word embedding encoding to obtain semantic information associated with image content. K-means clustering is used to perform hierarchical text feature extraction, analyze data distribution, and extract text features at different levels.

[0195] Multimodal information fusion: Construct a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features and text features of different scales to obtain fused features;

[0196] Caption generation: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant.

[0197] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0201] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

[0202] Many other changes and modifications can be made without departing from the spirit and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for generating underwater image captions based on multimodal information fusion, characterized in that: The method comprises: Multi-scale image feature extraction: Acquire underwater images and use the faster region-based convolutional neural network Faster R-CNN to extract multi-scale image features, including full-image features and regional features; Hierarchical text feature extraction: Utilize the CLIP model pre-trained with contrastive learning for text word embedding encoding to obtain semantic information associated with image content. K-means clustering is then used for hierarchical text feature extraction to analyze data distribution and extract text features at different levels. Multimodal information fusion: Construct a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features and text features of different scales to obtain fused features; Caption generation: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant. The text feature extraction stage specifically includes: CLIP-based word embedding encoding: The text annotated in the underwater image caption dataset is segmented and counted to construct a corresponding vocabulary. The words in the vocabulary are then embedded using the pre-trained CLIP model to generate word vectors related to the image content. The shape of the word vectors is [N×512], where N represents the number of words in the vocabulary and 512 is the dimension of the word embedding vector output by the CLIP model. Generate text clustering features: Use the K-means algorithm to cluster the word embedding vectors generated by CLIP. First, perform low-level clustering to cluster the word vectors of all words to obtain low-level cluster centers. The shape of the clustering result is [N1×512], where N1 represents the number of low-level cluster centers. Generate high-level clusters: Based on the vector set of low-level cluster centers, high-level clustering is performed; the low-level cluster center vectors are further clustered to obtain high-level cluster centers. The shape of the clustering result is [N2×512], where N2 represents the number of high-level cluster centers; Text feature representation of hierarchical cluster centers: The cluster centers represent samples similar to them, so each cluster center serves as a classification feature of these words, enhancing the expression of text semantics. Low-level cluster centers represent more detailed semantic categories, while high-level cluster centers represent more abstract and broad categories. Information Fusion: The text features of low-level and high-level clusters are gradually integrated with underwater image features to provide rich semantic support for the subsequent image captioning task. Through the above steps, the word embeddings generated by CLIP and the two-level clustering method based on the K-means algorithm effectively extract the hierarchical text features of underwater image captions, enhancing the multimodal information association between text and image. The multimodal information fusion stage specifically includes: Assume that at time step t of the underwater image caption generation model, the full image feature F t and regional characteristics R t =[r1, r2, ..., r n ], extracted by target detection, and then concatenated to form a combined image representation, which is represented as V t , where V t Aggregates multi-scale image information and is defined as: V t =[F t ,R t ]. (3) In the multi-head attention module, by t And the output of the j-th level text clustering module, representing the extracted text feature vector C j , the attention mechanism is applied to realize feature fusion, and the attention weight is calculated as: Where W q and W k It is the query and key value matrix that realizes linear dimensional transformation in the self-attention mechanism, and the parameter d k Yes (W k C j ) dimension, parameter α t Represents the attention weight, which quantifies the V of the combined full-image features and regional features t and text clustering feature C j the correlation between Next, the weighted text clustering features are calculated using the following formula: In the formula represents the weighted text clustering feature, W v represents the linear transformation matrix; Then, and V t Perform residual connection and then perform layer normalization. The formula is: Among them, the function LayerNorm(·) represents the layer normalization operation, which is based on the learnable parameters of all samples to scale and offset, X t It is the output of the layer normalization operation. Layer normalization stabilizes the output distribution by normalizing the activation function output values ​​within each batch, thereby accelerating the training process and enhancing the generalization ability of the model. X t It is fed into a feedforward neural network, then added to itself and normalized by performing layer normalization; Fusion vt =LayerNorm(X t +FFN(X t )), (7) Fusion vt The FFN(·) function represents the two-layer feedforward network operation, which is a supplement to the self-attention mechanism and enhances the expression ability of the model. Finally, the output fusion feature Fusion vt Sent to a Transformer-based underwater image caption generator to predict the next word in the sentence sequence; Through two-level clustering, we obtain the text feature vectors of two groups of cluster centers. In the process of information fusion, we combine them step by step according to the global attention mechanism fusion mode to obtain a vector that integrates more feature information. We adopt a hierarchical information fusion strategy. The higher-level text cluster features are fused with the image features of the previous layer, focusing on identifying the main categories of words. The lower-level text cluster features are fused in the later layers to express more concrete information, and finally the fusion features of image and text are obtained. The model training in the subtitle generation stage specifically includes: Assume that the underwater image caption generation model generates the description y of image I as: y=(y1,y2,...,y T ) (8) where y i is the i-th word generated, T is the length of the description; The training process of the underwater image caption generator includes two stages: cross-entropy optimization and optimization of CIDEr, an evaluation index for the similarity between image description text and reference values. In the cross-entropy optimization stage, the cross-entropy loss function is used to quantify the model's ability to predict word sequences, which is defined as: where y t is the output of the generator at time t, y 1：t-1 is the output of the generator before time t, given the underwater image caption generation model parameter θ, symbol p θ Indicates that the output of the generator is y t , and the previous word is y 1：t-1 The probability distribution of The generator is trained by minimizing the cross-entropy loss to generate sentences similar to the reference sentences. When the cross-entropy optimization reaches a certain level and overfitting or training stagnation occurs, reinforcement learning technology is further applied to directly optimize the value of the evaluation parameter CIDEr of the generated text; The self-critical sequential training (SCST) method is used to optimize the model based on the expected reward. The loss function of reinforcement learning is defined as: The reward function r(·) is the CIDEr parameter value, which is used to measure the consistency between the description generated by the model and the reference text. By maximizing the mathematical expectation of the CIDEr parameter, the model's ability to generate high-quality underwater image captions is further improved.

2. The method for generating underwater image captions based on multimodal information fusion according to claim 1, characterized in that: In the image feature extraction stage, Faster R-CNN is used for image feature extraction, with the pre-trained residual network ResNet101 as the backbone network; multi-scale image features expand the global and local information expression of underwater images from both full-image features and regional features; Assume that a three-channel true color underwater image I is input. ResNet101 is a typical convolutional neural network containing a residual module. It uses different types of convolution kernels to perform a series of convolution operations. It outputs a full-image feature map F, which represents the scene information of the underwater image. F=ResNet101(I). (1) Faster R-CNN uses the region proposal network RPN to generate N proposed target regions, represented as {r1, r2, ..., r N }, it predicts whether each position contains an object and calculates the offset of the bounding box of the object. The region of interest pooling RoIPooling is used to process the region of interest of inputs of different sizes. It outputs N regional feature vectors labeled R, corresponding to N proposed objects, as shown below: R=RoIpooling(r1,r2,…,r N ). (2) Finally, a fully connected network is used to combine bounding box regression and softmax function to predict the bounding box and category of the candidate object. The multi-scale image information provides support for word sequence prediction in the underwater image caption generator.

3. An underwater image caption generation system based on multimodal information fusion, characterized in that: The system is applied to the method according to any one of claims 1 to 2, and the system includes: Image feature extraction module: Acquire underwater images and use Faster R-CNN to extract multi-scale image features, including full-image features and regional features; Text feature extraction module: Use CLIP to perform text word embedding encoding to obtain semantic information associated with image content, use K-means clustering to perform hierarchical text feature extraction, analyze data distribution, and extract text features at different levels; Multimodal information fusion module: Constructs a multimodal information fusion method based on a multi-head attention mechanism to effectively fuse image features of different scales and hierarchical text features to obtain fused features; Caption generation module: Based on the fusion features, the preset Transformer underwater image caption generator is used to generate image captions for underwater images that express the image content and are contextually relevant.

4. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating underwater image subtitles based on multimodal information fusion according to any one of claims 1 to 2 is implemented.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for generating underwater image subtitles based on multimodal information fusion according to any one of claims 1 to 2.

Images