Multi-modal transformer ship identification method and system for complex maritime environment

Abstract

This invention provides a multimodal Transformer ship recognition method and system for complex maritime environments, relating to the field of maritime visual analysis technology. The method includes the following steps: acquiring multi-view ship images and preprocessing them; segmenting the preprocessed images into fixed-size image blocks, mapping each image block to a high-dimensional feature vector through linear projection, and adding positional encoding to generate an image block feature sequence; extracting multi-scale features from the preprocessed images using a convolutional neural network and concatenating them to obtain a visual enhancement feature sequence; mapping characters in a pre-constructed maritime ship name semantic library to semantic vectors using a word vector model to generate textual rule features and concatenating them to form a textual feature vector; inputting the visual enhancement feature sequence and textual feature vector into a multi-layer encoder to perform visual feature and textual feature interaction, modulating the feature fusion ratio, and outputting the ship name string and cargo volume prediction results through a decoder.

Description

Technical Field

[0001] This invention relates to the field of maritime visual analysis technology, and in particular to a multimodal Transformer ship identification method and system for complex maritime environments. Background Technology

[0002] Vessel identification and status assessment are crucial aspects of maritime supervision, port scheduling, shipping safety, and logistics management. Accurate and real-time acquisition of vessel names and cargo information is essential for vessel traffic management, dangerous goods monitoring, tax collection, and prevention of illegal activities. With the development of computer vision and artificial intelligence technologies, vision-based automatic identification technologies are gradually becoming an important supplement to traditional manual visual inspection and Automatic Identification Systems (AIS). However, the maritime operating environment is complex and ever-changing, often facing challenges such as fog, rain, low light, strong reflections, limited shooting angles, and vessel sway, which places extremely high demands on the robustness, accuracy, and adaptability of the identification system.

[0003] Currently, ship identification technology mainly relies on traditional optical character recognition techniques combined with image enhancement methods. Typical solutions include automatic identification systems (AIS), radar-guided image acquisition, edge-detection-based ship name region localization, and general-purpose OCR engine character recognition. However, traditional methods typically use single-modal visual features for recognition, lacking effective utilization of semantic knowledge in the maritime domain. In complex maritime environments such as low light, strong reflections, and rain or fog, image quality deteriorates significantly, resulting in low accuracy in ship name character recognition. Furthermore, existing technologies struggle to deeply integrate hull structural features with ship name semantic information, failing to achieve adaptive coordination between visual and textual features under complex background interference. They also lack effective modeling of prior knowledge such as ship name grammar rules, leading to poor robustness of recognition results and failing to meet the practical application needs of ship identification in complex maritime environments. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal Transformer ship identification method and system for complex maritime environments, in order to solve the problem mentioned in the background art that the current ship identification technology usually uses single-modal visual features for identification, lacks effective utilization of semantic knowledge in the ship domain, and is difficult to deeply integrate the ship's structural features with the ship's name semantic information, resulting in poor robustness of the identification results.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal Transformer ship recognition method for complex maritime environments, comprising the following steps: acquiring multi-view ship images containing hull structure and hull text information, and preprocessing them; segmenting the preprocessed images into fixed-size image blocks, mapping each image block to a high-dimensional feature vector through linear projection, and adding positional encoding to generate an image block feature sequence; extracting features from the preprocessed images using a convolutional neural network to obtain multi-scale features, concatenating the image block feature sequence with the multi-scale features to obtain a visual enhancement feature sequence; and using a word vector model to apply pre-constructed sea... Characters in the ship name semantic library are mapped to semantic vectors, and text rule features are generated by combining ship name grammar rules. The semantic vectors and text rule features are concatenated to form a text feature vector. The visual enhancement feature sequence and the text feature vector are input into a multi-layer Transformer encoder. The visual features and text features interact through a bidirectional cross-modal attention mechanism, and an adaptive gating mechanism is introduced to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector to obtain multi-modal fusion features. Based on the multi-modal fusion features, the ship name string recognition result is output through the ship name decoder, and the ship load prediction result is output through the load volume decoder.

[0006] Optionally, the preprocessing steps specifically include: normalizing the size of the acquired multi-view ship images, uniformly adjusting the images to 640×640 pixels; converting the images from the BGR color space to the RGB color space through color space correction and performing standardization processing; randomly flipping the images, randomly rotating them within ±15 degrees, and randomly scaling them by 0.8-1.2 times through geometric transformation; and generating adversarial networks to perform data augmentation strategies such as weather simulation and random occlusion of ship name areas.

[0007] Optionally, the step of generating the image patch feature sequence specifically includes: dividing the preprocessed image into 16x16 pixel image patches; mapping each image patch into a 768-dimensional feature vector through linear projection; and adding a learnable sine-cosine position code to each feature vector to generate the image patch feature sequence.

[0008] Optionally, the step of using a convolutional neural network to extract features from the preprocessed image specifically includes: employing a convolutional neural network containing multiple convolutional layers and pooling layers, wherein the convolutional layers use 3×3 small convolutional kernels with a stride of 1, and extract features at different levels step by step using 32, 64, and 128 convolutional kernels; the pooling layers use 2×2 max pooling operation to downsample and obtain multi-scale feature maps; and concatenating the multi-scale feature maps with the image patch feature sequence to obtain a visual enhancement feature sequence.

[0009] Optionally, the step of interacting visual features and text features through a bidirectional cross-modal attention mechanism specifically includes: inserting cross-modal attention modules into the 4th, 8th, and 12th layers of the multi-layer Transformer encoder; for the text modality, the query vector of the cross-modal attention module comes from the text features, and the key and value come from the visual features; for the visual modality, the query vector of the cross-modal attention module comes from the visual features, and the key and value come from the text features, so as to enable bidirectional information flow.

[0010] Optionally, the step of introducing an adaptive gating mechanism to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector specifically includes: calculating the gating value. Through the fusion formula The multimodal fusion features are obtained; where: It is the sigmoid activation function. As a visual feature, For text features, , is the learnable weight matrix, b is the bias term; the gate value g∈[0,1] is adaptively learned during training. When the ship name region is clear, g approaches 1 to emphasize visual features, and when the ship name region is blurry, g approaches 0.5 to achieve balanced fusion.

[0011] Optionally, the step of outputting the ship name string recognition result through the ship name decoder specifically includes: generating ship name candidate regions through a region proposal network based on the multimodal fusion features, and retaining high-confidence regions after screening; using a Transformer decoder, when generating each character, calculating the attention weights of the generated character sequence and the visual features in the multimodal fusion features through a visual-text dynamic alignment mechanism, and performing a weighted sum to obtain the context vector of the current character; predicting the probability distribution of the next character based on the context vector and the generated character sequence; and using a beam search algorithm, retaining a preset number of candidate sequences with the highest probability ranking at each step until the termination condition is met, and outputting the complete ship name string with the highest probability.

[0012] Optionally, the step of outputting the ship load prediction result through the load decoder specifically includes: predicting the ship's waterline position information and hull structure information through semantic segmentation based on the multimodal fusion features; calculating the freeboard height and the hull submersion ratio based on the waterline position information and the hull structure information; combining the freeboard height, the hull submersion ratio, and the hull structure information with the multi-scale features fused through the feature pyramid structure, inputting them into a multilayer perceptron, and outputting the classification result or tonnage regression value of the ship load.

[0013] Optionally, the system may also include self-learning and feedback optimization steps: an incremental learning strategy is adopted, which triggers incremental training of the model after the system collects a predetermined number of new sample images. During the training process, a cosine annealing learning rate is used and most of the old model knowledge is retained; a reinforcement learning strategy is adopted, which uses environmental parameters, image quality scores and recognition confidence as states, image preprocessing enhancement actions as space, and a function composed of recognition accuracy and efficiency as rewards to train the agent to dynamically optimize image preprocessing parameters; the model performance is evaluated periodically, and if the performance index drops beyond the threshold or a new error pattern appears, the model is retrained on all the data.

[0014] On the other hand, the present invention also provides a multimodal Transformer ship identification system for complex maritime environments, comprising: an acquisition module for acquiring multi-view ship images containing hull structure and hull text information, and preprocessing them; a visual feature generation module for segmenting the preprocessed image into fixed-size image blocks, mapping each image block to a high-dimensional feature vector through linear projection, and adding position encoding to generate an image block feature sequence; a visual feature enhancement module for extracting features from the preprocessed image using a convolutional neural network to obtain multi-scale features, and concatenating the image block feature sequence with the multi-scale features to obtain a visual enhancement feature sequence; and a text feature generation module for generating text features through word vector models. The system maps characters from a pre-built maritime vessel name semantic library to semantic vectors, and generates text rule features by combining vessel name grammar rules. The semantic vectors and text rule features are then concatenated to form a text feature vector. A fusion module inputs the visual enhancement feature sequence and the text feature vector into a multi-layer Transformer encoder. A bidirectional cross-modal attention mechanism is used to interact the visual and text features, and an adaptive gating mechanism is introduced to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector, resulting in a multi-modal fusion feature. A prediction module, based on the multi-modal fusion feature, outputs the vessel name string recognition result through a vessel name decoder and the vessel load prediction result through a load capacity decoder.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This application achieves a significant improvement in the accuracy and robustness of ship recognition in complex maritime environments by constructing a deep fusion architecture of visual and textual features. By enhancing image patch features with multi-scale visual features and combining them with textual features generated from a pre-built maritime semantic library, the model can simultaneously utilize visual texture information and domain prior knowledge. Through a bidirectional cross-modal attention mechanism and an adaptive gating module, deep interaction and dynamic fusion of visual and textual features are achieved, effectively solving the problem of single-modal feature failure under adverse conditions such as low light and rain / fog obstruction. Based on the fused multimodal features, ship name recognition and load assessment are performed in parallel, which not only achieves end-to-end comprehensive ship information perception, but also significantly improves the accuracy of ship name character recognition and the reliability of load assessment through the synergistic enhancement of semantic knowledge and visual features, giving the system a strong generalization ability to cope with complex and ever-changing maritime environments. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0017] Figure 2 This is a schematic diagram of the multimodal Transformer architecture of the present invention.

[0018] Figure 3 This is a flowchart of the loading capacity evaluation process for this invention.

[0019] Figure 4 This is a schematic diagram of the system structure of the present invention.

[0020] In the diagram: 10 - Acquisition module, 20 - Visual feature generation module, 30 - Visual feature enhancement module, 40 - Text feature generation module, 50 - Fusion module, 60 - Prediction module. Detailed Implementation

[0021] The present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0024] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0025] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Please refer to Figures 1-3This invention discloses a multimodal Transformer ship recognition method for complex maritime environments, comprising the following steps: acquiring multi-view ship images containing hull structure and hull text information, and preprocessing them; segmenting the preprocessed images into fixed-size image blocks, mapping each image block to a high-dimensional feature vector through linear projection, and adding positional encoding to generate an image block feature sequence; extracting features from the preprocessed images using a convolutional neural network to obtain multi-scale features, concatenating the image block feature sequence with the multi-scale features to obtain a visual enhancement feature sequence; and using a word vector model to integrate features from a pre-constructed maritime ship name semantic database. Characters are mapped to semantic vectors, and textual rule features are generated by combining ship name grammar rules. The semantic vectors and textual rule features are concatenated to form a textual feature vector. The visual enhancement feature sequence and the textual feature vector are input into a multi-layer Transformer encoder. The visual features and textual features interact through a bidirectional cross-modal attention mechanism, and an adaptive gating mechanism is introduced to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the textual feature vector to obtain multi-modal fusion features. Based on the multi-modal fusion features, the ship name string recognition result is output through the ship name decoder, and the ship loading volume prediction result is output through the loading volume decoder.

[0028] Specifically, multi-view camera equipment is first deployed in key locations such as ports and waterways, using a combination of circular and linear arrangements to collect multi-view images of ships in real time, including hull structure and hull text information.

[0029] The camera features autofocus and auto exposure, and integrates a light sensor and a weather sensor to monitor environmental conditions in real time. It dynamically adjusts the acquisition parameters based on light intensity and visibility to ensure clear original images are obtained. When the light intensity is low, it automatically increases the sensitivity and extends the exposure time; when the visibility is <500m, it activates the lens defogging function and reduces the zoom to 1-3x.

[0030] The acquired images then undergo preprocessing, including basic processing such as size normalization and color space correction. Next, the preprocessed images are segmented into fixed-size image patches. Each patch is mapped to a high-dimensional feature vector using linear projection, and positional encoding is added to generate an image patch feature sequence. Simultaneously, a convolutional neural network is used to extract features from the preprocessed images, obtaining multi-scale features. The image patch feature sequence is then concatenated with the multi-scale features to obtain a visually enhanced feature sequence that integrates local details and global structure.

[0031] In terms of text feature generation, a maritime ship name semantic library containing standard ship names of more than 20,000 known ships, abbreviations of various provincial administrative regions, and ship type keywords is pre-built. The characters in the semantic library are mapped to semantic vectors through a word vector model, and text rule features are generated by combining ship name grammar rules. The two are then concatenated into a text feature vector with the same dimension as the visual features.

[0032] Subsequently, the visual enhancement feature sequence and the text feature vector are input into a multi-layer Transformer encoder. During the encoding process, a bidirectional cross-modal attention mechanism is used to achieve deep interaction between visual features and text features. An adaptive gating mechanism is introduced to dynamically adjust the fusion ratio of the two according to the image quality, and finally, a multi-modal fusion feature that fully integrates visual information and semantic priors is obtained.

[0033] Based on this multimodal fusion feature, the ship name decoder and the load capacity decoder are processed in parallel to output the ship name string recognition result and the ship load capacity prediction result, thus realizing the comprehensive perception of ship information in complex maritime environments.

[0034] This application achieves a significant improvement in the accuracy and robustness of ship recognition in complex maritime environments by constructing a deep fusion architecture of visual and textual features. By enhancing image patch features with multi-scale visual features and combining them with textual features generated from a pre-built maritime semantic library, the model can simultaneously utilize visual texture information and domain prior knowledge. Through a bidirectional cross-modal attention mechanism and an adaptive gating module, deep interaction and dynamic fusion of visual and textual features are achieved, effectively solving the problem of single-modal feature failure under adverse conditions such as low light and rain / fog obstruction. Based on the fused multimodal features, ship name recognition and load assessment are performed in parallel, which not only achieves end-to-end comprehensive ship information perception, but also significantly improves the accuracy of ship name character recognition and the reliability of load assessment through the synergistic enhancement of semantic knowledge and visual features, giving the system a strong generalization ability to cope with complex and ever-changing maritime environments.

[0035] In some embodiments, the preprocessing steps specifically include: normalizing the size of the acquired multi-view ship images to uniformly adjust the images to 640×640 pixels; converting the images from the BGR color space to the RGB color space through color space correction and performing standardization processing; performing random flipping, random rotation within ±15 degrees, and random scaling of 0.8-1.2 times on the images through geometric transformation; and generating adversarial networks to perform data augmentation strategies such as weather simulation and random occlusion of ship name areas.

[0036] Specifically, the acquired multi-view ship images were normalized to a fixed size of 640×640 pixels to ensure consistency in subsequent processing. Color space correction was used to convert the images from the camera's default BGR color space to RGB, followed by standardization to effectively reduce the impact of lighting differences on feature extraction. For geometric transformations, the images were randomly flipped in both horizontal and vertical directions to simulate changes in the camera's viewpoint; the rotation angle was controlled within ±15 degrees to simulate the swaying amplitude of a ship during navigation; and the scaling ratio was set to 0.8-1.2 times to simulate changes in the distance between the ship and the camera.

[0037] In addition, a data augmentation strategy using generative adversarial networks to perform weather simulations was adopted to generate ship images under complex weather conditions such as rain, fog, and snow, thereby enhancing the model's adaptability to harsh environments. At the same time, a scenario was simulated where the ship's name area was partially obscured by objects such as cables and containers, and the robustness of the model under occlusion conditions was improved through random occlusion operations.

[0038] This application ensures the consistency and diversity of input data by normalizing the size, correcting the color space, and performing geometric transformations on the acquired images, enabling the model to adapt to different shooting distances, angles, and lighting conditions. The introduction of generative adversarial networks for weather simulation and random occlusion data augmentation significantly improves the robustness of the model in harsh environments such as rain, fog, snow, and partial occlusion, effectively avoiding overfitting and providing high-quality training samples for subsequent feature extraction.

[0039] In some embodiments, the step of generating the image patch feature sequence specifically includes: segmenting the preprocessed image into 16x16 pixel image patches; mapping each image patch into a 768-dimensional feature vector through linear projection; and adding a learnable sine-cosine position code to each feature vector to generate the image patch feature sequence.

[0040] Specifically, the preprocessed 640×640 pixel image is uniformly divided into fixed-size 16×16 pixel image blocks, resulting in a total of 1600 image blocks. Each image block undergoes dimensionality transformation through a linear projection layer, flattening the image block and mapping it to a 768-dimensional feature vector. This process achieves embedding from the original pixel space to a high-dimensional feature space.

[0041] After linear projection, a learnable sine-cosine positional code is added to each feature vector. This injects the spatial location information of the image patch in the original image into the feature sequence, ensuring that the Transformer encoder can perceive the relative positional relationships of the image patches. Finally, an image patch feature sequence carrying positional information is generated. This provides a basic visual feature representation for subsequent multimodal fusion.

[0042] This application achieves efficient image-to-sequence conversion by segmenting the image into fixed-size 16×16 pixel image blocks and linearly projecting them into 768-dimensional feature vectors, thus adapting to the input requirements of the Transformer architecture. By adding sine-cosine positional encoding, the model can perceive the spatial positional relationship of the image blocks, preserving the integrity of the visual structure and providing a structured visual feature foundation for subsequent multimodal fusion.

[0043] In some embodiments, the step of using a convolutional neural network to extract features from the preprocessed image specifically includes: employing a convolutional neural network containing multiple convolutional layers and pooling layers, wherein the convolutional layers use 3×3 small convolutional kernels with a stride of 1, and extract features at different levels step by step using 32, 64, and 128 convolutional kernels; the pooling layers use 2×2 max pooling operation to downsample and obtain multi-scale feature maps; and concatenating the multi-scale feature maps with the image patch feature sequence to obtain a visual enhancement feature sequence.

[0044] Specifically, a network structure containing multiple convolutional and pooling layers is constructed. The convolutional layers use 3×3 small convolutional kernels with a stride of 1. By stacking 32, 64, and 128 convolutional kernels layer by layer, hierarchical features from edge textures to component structures are gradually extracted. Batch normalization and ReLU activation functions are introduced after each convolutional layer to enhance the network's expressive power. The pooling layers use 2×2 max pooling, reducing the feature map resolution while preserving the main features, achieving downsampling and receptive field expansion. Through the above multi-layer convolutional and pooling structure, feature maps containing semantic information at different scales are finally obtained. These multi-scale feature maps are then concatenated with the previously generated image patch feature sequence, allowing each image patch feature to obtain local contextual information from the CNN, resulting in a more information-rich visual enhancement feature sequence.

[0045] This application utilizes a multi-layer convolutional neural network to progressively extract multi-scale features of 32, 64, and 128 channels. Through 3×3 small convolutional kernels and 2×2 max pooling, it effectively captures hierarchical visual information from edge texture to component structure. By concatenating the multi-scale feature maps with image block feature sequences, each image block feature integrates local context and global semantics, significantly enhancing the expressive power and discriminative power of visual features.

[0046] In some embodiments, the step of interacting visual features and text features through a bidirectional cross-modal attention mechanism specifically includes: inserting cross-modal attention modules into the 4th, 8th, and 12th layers of the multi-layer Transformer encoder, respectively; for the text modality, the query vector of the cross-modal attention module comes from the text features, and the key and value come from the visual features; for the visual modality, the query vector of the cross-modal attention module comes from the visual features, and the key and value come from the text features, so as to enable bidirectional information flow.

[0047] Specifically, a 12-layer encoder structure is adopted, with cross-modal attention modules inserted in layers 4, 8, and 12 to achieve modal interaction at different levels. In layer 4, cross-modal attention focuses on the initial alignment of local features (such as the strokes of characters in a ship's name), establishing a connection between character-level visual features and character semantic features. In layer 8, the attention mechanism applies to mid-level features (such as character combinations), establishing a connection between ship hull structural features and ship name semantic features (such as the specific hull structure corresponding to the word "oil"). In layer 12, the integration of global features (such as the positional relationship between the ship hull and the ship's name) is completed, providing a unified multimodal representation for subsequent decoding.

[0048] Each cross-modal attention module employs a dual-branch parallel design, with the visual branch output and the text branch output serving as query vectors respectively. These vectors cross-access key-value pairs from the other branch, achieving deep fusion of visual and textual features. For the text modality, the module's query vector comes from textual features, while the keys and values ​​come from visual features, resulting in visual → text attention. This achieves visual feature enhancement of text features; for the visual modality, the query vector comes from visual features, and the keys and values ​​come from text features, with text → visual attention: This enables textual features to guide visual features. This two-way information flow mechanism ensures deep interaction and collaborative optimization between visual and textual features.

[0049] This application inserts cross-modal attention modules into layers 4, 8, and 12 of the Transformer encoder, respectively, to achieve progressive alignment and interaction between visual and textual features at the local, mid-level, and global levels. The bidirectional information flow mechanism enables textual features to guide visual features to focus on key areas, while visual features provide specific visual evidence for textual features. The two reinforce each other, generating multimodal fusion features with richer semantics and more accurate localization.

[0050] In some embodiments, the step of introducing an adaptive gating mechanism to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector specifically includes: calculating the gating value. Through the fusion formula The multimodal fusion features are obtained; where: It is the sigmoid activation function. As a visual feature, For text features, , is the learnable weight matrix, b is the bias term; the gate value g∈[0,1] is adaptively learned during training. When the ship name region is clear, g approaches 1 to emphasize visual features, and when the ship name region is blurry, g approaches 0.5 to achieve balanced fusion.

[0051] Specifically, the fusion ratio of visual and textual features is dynamically modulated through learnable gating units. The specific calculation process is as follows: First, the gating value g = σ(W_V F_V + W_T F_T + b) is calculated, where σ is the sigmoid activation function, compressing the gating value to the [0,1] interval; F_V is the visual enhancement feature sequence, F_T is the text feature vector; W_V and W_T are the learnable weight matrices of visual and textual features, respectively, and b is the bias term. After obtaining the gating value, the final multimodal fusion feature is calculated using the fusion formula F = g ⊙ F_V + (1-g) ⊙ F_T. This gating mechanism learns adaptively during training: when the ship name region in the input image is clear and the visual features are reliable, the model tends to make g approach 1, that is, to emphasize the contribution of visual features; when the ship name region is blurry, occluded, or insufficiently lit, the model automatically adjusts to make g approach 0.5, achieving a balanced fusion of visual features and textual priors. This dynamic modulation mechanism effectively improves the model's adaptability in complex environments.

[0052] This application employs an adaptive gating mechanism to dynamically calculate the fusion weights g, enabling the model to flexibly adjust the contribution ratio of visual and textual features based on the input image quality: emphasizing visual features when the ship name is clear, and balancing the fusion of prior textual knowledge when the image is blurry or occluded. This dynamic modulation strategy effectively improves the model's adaptability and recognition stability in complex environments, avoiding the performance degradation caused by a fixed fusion ratio.

[0053] In some embodiments, the step of outputting the ship name string recognition result through the ship name decoder specifically includes: generating ship name candidate regions through a region proposal network based on the multimodal fusion features, and retaining high-confidence regions after screening; using a Transformer decoder, when generating each character, calculating the attention weights of the visual features in the multimodal fusion features and the generated character sequence through a visual-text dynamic alignment mechanism, and performing a weighted summation to obtain the context vector of the current character; predicting the probability distribution of the next character based on the context vector and the generated character sequence; and using a beam search algorithm, retaining a preset number of candidate sequences with the highest probability ranking at each step until the termination condition is met, and outputting the complete ship name string with the highest probability.

[0054] Specifically, firstly, based on multimodal fusion features, a region proposal network is used to generate candidate ship name regions. The region proposal network uses a 3×3 sliding window to traverse the feature map, generating 128 candidate boxes. After cross intersection and union (CLU) filtering and non-maximum suppression, the top 32 high-confidence candidate ship name regions are retained, focusing on the core recognition region.

[0055] Subsequently, a Transformer decoder is used to generate the character sequence. At each step of character generation, the query vector for the current step is generated from the previously generated i-1 character sequences through an embedding layer and positional encoding. The query vector The query vector is 768-dimensional, consistent with the feature vector dimension, and is calculated through a visual-text dynamic alignment mechanism. The attention weight distribution is obtained by taking the dot product of all visual features K (768×1600) output by the encoder. This allows for precise focusing on the key visual features of the ship's name; subsequently, the visual features V (768×1600) are weighted and summed to obtain the context vector. This context vector integrates the visual details and global semantic information required for the current character generation. The decoder is based on the context vector. Given a sequence of characters, predict the probability distribution of the next character using a fully connected layer. Finally, a beam search algorithm with a beam width of 3 is adopted. At each step, the three candidate characters with the highest probability are retained to gradually build a candidate sequence until the ship name length threshold is reached or a terminator is encountered. Finally, the complete ship name string with the highest probability is output to ensure a balance between recognition accuracy and efficiency.

[0056] Based on multimodal fusion features, this application accurately locates candidate ship name regions through a region proposal network and combines a visual-text dynamic alignment mechanism to focus on key visual information at each decoding step, thus achieving end-to-end ship name string generation. The beam search algorithm optimizes the global probability of sequence output while ensuring efficiency, significantly improving the accuracy and completeness of ship name recognition and eliminating the dependence on traditional OCR engines.

[0057] In some embodiments, the step of outputting the ship load prediction result through the load decoder specifically includes: predicting the ship's waterline position information and hull structure information through semantic segmentation based on the multimodal fusion features; calculating the freeboard height and the hull submersion ratio based on the waterline position information and the hull structure information; combining the freeboard height, the hull submersion ratio, and the hull structure information with the multi-scale features fused through the feature pyramid structure, inputting them into a multilayer perceptron, and outputting the classification result or tonnage regression value of the ship load.

[0058] Specifically, firstly, based on multimodal fusion features, each pixel is classified through a semantic segmentation branch to predict its probability of belonging to the waterline, hull, water surface, or background, thereby accurately obtaining waterline location information and hull structure information. After obtaining the waterline location, the hull principal axis direction angle is calculated based on the hull contour detection results to determine whether the ship is tilting; after perspective transformation correction, the vertical distance from the waterline to the deck is calculated to obtain the freeboard height, and the proportion of the hull submerged area to the total hull side area is calculated to obtain the submersion ratio.

[0059] To fully utilize multi-scale information, feature maps from multiple intermediate layers of the Transformer encoder are upsampled and fused using a feature pyramid structure to obtain multi-scale features rich in spatial details. Freeboard height, hull water immersion ratio, and hull structure information are combined with the multi-scale features fused via the feature pyramid structure, and global pooling is used to obtain a high-dimensional feature representation. Finally, this representation is input into a multilayer perceptron, which outputs a classification result of the ship's load (e.g., empty, half-loaded, fully loaded) or a tonnage regression value (specific load tonnage value) based on task requirements.

[0060] This application accurately detects the waterline position and hull structure through semantic segmentation branches, and combines the calculation of freeboard height and submersion ratio to achieve load assessment without relying on specific draft marks. By combining the above geometric features with multi-scale features after feature pyramid fusion, and inputting them into a multilayer perceptron for classification or regression, the model can comprehensively consider visual details and global semantics, which greatly improves the accuracy and generalization ability of load prediction.

[0061] In some embodiments, the system further includes a self-learning and feedback optimization step: an incremental learning strategy is adopted, and when the system collects a predetermined number of new sample images, incremental training of the model is triggered. During the training process, a cosine annealing learning rate is used and most of the old model knowledge is retained; a reinforcement learning strategy is adopted, with environmental parameters, image quality scores and recognition confidence as states, image preprocessing enhancement actions as space, and a function composed of recognition accuracy and efficiency as reward, to train the agent to dynamically optimize image preprocessing parameters; the model performance is evaluated periodically, and if the performance index drops beyond a threshold or a new error pattern appears, the model is retrained with all the data.

[0062] Specifically, the system also possesses self-learning and feedback optimization capabilities to continuously improve recognition performance. Regarding incremental learning, the system is designed to trigger incremental training after acquiring 1000 new sample images. During training, a cosine annealing learning rate strategy is employed, and techniques such as knowledge distillation are used to retain 90% of the old model knowledge, avoiding catastrophic forgetting, while simultaneously absorbing the feature distribution from new samples.

[0063] In terms of reinforcement learning, a deep Q-network is constructed. The state space is composed of the current environmental parameters (light intensity, weather conditions), image quality score, and recognition confidence. The action space is composed of discrete actions such as image preprocessing enhancement actions (e.g., increasing brightness, improving contrast, adjusting rotation angle, etc.). The reward is a weighted function r = 0.8 × accuracy + 0.2 × (1 - time) of recognition accuracy and computational efficiency. The agent is trained to dynamically optimize the image preprocessing parameters, enabling the system to adaptively adjust the acquisition and processing strategies.

[0064] In addition, the system regularly conducts a comprehensive evaluation of the model's performance, monitoring indicators such as ship name recognition accuracy, load assessment error, and recognition time. If the accuracy drops below a preset threshold or a new error pattern appears, it triggers a full retraining of the entire dataset to ensure that the model always maintains optimal performance.

[0065] This application introduces an incremental learning strategy, enabling the system to efficiently update the model as new samples accumulate, while retaining historical knowledge and avoiding catastrophic forgetting. Reinforcement learning dynamically optimizes image preprocessing parameters, allowing the system to adaptively adjust acquisition strategies to cope with environmental changes. Regularly evaluating model performance and triggering full retraining ensures the long-term stability and accuracy of the system. This self-learning feedback mechanism enables the model to continuously evolve, adapting to new types of ships and complex, ever-changing maritime scenarios.

[0066] Please refer to Figure 4On the other hand, the present invention also provides a multimodal Transformer ship identification system for complex maritime environments, comprising: an acquisition module for acquiring multi-view ship images containing hull structure and hull text information, and preprocessing them; a visual feature generation module for segmenting the preprocessed image into fixed-size image blocks, mapping each image block to a high-dimensional feature vector through linear projection, and adding position encoding to generate an image block feature sequence; a visual feature enhancement module for extracting features from the preprocessed image using a convolutional neural network to obtain multi-scale features, and concatenating the image block feature sequence with the multi-scale features to obtain a visual enhancement feature sequence; and a text feature generation module for generating text features through word vectors. The model maps characters from a pre-built maritime vessel name semantic library to semantic vectors, and generates text rule features by combining vessel name grammar rules. The semantic vectors and text rule features are then concatenated into a text feature vector. A fusion module inputs the visual enhancement feature sequence and the text feature vector into a multi-layer Transformer encoder. A bidirectional cross-modal attention mechanism facilitates the interaction between visual and text features, and an adaptive gating mechanism dynamically modulates the fusion ratio between the visual enhancement feature sequence and the text feature vector to obtain multimodal fusion features. A prediction module, based on the multimodal fusion features, outputs the vessel name string recognition result through a vessel name decoder and the vessel load prediction result through a load capacity decoder. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0068] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal Transformer ship identification method for complex maritime environments, characterized by the following steps: include: Acquire multi-view images of the ship, including hull structure and hull text information, and perform preprocessing; The preprocessed image is divided into fixed-size image blocks. Each image block is mapped to a high-dimensional feature vector through linear projection, and positional encoding is added to generate an image block feature sequence. The preprocessed image is used to extract features using a convolutional neural network to obtain multi-scale features. The image patch feature sequence is then concatenated with the multi-scale features to obtain a visual enhancement feature sequence. The characters in the pre-built maritime vessel name semantic library are mapped to semantic vectors using a word vector model, and text rule features are generated by combining the vessel name grammar rules. The semantic vectors and the text rule features are then concatenated to form a text feature vector. The visual enhancement feature sequence and the text feature vector are input into a multi-layer Transformer encoder. The visual features and text features interact through a bidirectional cross-modal attention mechanism. An adaptive gating mechanism is introduced to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector to obtain multi-modal fusion features. Based on the multimodal fusion features, the ship name string recognition result is output through the ship name decoder, and the ship load prediction result is output through the load volume decoder.

2. The multimodal Transformer ship identification method for complex maritime environments according to claim 1, characterized in that, The preprocessing steps specifically include: The acquired multi-view ship images were normalized in size, and the images were uniformly adjusted to 640×640 pixels. The image is converted from the BGR color space to the RGB color space and then normalized by color space correction. The image is randomly flipped, randomly rotated within ±15 degrees, and randomly scaled by 0.8-1.2 times using geometric transformations. Generative adversarial networks are used to perform data augmentation strategies such as weather simulation and random occlusion of ship name regions.

3. The multimodal Transformer ship identification method for complex maritime environments according to claim 2, characterized in that, The steps for generating image patch feature sequences specifically include: The preprocessed image is divided into 16x16 pixel image blocks; Each image patch is mapped to a 768-dimensional feature vector through linear projection; Learnable sine-cosine positional encodings are added to each feature vector to generate the image patch feature sequence.

4. The multimodal Transformer ship identification method for complex maritime environments according to claim 3, characterized in that, The specific steps of using a convolutional neural network to extract features from the preprocessed image include: A convolutional neural network containing multiple convolutional layers and pooling layers is adopted. The convolutional layers use 3×3 small convolutional kernels with a stride of 1. Features at different levels are extracted step by step through 32, 64, and 128 convolutional kernels. The pooling layer uses a 2×2 max pooling operation for downsampling to obtain multi-scale feature maps; The multi-scale feature map is concatenated with the image patch feature sequence to obtain the visual enhancement feature sequence.

5. The multimodal Transformer ship identification method for complex maritime environments according to claim 4, characterized in that, The steps for interacting visual features and text features through a bidirectional cross-modal attention mechanism specifically include: Cross-modal attention modules are inserted into layers 4, 8, and 12 of the multi-layer Transformer encoder, respectively. For the text modality, the query vector of the cross-modal attention module comes from text features, and the keys and values ​​come from visual features; for the visual modality, the query vector of the cross-modal attention module comes from visual features, and the keys and values ​​come from text features, to enable bidirectional information flow.

6. The multimodal Transformer ship identification method for complex maritime environments according to claim 1, characterized in that, The step of introducing an adaptive gating mechanism to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector specifically includes: Calculate the gate value Through the fusion formula Obtain multimodal fusion features; In the formula: It is the sigmoid activation function. As a visual feature, For text features, , Here, b is the learnable weight matrix, and b is the bias term; The gating value g∈[0,1] is adaptively learned during training. When the ship name region is clear, g approaches 1 to emphasize visual features, and when the ship name region is blurry, g approaches 0.5 to achieve balanced fusion.

7. The multimodal Transformer ship identification method for complex maritime environments according to claim 6, characterized in that, The step of outputting the ship name string recognition result through the ship name decoder specifically includes: Based on the multimodal fusion features, candidate ship name regions are generated through a region proposal network, and high-confidence regions are retained after screening. Using a Transformer decoder, when generating each character, the attention weights of the visual features in the multimodal fusion features and the generated character sequence are calculated through a visual-text dynamic alignment mechanism, and then weighted and summed to obtain the context vector of the current character. Based on the context vector and the generated character sequence, predict the probability distribution of the next character; The beam search algorithm is used to retain a preset number of candidate sequences with the highest probability at each step until the termination condition is met, and then outputs the complete ship name string with the highest probability.

8. The multimodal Transformer ship identification method for complex maritime environments according to claim 7, characterized in that, The step of outputting the ship load prediction result through the load decoder specifically includes: Based on the multimodal fusion features, the ship's waterline position information and hull structure information are predicted through semantic segmentation branch; Based on the waterline position information and hull structure information, calculate the freeboard height and the hull water immersion ratio. The freeboard height, the hull water immersion ratio, and the hull structure information are combined with multi-scale features fused through a feature pyramid structure, and then input into a multilayer perceptron to output the classification result of the ship's load or the tonnage regression value.

9. The multimodal Transformer ship identification method for complex maritime environments according to claim 1, characterized in that, It also includes system self-learning and feedback optimization steps: An incremental learning strategy is adopted. When the system collects a predetermined number of new sample images, it triggers incremental training of the model. During the training process, a cosine annealing learning rate is used and most of the knowledge of the old model is retained. A reinforcement learning strategy is adopted, with environmental parameters, image quality scores and recognition confidence as states, image preprocessing enhancement actions as space, and recognition accuracy and efficiency as rewards, to train the agent to dynamically optimize image preprocessing parameters. Regularly evaluate model performance. If the performance metrics drop below a threshold or a new error pattern emerges, trigger a full retraining of the model on all available data.

10. A multimodal Transformer ship identification system for complex maritime environments, characterized in that, include: The acquisition module is used to acquire multi-view images of ships containing hull structure and hull text information, and to perform preprocessing. The visual feature generation module is used to segment the preprocessed image into fixed-size image blocks, map each image block into a high-dimensional feature vector through linear projection, and add position encoding to generate an image block feature sequence. The visual feature enhancement module is used to extract features from the preprocessed image using a convolutional neural network to obtain multi-scale features, and to concatenate the image patch feature sequence with the multi-scale features to obtain a visual enhancement feature sequence. The text feature generation module is used to map characters in the pre-built maritime vessel name semantic library into semantic vectors through a word vector model, and generate text rule features by combining vessel name grammar rules. The semantic vectors and the text rule features are then concatenated into a text feature vector. The fusion module is used to input the visual enhancement feature sequence and the text feature vector into a multi-layer Transformer encoder, perform visual feature and text feature interaction through a bidirectional cross-modal attention mechanism, and introduce an adaptive gating mechanism to dynamically modulate the fusion ratio of the visual enhancement feature sequence and the text feature vector to obtain multimodal fusion features. The prediction module is used to output the ship name string recognition result through the ship name decoder and the ship load prediction result through the load volume decoder, based on the multimodal fusion features.

Images