Vision-language multi-mode-based ship license plate identification method

Abstract

The invention discloses a ship license plate recognition method based on vision-language multiple modes. The method comprises the following steps: S1, acquiring an original image of a to-be-recognized area through image acquisition equipment; s2, super-resolution reconstruction and data enhancement preprocessing are carried out on the collected images, and a ship license plate recognition data set is generated; s3, constructing a vision-language multi-mode ship license plate recognition model, wherein the model comprises a vision module, a language module and a fusion module; s4, inputting the data set generated in the S2 into a visual module for pre-training; s5, performing language module pre-training by utilizing a ship brand corpus; and S6, loading the pre-training weights obtained in S4 and S5, inputting the data set generated in S2 into the ship license plate recognition model, dynamically weighting the visual features and the language features by adopting a gating fusion strategy, generating multi-modal joint representation, and optimizing the ship license plate recognition model through a multi-modal fusion loss function. According to the method, the accuracy of ship license plate identification is effectively improved, and the adaptability to shielded and blurred images is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a ship license plate recognition method based on vision-language multimodality. Background Art

[0002] As a vital component of the integrated transportation system, the intelligent development of inland waterway shipping is of strategic significance for improving shipping efficiency. As the only legal identification of a vessel, the ship's license plate, composed of a specific character sequence, plays a key role in vessel supervision, navigation safety, and port management. With the rapid growth in the number of inland waterway vessels, traditional manual verification methods are no longer sufficient to meet modern management requirements, necessitating technological innovation to achieve automated identification.

[0003] The core challenges of ship license plate character recognition mainly come from complex environmental interference. Water surface reflections, haze, low light, and tilted viewing angles lead to low image quality. At the same time, the characters on ship license plates are diverse, and there are non-standard fonts, blurs, breaks, or occlusions on ship license plate characters, resulting in a high misrecognition rate. In order to overcome these difficulties, it is necessary to research and apply advanced scene text recognition technologies. These technologies should be able to effectively improve the accuracy of ship license plate recognition, and the processing speed should meet real-time requirements to adapt to complex and changing dynamic scenes. In addition, ship license plate recognition technology should also have good generalization capabilities and be able to operate stably under different water conditions and lighting changes to ensure that high-precision recognition effects can be maintained in all kinds of scenes.

[0004] The current development of deep learning technology has provided new ideas for overcoming these problems. Current ship license plate recognition technologies are mainly divided into two categories: one is the traditional optical character recognition method, and the other is the recognition method that integrates multimodal information. Traditional optical character recognition relies on single image features, is not robust enough for low-quality images, and cannot utilize the semantic logic between characters, resulting in a high misrecognition rate. In contrast, the recognition method that integrates multimodal information enhances the reasoning ability for blurred and occluded characters through cross-modal feature interaction, significantly improving the accuracy and robustness of ship license plate recognition in complex environments. It is suitable for scenarios such as ship supervision, maritime management, and port logistics, and provides strong support for the information management and safety supervision of inland waterway shipping. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention proposes a ship license plate recognition method based on vision-language multimodality, which effectively improves the accuracy of ship license plate recognition and enhances its adaptability to occluded and blurred images.

[0006] The technical solutions of the present invention are as follows:

[0007] A ship license plate recognition method based on vision-language multimodality includes the following steps:

[0008] S1: Obtain the original image of the area to be identified through the image acquisition device;

[0009] S2: Perform super-resolution reconstruction and data enhancement preprocessing on the images collected by S1 to generate a ship license plate recognition dataset;

[0010] S3: Constructing a visual-linguistic multimodal ship license plate recognition model, the model including a visual module, a language module, and a fusion module;

[0011] S4: Input the dataset generated by S2 into the vision module for pre-training;

[0012] S5: Use the ship brand corpus to pre-train the language module;

[0013] S6: Load the pre-trained weights obtained in S4 and S5, input the dataset generated by S2 into the license plate recognition model, use the gated fusion strategy to dynamically weight the visual features and language features, generate a multimodal joint representation, and optimize the license plate recognition model through the multimodal fusion loss function.

[0014] Furthermore, step S2 includes the following sub-steps:

[0015] S2.1: Normalize the image size and adjust it to a preset resolution. Then, perform super-resolution reconstruction on the image using a generative adversarial network.

[0016] S2.2: Perform geometric transformations on the image, including rotation, translation, affine, and perspective. In addition, data enhancement methods such as motion blur, Gaussian blur, and Gaussian pyramid are also used.

[0017] Furthermore, step S3 includes the following sub-steps:

[0018] S3.1: The visual-linguistic multimodal ship license plate recognition network consists of three basic parts: the vision module (Vision Model, VM), the language module (Language Model, LM) and the fusion module. First, the vision module uses SwinTransformer as the backbone to extract multi-scale image features and output feature maps. Then, it passes through the position attention module and generates a query vector Q through custom positional encoding (PositionalEncoding). The attention weight is calculated by combining the key K and value V of the feature map to generate a visual feature sequence and output a character probability distribution. Secondly, the language module is based on the bidirectional context modeling language network (BCN) of the Transformer architecture. The input is the character probability distribution obtained by the visual model after softmax and dimensionality increase operations. The mask mechanism is used to realize context-aware character prediction and output a language feature sequence. Then, the fusion module aligns the visual features with the language features, and dynamically adjusts the weight between the two inputs through a gating mechanism to distribute the contribution of the two in the final prediction and output the final prediction result.

[0019] S3.2: By completely decoupling the vision and language modules and blocking the gradient flow between them, the language module is forced to independently learn language rules rather than relying on implicit information from visual features. This allows the language module to focus on logical error correction and avoid error propagation caused by visual noise. The vision module is pre-trained on image data, while the language model is pre-trained on large-scale text data, improving model generalization. Positional encoding is introduced in the vision module, combined with a self-attention mechanism to fix character position information, addressing the attention drift problem of traditional attention mechanisms. The language module uses contextual information to predict the current character through a masking mechanism, allowing only access to the context to the left and right, while masking the current position information to prevent information leakage. During iterative training, the language module's first input is the prediction output from the vision module, while subsequent inputs are the prediction results from the fusion module in the previous iteration, gradually reducing the impact of noisy input. The predicted text length is iteratively adjusted to address sequence length deviation caused by padding masks. Pseudo-labels are generated using prediction results from multiple iterations, and high-confidence samples are selected for self-training, enhancing the model's generalization.

[0020] Furthermore, step S4 includes the following sub-steps:

[0021] S4.1: The visual module can be formulated as three stages: feature extraction, sequence modeling, and character prediction. SwinTransformer is used as the backbone network for feature extraction and sequence modeling, and the position attention module is used as the prediction network to transcribe visual features into character probabilities in parallel using a query-based approach.

[0022] S4.2: The Swin Transformer is an innovative visual Transformer model designed for efficient processing of high-resolution data such as images. The core idea is to reduce computational complexity by extracting multi-scale features through a hierarchical structure and a self-attention mechanism based on a moving window, while maintaining global information interaction capabilities. A hierarchical architecture is introduced for multi-stage feature extraction: imitating the hierarchical design of CNN, the model is divided into 4 stages, and each stage gradually merges image blocks (Patch Merging), reduces the resolution and increases the number of channels to generate multi-scale feature maps. Specifically, in the first stage, a block segmentation module is introduced to split the input RGB image of size H×W into non-overlapping patches of equal size 4×4. Each patch is flattened and regarded as a token with a feature dimension of 48. The number of tokens obtained is Apply a linear embedding layer to resize its feature dimension to C, and apply a Swin Transformer Block module with improved self-attention to these tokens, keeping the number of tokens unchanged;

[0023] In order to generate hierarchical representations, as the network deepens, the number of tokens is reduced by merging image blocks. The first block merging layer concatenates features of adjacent 2×2 blocks and adjusts the resolution to That is, the number of tokens becomes the original The feature dimension is expanded by 4 times, and then the linear layer is applied to adjust the dimension to 2C. Then the Swin TransformerBlock is applied to transform the feature to keep the resolution unchanged. The same process is repeated twice, and the output resolution is and The token feature dimensions are 4C and 8C respectively. Each stage changes the dimension of the tensor, thus forming a hierarchical representation;

[0024] In order to achieve efficient modeling, Swin Transformer divides each window uniformly in a non-overlapping manner and calculates self-attention within the local window. In order to introduce cross-window connections while maintaining efficient calculation of non-overlapping windows, a moving window partitioning method is adopted. Two partitioning methods are used alternately in the Swin Transformer Block module: The first Swin Transformer Block module uses a regular window partitioning strategy, starting from the upper left corner pixel, to evenly divide the 8×8 feature map into 2×2 windows of size 4×4 (M=4). Then, the next module uses a different window configuration from the previous layer, moving the window out of the regularly partitioned window. pixels;

[0025] When calculating self-attention, we add a relative position bias to each head during the similarity calculation process. The calculation formula is as follows:

[0026]

[0027] in is the query, key, and value matrix, C is the dimension of query and key, M 2 is the number of blocks in the window;

[0028] S4.3: The essence of the positional attention module is to introduce positional encoding into the attention mechanism decoding process. When a sequence input enters the decoder, all tokens in the sequence are input simultaneously and processed in parallel. Without positional information, identical tokens will have no syntactic or semantic differences to the decoder and will produce the same output. Therefore, in order for the model to take into account the information of each position in the input sequence during learning, positional encoding is required.

[0029] The position attention module transcribes visual features into character probabilities in parallel. This design enables the model to better process position information, thereby improving the performance of text image recognition. Position attention is based on the query paradigm:

[0030]

[0031] in is the positional encoding of the character sequence, T is the length of the character sequence, and the matrix V is generated through linear transformation. Position attention decoding can speed up decoding by fixing the position information of each letter and output the decoding results in parallel.

[0032] Furthermore, step S5 includes the following sub-steps:

[0033] S5.1: The ship license plate corpus primarily integrates ship name information from ship databases and the internet, as well as ship license plate text data extracted from a portion of image datasets. The construction of this corpus fully considers the naming characteristics of inland waterway vessels. Since the port of registration of inland waterway vessels is often included as part of the license plate content, the corpus also incorporates text data with regional characteristics, such as the names of various provinces, cities, and regions.

[0034] S5.2: The language module adopts a context-based bidirectional language model. The bidirectional model can capture more information than the unidirectional model, resulting in a stronger feature abstraction capability than the unidirectional model. Given a text string y = (y1,…,y n ), where the text length is n and the number of character categories is c, then the bidirectional model yi The conditional probability is P = (y i |y n ,…,y i+1 ,y i-1 ,…,y1);

[0035] In the specific implementation, BCN is a variant of the multi-layer Transformer decoder. Each layer structure contains a feedforward network with a multi-head attention mechanism, residual connections, and layer normalization. The character vector is directly input into the multi-head attention module instead of the first layer of the network. An attention mask is added to the multi-head attention to prevent self-attention. To avoid information leakage between time steps, the self-attention mechanism is not used. The specific operation is expressed as follows:

[0036]

[0037] K i =V i =P(y i )W l

[0038]

[0039] in The first layer of the network encodes the position of the character sequence, and the other layers of the network encode the output of the previous layer. By character probability After linear mapping matrix, we get: An attention mask used to prevent attention from being paid to the current character. The code matrix, by stacking multiple layers, finally determines the bidirectional representation F of the text y l .

[0040] The step S6 includes the following sub-steps:

[0041] S6.1: First, the pre-trained vision module inputs the 128×384 3-channel training image and performs patch partitioning. This converts the smallest unit of the image from pixels to patches, generating a feature map of size 64×192 and 48 channels.

[0042] S6.2: A Linear Embedding module is used to adjust the number of channels, and then two SwinTransformer Block modules are used for feature extraction. The first stage obtains a feature map with 96 channels and a size of 64×192.

[0043] S6.3: The obtained feature map is first passed through a Patch Merging module to merge patches, and then passed through two Swin Transformer Block modules for feature extraction, resulting in a second-stage feature map with 192 channels and a size of 32×96.

[0044] S6.4: Repeat the above S6.3 process twice, where the Swin Transformer Block modules are 18 and 2 respectively, to obtain:

[0045] The third stage has 384 channels and a feature map of size 32×96;

[0046] The fourth stage has a feature map with 512 channels and a size of 32×96.

[0047] S6.5: Then, the fourth stage feature map is passed through the Position Attention module to convert the visual features into character probabilities, and the obtained visual prediction F v The output size is 52×819, where 52 is the maximum length of the character sequence and 819 is the total number of characters;

[0048] S6.6: Input the result of the visual module in step S6.5 into the pre-trained language module for correction to determine whether the visual output is reasonable. Check each character one by one, iterate 3 times, and output the language prediction F with a size of 52×819. l ;

[0049] S6.7: The iterative joint training stage integrates visual features and language features. The image-based visual model and the text-based language model belong to different modalities. To align the visual features with the language features, a gated mechanism is used for the final prediction:

[0050] G=σ(F v ,F l )W f

[0051] F f =G⊙F v +(1-G)⊙F l

[0052] in The gating mechanism dynamically weights and fuses visual and language features in the channel dimension through adaptive weight allocation.

[0053] S6.8: The model adopts multi-task objectives for end-to-end training. By introducing a multimodal fusion loss function, the total loss consists of three parts: the visual module loss, the language module loss, and the fusion module loss. The loss function is expressed as follows:

[0054]

[0055] in Represent the visual features F v 、Language Features F l and fusion feature F f The cross entropy loss, denote the language loss and fusion loss of the i-th iteration respectively, λ l ,λ f In order to balance the weight coefficients of different loss terms, this multi-objective optimization strategy promotes the collaborative learning of cross-modal features by jointly optimizing vision, language and fusion modules.

[0056] The beneficial effects of the present invention are as follows:

[0057] 1. Through the multimodal fusion of visual and language information, ship license plates can be effectively recognized in complex environments (such as occlusion, blur, water stains, etc.). Especially in scenes with low definition, missing parts of the license plate, or overlapping characters, the joint vision-language model can infer the correct result based on contextual information and character relationships, greatly improving recognition accuracy.

[0058] 2. A specially constructed ship license plate corpus is used for pre-training, enabling the language model to learn the character collocation patterns and sequence structures specific to these fields, maintaining the coherence of the overall recognition structure when vision is unreliable, thereby effectively modeling the rationality between characters during recognition. In the model inference stage, erroneous characters that may appear in the visual recognition stage can be automatically corrected, thereby improving the stability and credibility of the final recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flowchart of a ship plate recognition method based on vision-language multimodality of the present invention;

[0060] Figure 2 This is a model block diagram of a ship plate recognition method based on vision-language multimodality of the present invention;

[0061] Figure 3 This is a visual module block diagram of a ship license plate recognition method based on vision-language multimodality of the present invention;

[0062] Figure 4The present invention provides a block diagram of a SwimTransformer module of a visual module of a ship license plate recognition method based on vision-language multimodality. DETAILED DESCRIPTION

[0063] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0064] Reference Figures 1 to 4 , a ship license plate recognition method based on vision-language multimodality, comprising the following steps:

[0065] S1: Use image acquisition equipment to collect images of different ships in various water environments. The images should ensure that the ship's license plate is included in the images, including different lighting conditions and meteorological changes. In order to capture diverse image data, these cameras should be arranged at different angles and heights to cover as many perspectives as possible. Regularly record images of passing ships and only retain the parts with the ship's license plate. These screened ship images will serve as the preliminary dataset for use in the subsequent S2 step to create the dataset.

[0066] S2: Perform super-resolution reconstruction and data enhancement preprocessing on the images collected in S1 to generate a ship license plate recognition dataset, including the following sub-steps:

[0067] S2.1: Normalize the image size and adjust the resolution to 128×384. Then perform super-resolution reconstruction of the image using a generative adversarial network.

[0068] S2.2: Perform geometric transformations on the image, including rotation, translation, affine, and perspective. In addition, data enhancement methods such as motion blur, Gaussian blur, and Gaussian pyramid are also used.

[0069] S3: Constructing a visual-linguistic multimodal ship license plate recognition model, the model including a visual module, a language module, and a fusion module;

[0070] Reference Figure 2 The construction of the visual-linguistic multimodal ship license plate recognition model described in step S3 includes the following sub-steps:

[0071] S3.1: The visual-linguistic multimodal ship license plate recognition network consists of three basic parts: the vision module (Vision Model, VM), the language module (Language Model, LM) and the fusion module. First, the vision module uses SwinTransformer as the backbone to extract multi-scale image features and output feature maps. Then, it passes through the position attention module and generates a query vector Q through custom positional encoding (PositionalEncoding). The attention weight is calculated by combining the key K and value V of the feature map to generate a visual feature sequence and output a character probability distribution. Secondly, the language module is based on the bidirectional context modeling language network (BCN) of the Transformer architecture. The input is the character probability distribution obtained by the visual model after softmax and dimensionality increase operations. The mask mechanism is used to realize context-aware character prediction and output a language feature sequence. Then, the fusion module aligns the visual features with the language features, and dynamically adjusts the weight between the two inputs through a gating mechanism to distribute the contribution of the two in the final prediction and output the final prediction result.

[0072] S3.2: By completely decoupling the vision and language modules and blocking the gradient flow between them, the language module is forced to independently learn language rules rather than relying on implicit information from visual features. This allows the language module to focus on logical error correction and avoid error propagation caused by visual noise. The vision module is pre-trained on image data, while the language model is pre-trained on large-scale text data, improving model generalization. Positional encoding is introduced in the vision module, combined with a self-attention mechanism to fix character position information, addressing the attention drift problem of traditional attention mechanisms. The language module uses contextual information to predict the current character through a masking mechanism, allowing only access to the context to the left and right, while masking the current position information to prevent information leakage. During iterative training, the language module's first input is the prediction output from the vision module, while subsequent inputs are the prediction results from the fusion module in the previous iteration, gradually reducing the impact of noisy input. The predicted text length is iteratively adjusted to address sequence length deviation caused by padding masks. Pseudo-labels are generated using prediction results from multiple iterations, and high-confidence samples are selected for self-training, enhancing the model's generalization.

[0073] S4: Use the dataset generated by S2 to input into the vision module for pre-training, such as Figure 3 As shown, it includes the following sub-steps:

[0074] S4.1: The visual module can be formulated as three stages: feature extraction, sequence modeling, and character prediction. SwinTransformer is used as the backbone network for feature extraction and sequence modeling, and the position attention module is used as the prediction network to transcribe visual features into character probabilities in parallel using a query-based approach.

[0075] S4.2: The Swin Transformer is an innovative visual Transformer model designed for efficient processing of high-resolution data such as images. Its core idea is to reduce computational complexity by extracting multi-scale features through a hierarchical structure and a self-attention mechanism based on a moving window, while maintaining global information interaction capabilities. A hierarchical architecture is introduced for multi-stage feature extraction: imitating the hierarchical design of CNN, the model is divided into four stages, and each stage gradually merges image blocks (Patch Merging), reduces resolution and increases the number of channels to generate multi-scale feature maps. Specifically, in the first stage, a block segmentation module is introduced to split the input RGB image of size H×W into non-overlapping equal-sized 4×4 blocks (patches). Each patch is flattened and regarded as a token with a feature dimension of 48. The number of tokens obtained is Apply a linear embedding layer to resize its feature dimension to C, and apply a Swin Transformer Block module with improved self-attention to these tokens, keeping the number of tokens unchanged;

[0076] In order to generate hierarchical representations, as the network deepens, the number of tokens is reduced by merging image blocks. The first block merging layer concatenates features of adjacent 2×2 blocks and adjusts the resolution to That is, the number of tokens becomes the original The feature dimension is expanded by 4 times, and then the linear layer is applied to adjust the dimension to 2C. Then the Swin TransformerBlock is applied to perform feature transformation, keeping the resolution unchanged. The same process is repeated twice, and the output resolution is and The token feature dimensions are 4C and 8C respectively. Each stage changes the dimension of the tensor, thus forming a hierarchical representation.

[0077] In order to achieve efficient modeling, Swin Transformer divides each window uniformly in a non-overlapping manner and calculates self-attention within the local window. In order to introduce cross-window connections while maintaining efficient calculation of non-overlapping windows, a moving window partitioning method is adopted. Two partitioning methods are used alternately in the Swin Transformer Block module: the first Swin Transformer Block module uses a regular window partitioning strategy, starting from the upper left corner pixel, and evenly divides the 8×8 feature map into 2×2 windows of size 4×4 (M=4). Then, the next module uses a different window configuration from the previous layer and moves the window out of the regularly partitioned window. pixels;

[0078] When calculating self-attention, we add a relative position bias to each head during the similarity calculation process. The calculation formula is as follows:

[0079]

[0080] in is the query, key, and value matrix, C is the dimension of query and key, M 2 is the number of blocks in the window;

[0081] S4.3: The essence of the positional attention module is to introduce positional encoding into the attention mechanism decoding process. When a sequence input enters the decoder, all tokens in the sequence are input simultaneously and processed in parallel. Without positional information, identical tokens will have no syntactic or semantic differences to the decoder and will produce the same output. Therefore, in order for the model to take into account the information of each position in the input sequence during learning, positional encoding is required.

[0082] The position attention module transcribes visual features into character probabilities in parallel. This design enables the model to better process position information, thereby improving the performance of text image recognition. Position attention is based on the query paradigm:

[0083]

[0084] in is the positional encoding of the character sequence, T is the length of the character sequence, and the matrix V is generated through linear transformation. Position attention decoding can speed up decoding by fixing the position information of each letter and output the decoding results in parallel.

[0085] S5: Use the ship brand corpus to pre-train the language module, including the following sub-steps:

[0086] S5.1: The ship license plate corpus mainly integrates ship name information crawled from ship databases and the internet, and also includes ship license plate text data extracted from some image datasets. The construction of this corpus fully considers the naming characteristics of inland waterway vessels. Since the registration port of inland waterway vessels is usually included as part of the ship license content, this corpus also integrates some text data with regional characteristics, such as the names of various provinces, cities, and regions.

[0087] S5.2: The language module adopts a context-based bidirectional language model. The bidirectional model can capture more information than the unidirectional model, resulting in a stronger feature abstraction capability than the unidirectional model. Given a text string y = (y1,…,y n ), where the text length is n and the number of character categories is c, then the bidirectional model y i The conditional probability is P = (y i |y n ,…,y i+1 ,y i-1 ,…,y1);

[0088] In the specific implementation, BCN is a variant of the multi-layer Transformer decoder. Each layer structure contains a feedforward network with a multi-head attention mechanism, residual connections, and layer normalization. The character vector is directly input into the multi-head attention module instead of the first layer of the network. An attention mask is added to the multi-head attention to prevent self-attention. To avoid information leakage between time steps, the self-attention mechanism is not used. The specific operation is expressed as follows:

[0089]

[0090] K i =V i =P(y i )W l

[0091]

[0092] in The first layer of the network encodes the position of the character sequence, and the other layers of the network encode the output of the previous layer. By character probability After linear mapping matrix, we get: An attention mask used to prevent attention from being paid to the current character. The code matrix, by stacking multiple layers, finally determines the bidirectional representation F of the text y l .

[0093] S6: Load the pre-trained weights obtained in S4 and S5, input the dataset generated in S2 into the license plate recognition model, use the gated fusion strategy to dynamically weight the visual features and language features, generate a multimodal joint representation, and optimize the license plate recognition model using the multimodal fusion loss function. This includes the following sub-steps:

[0094] S6.1: First, the pre-trained vision module inputs the 128×384 3-channel training image and performs patch partitioning. This converts the smallest unit of the image from pixels to patches, generating a feature map of size 64×192 and 48 channels.

[0095] S6.2: A Linear Embedding module is used to adjust the number of channels, and then two SwinTransformer Block modules are used for feature extraction. The first stage obtains a feature map with 96 channels and a size of 64×192.

[0096] S6.3: The obtained feature map is first passed through a Patch Merging module to merge patches, and then passed through two Swin Transformer Block modules for feature extraction, resulting in a second-stage feature map with 192 channels and a size of 32×96.

[0097] S6.4: Repeat the above S6.3 process twice, where the Swin Transformer Block modules are 18 and 2 respectively, to obtain:

[0098] The third stage has 384 channels and a feature map of size 32×96;

[0099] The fourth stage has a feature map with 512 channels and a size of 32×96.

[0100] S6.5: Then, the fourth stage feature map is passed through the Position Attention module to convert the visual features into character probabilities, and the obtained visual prediction F v The output size is 52×819, where 52 is the maximum length of the character sequence and 819 is the total number of characters;

[0101] S6.6: Input the result of the visual module in step S6.5 into the pre-trained language module for correction to determine whether the visual output is reasonable. Check each character one by one, iterate 3 times, and output the language prediction F with a size of 52×819. l ;

[0102] S6.7: The iterative joint training stage integrates visual features and language features. The image-based visual model and the text-based language model belong to different modalities. To align the visual features with the language features, a gated mechanism is used for the final prediction:

[0103] G=σ(F v ,F l )W f

[0104] F f =G⊙F v +(1-G)⊙F l

[0105] in The gating mechanism dynamically weights and fuses visual and language features in the channel dimension through adaptive weight allocation.

[0106] S6.8: The model adopts multi-task objectives for end-to-end training. By introducing a multimodal fusion loss function, the total loss consists of three parts: the visual module loss, the language module loss, and the fusion module loss. The loss function is expressed as follows:

[0107]

[0108] in Represent the visual features F v 、Language Features F l and fusion feature F f The cross entropy loss, denote the language loss and fusion loss of the i-th iteration respectively, λ l ,λ f In order to balance the weight coefficients of different loss terms, this multi-objective optimization strategy promotes the collaborative learning of cross-modal features by jointly optimizing vision, language and fusion modules.

[0109] This invention is expected to address issues such as occlusion, blur, and incomplete characters that traditional models struggle to address. By using a language modeler to predict missing or ambiguous characters, it significantly improves the accuracy of license plate recognition. Applicable to a variety of data collection scenarios, including shore-based cameras, drones, and law enforcement vessels, it enhances the intelligence and automation of automatic ship identification systems, providing efficient support for port security inspections, navigation supervision, and illegal vessel identification. It also promotes the informatization and intelligent transformation of maritime supervision, aids the development of smart water transport and digital ports, and has a practical impact on traffic safety and resource management.

[0110] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A ship plate recognition method based on vision-language multimodality, characterized in that: The method comprises the following steps: S1: Obtain the original image of the area to be identified through the image acquisition device; S2: Perform super-resolution reconstruction and data enhancement preprocessing on the images collected by S1 to generate a ship license plate recognition dataset; S3: Constructing a visual-linguistic multimodal ship license plate recognition model, the model including a visual module, a language module, and a fusion module; S4: Input the dataset generated by S2 into the vision module for pre-training; S5: Use the ship brand corpus to pre-train the language module; S6: Load the pre-trained weights obtained in S4 and S5, input the dataset generated by S2 into the license plate recognition model, use the gated fusion strategy to dynamically weight the visual features and language features, generate a multimodal joint representation, and optimize the license plate recognition model through the multimodal fusion loss function.

2. A ship plate recognition method based on vision-language multimodality as claimed in claim 1, characterized in that: In step S2, the pre-processing includes the following sub-steps: S2.1: Normalize the image size and adjust it to a preset resolution. Then, perform super-resolution reconstruction on the image using a generative adversarial network. S2.2: Perform geometric transformations on the image, including rotation, translation, affine, and perspective. In addition, data enhancement methods such as motion blur, Gaussian blur, and Gaussian pyramid are also used.

3. A method for ship plate recognition based on vision-language multimodality according to claim 1 or 2, characterized in that: The step S3 includes the following sub-steps: S3.1: The visual-linguistic multimodal ship license plate recognition network consists of three basic parts: the visual module, the language module, and the fusion module. First, the visual module uses the Swin Transformer as the backbone to extract multi-scale image features and output a feature map. Then, it passes through the position attention module to generate a query vector Q through a custom position encoding. The attention weight is calculated by combining the key K and value V of the feature map, generating a visual feature sequence and outputting a character probability distribution. Secondly, the language module is based on a bidirectional context modeling language network of the Transformer architecture. The input is the character probability distribution obtained by the visual model after softmax and dimensionality increase operations. The mask mechanism is used to achieve context-aware character prediction and output a language feature sequence. Then, the fusion module aligns the visual features with the language features, dynamically adjusts the weight between the two inputs through a gating mechanism, distributes the contribution of the two in the final prediction, and outputs the final prediction result. S3.2: By completely decoupling the visual module and the language module, blocking the gradient flow between the two, the language module is forced to learn language rules independently rather than relying on the implicit information of visual features, so that the language module can focus on logical error correction; the visual module is pre-trained based on image data, and the language model is pre-trained using large-scale text data. Position encoding is introduced in the visual module, and the character position information is fixed in combination with the self-attention mechanism. The language module uses the context information through the mask mechanism to predict the current character, and only allows access to the context on the left and right sides, but blocks the current position information. During iterative training, the first input of the language module is the predicted output result from the visual module, and the subsequent input is the prediction result of the fusion module in the previous iteration. The predicted text length is adjusted through iteration, and pseudo labels are generated using the prediction results of multiple iterations. High-confidence samples are screened for self-training.

4. A method for ship plate recognition based on vision-language multimodality according to claim 1 or 2, characterized in that: The step S4 includes the following sub-steps: S4.1: The visual module is formulated as three stages: feature extraction, sequence modeling, and character prediction. SwinTransformer is used as the backbone network for feature extraction and sequence modeling, and the position attention module is used as the prediction network to transcribe visual features into character probabilities in parallel using a query-based approach. S4.2: The Swin Transformer is a visual Transformer model that reduces computational complexity by extracting multi-scale features through a hierarchical structure and a self-attention mechanism based on a moving window, while maintaining global information interaction capabilities. It introduces a hierarchical architecture for multi-stage feature extraction: imitating the hierarchical design of CNN, the model is divided into four stages, each of which gradually merges image blocks, reduces resolution and increases the number of channels to generate multi-scale feature maps; the first stage introduces a block segmentation module to split the input RGB image of size H×W into non-overlapping patches of equal size 4×4. Each patch is flattened and regarded as a token with a feature dimension of 48, and the number of tokens obtained is Apply a linear embedding layer to resize its feature dimension to C, and apply a Swin Transformer Block module with improved self-attention to these tokens, keeping the number of tokens unchanged; In order to generate hierarchical representations, as the network deepens, the number of tokens is reduced by merging image blocks. The first block merging layer concatenates features of adjacent 2×2 blocks and adjusts the resolution to That is, the number of tokens becomes the original The feature dimension is expanded by 4 times, and then the linear layer is applied to adjust the dimension to 2C. Then the Swin TransformerBlock is applied to perform feature transformation, keeping the resolution unchanged. The same process is repeated twice, and the output resolution is and The token feature dimensions are 4C and 8C respectively. Each stage changes the dimension of the tensor, thus forming a hierarchical representation. In order to achieve efficient modeling, Swin Transformer divides each window evenly in a non-overlapping manner and calculates self-attention within the local window. In order to introduce cross-window connections while maintaining efficient calculation of non-overlapping windows, a moving window division method is adopted. Two division methods are used alternately in the Swin Transformer Block module: the first Swin Transformer Block module uses a regular window division strategy, starting from the upper left corner pixel, and evenly divides the 8×8 feature map into 2×2 windows of size 4×4. Then, the next module uses a different window configuration from the previous layer and moves the window out of the regularly divided window. pixels; When calculating self-attention, a relative position bias is added to each head during the similarity calculation process. The calculation formula is as follows: in is the query, key, and value matrix, C is the dimension of query and key, M 2 is the number of blocks in the window; S4.3: The positional attention module introduces positional encoding into the attention mechanism decoding process. When a sequence input enters the decoder, all tokens in the sequence are input simultaneously and processed in parallel. If position information is not provided, the same token will have no syntactic or semantic differences to the decoder and will produce the same output. Therefore, positional encoding is required to allow the model to take into account the information of each position in the input sequence during the learning process. The position attention module transcribes visual features into character probabilities in parallel. Position attention is based on the query paradigm: in It is the position encoding of the character sequence, T is the length of the character sequence, and the matrix V is generated by linear transformation. Position attention decoding fixes the position information of each letter and outputs the decoding results in parallel.

5. The method for ship license plate recognition based on vision-language multimodality according to claim 1 or 2, wherein step S5 comprises the following sub-steps: S5.1: The ship license plate corpus integrates ship name information from ship databases and the internet, as well as ship license plate text data extracted from a portion of image datasets. The construction of this corpus fully considers the naming characteristics of inland waterway vessels. Since the port of registration of inland waterway vessels is part of the ship license content, the corpus also integrates text data with regional characteristics. S5.2: The language module adopts a context-based bidirectional language model. The bidirectional model can capture more information than the unidirectional model, resulting in a stronger feature abstraction capability than the unidirectional model. Given a text string y = (y1,…,y n ), where the text length is n and the number of character categories is c, then the bidirectional model y i The conditional probability is P = (y i |y n ,…,y i+1 ,y i-1 ,…,y1); BCN is a variant of the multi-layer Transformer decoder. Each layer structure contains a feedforward network with a multi-head attention mechanism, residual connections, and layer normalization. The character vector is directly input into the multi-head attention module instead of the first layer of the network. An attention mask is added to the multi-head attention to prevent self-viewing. To avoid information leakage between time steps, the self-attention mechanism is not used. The specific operation is expressed as follows: K i =V i =P(y i )W l in The first layer of the network encodes the position of the character sequence, and the other layers of the network encode the output of the previous layer. By character probability After linear mapping matrix, we get: An attention mask used to prevent attention from being paid to the current character. The code matrix, by stacking multiple layers, finally determines the bidirectional representation F of the text y l .

6. The method for ship license plate recognition based on vision-language multimodality according to claim 1 or 2, wherein step S6 comprises the following sub-steps: S6.1: First, the pre-trained vision module inputs the 128×384 3-channel training image and performs patch partitioning. This converts the smallest unit of the image from pixels to patches, generating a feature map of size 64×192 and 48 channels. S6.2: The number of channels is adjusted through a Linear Embedding module, and then through two Swin TransformerBlock modules for feature extraction. The first stage obtains a feature map with 96 channels and a size of 64×192. S6.3: The obtained feature map is first passed through a Patch Merging module to merge patches, and then passed through two SwinTransformer Block modules for feature extraction, resulting in a second-stage feature map with 192 channels and a size of 32×96. S6.4: Repeat the above S6.3 process twice, where the Swin Transformer Block modules are 18 and 2 respectively, to obtain: The third stage has 384 channels and a feature map of size 32×96; The fourth stage has a feature map with 512 channels and a size of 32×96. S6.5: Then, the fourth stage feature map is passed through the Position Attention module to convert the visual features into character probabilities, and the obtained visual prediction F v The output size is 52×819, where 52 is the maximum length of the character sequence and 819 is the total number of characters; S6.6: Input the result of the visual module in step S6.5 into the pre-trained language module for correction to determine whether the visual output is reasonable. Check each character one by one, iterate 3 times, and output the language prediction F with a size of 52×819. l ; S6.7: The iterative joint training stage integrates visual features and language features. The image-based visual model and the text-based language model belong to different modalities. To align the visual features with the language features, a gating mechanism is used for the final prediction: G=σ(F v ,F l )W f F f =G⊙F v +(1-G)F l in The gating mechanism dynamically weights and fuses visual and language features in the channel dimension through adaptive weight allocation. S6.8: The model uses multi-task objectives for end-to-end training. By introducing a multimodal fusion loss function, the total loss consists of three parts: the visual module loss, the language module loss, and the fusion module loss. The loss function is expressed as follows: in Represent the visual features F v 、Language Features F l and fusion feature F f The cross entropy loss, denote the language loss and fusion loss of the i-th iteration, respectively, and λ l ,λ f In order to balance the weight coefficients of different loss terms, this multi-objective optimization strategy promotes the collaborative learning of cross-modal features by jointly optimizing vision, language and fusion modules.

Images