Complex ship license plate identification method, system and device based on mask reconstruction and semantic reinforcement learning

Abstract

The invention discloses a complex ship license plate identification method, system and device based on mask reconstruction and semantic reinforcement learning. According to the method, through two innovative modes of ship board mask reconstruction pre-training and vision-language cross-modal semantic enhancement, the complex spatial position relation of ship board characters can be effectively learned, and information supplementation can be carried out on ship boards with visual signal loss through semantic information. Meanwhile, the two innovations serve as a ship license plate recognition pre-training mode, configuration is only needed in the training stage of the ship license plate recognition deep neural network model, configuration is not needed in the online testing stage once training of the model is completed, and therefore extra ship license plate recognition calculated amount cannot be brought. The method has good coping and recognition capabilities for complex conditions such as multi-line texts, fuzzy texts, shielding texts, vertical texts and inverted-sequence texts in the ship sign.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a complex ship plate recognition method, system and device based on mask reconstruction and semantic enhancement learning. Background Art

[0002] Waterway shipping has been one of the important modes of transportation in my country since ancient times, and it plays an extremely important role in my country's economic trade. In order to ensure the safety of ships coming and going and the efficient operation of ports, it is necessary to identify the ship and record information. The traditional ship identity identification adopts the ship automatic identification system (Automatic Identification System, referred to as AIS system), which is composed of base station equipment and shipborne equipment for communication. It is an identification method based on the communication of the Internet of Things. When the ship carries the AIS system, it can send information to the base station equipment for identity positioning. However, this solution also has certain limitations. It is impossible to obtain the image information of the ship entering and leaving in real time. The communication equipment currently has a certain delay, and the AIS system is not popular at present. There are still a large number of ships that are not installed. With the increasing deployment of monitoring equipment in ports, information is presented in carriers such as video images, and massive image data can be supported by computer vision technology. However, in the complex scene of ship license plates, the current common text recognition algorithm cannot effectively recognize, and the license plate recognition algorithm cannot meet the requirements. In order to build a modern port with intelligent digitization, there is an urgent need for a technology that can recognize ship license plates.

[0003] At present, there are few studies on ship license plate character recognition in the field of waterway transportation, and there are more complex problems in ship license plate character recognition. The present invention focuses on solving the following two types of sub-problems:

[0004] The image quality is low. The distance between the surveillance equipment and the ship, the movement of the ship, and the weather will cause the image quality to be low. This often leads to low brightness, low contrast, uneven lighting, perspective deformation, and incomplete occlusion. Traditional methods can only solve relatively clear pictures, or add additional super-resolution modules to solve the problem.

[0005] The layout of the text is irregular. There are a lot of multi-line texts (two or more lines); tilted text; vertical text; mirror text due to printing problems; disordered and reversed text; and non-continuous text due to the longer length of the information on the ship license plate.

[0006] In order to achieve accurate recognition of ship license plates in waterway intelligent transportation, fully grasp the text semantic information in ship license plates, and provide effective information for the safety control of waterway transportation, how to use computer technology to recognize ship license plate information is a very valuable research topic at present.

[0007] The image quality of the ship plate is low. Figure 3 As shown, the schematic diagram of the complex layout of the ship plate text is as follows Figure 4 shown. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies of the prior art and propose a complex ship license plate recognition method, system and device based on mask reconstruction and semantic enhancement learning, which are used to effectively recognize ship license plate characters in complex scenes.

[0009] The object of the present invention is achieved through the following technical solution: a complex ship plate recognition method based on mask reconstruction and semantic enhancement learning, the method comprising the following steps:

[0010] (1) Obtain a ship plate image, divide it into a sequence of image blocks, input it into a mask pre-trained encoder, and integrate features of different depths;

[0011] (2) Based on the visual encoder, the feature space of the visual vocabulary of the ship license plate image is converted into the latent feature space of the visual semantic vector; based on the text encoder, the features of the visual semantic vector are extracted and converted into the same feature space as the mask network layer for feature fusion;

[0012] (3) fusing the image features obtained in step (1) and the image features obtained in step (2) using a gated fusion operation;

[0013] (4) The gated fused features are saved as feature memory units, and the global features obtained in step (2) are extracted and spatially mapped so that the hidden feature dimension is consistent with the feature dimension of the memory unit;

[0014] (5) Compare the features in step (4) with the global text features output by the text encoder to enhance the semantics of the features output by the visual encoder;

[0015] (6) Based on the memory unit, the decoder generates a mask matrix by generating reverse pairs and decodes it in combination with the position information to obtain the license plate recognition result.

[0016] Furthermore, in step (1), the mask pre-training process is as follows: the ViT encoder is used to encode visual features, the decoder receives the latent representation as input, and reconstructs the entire image through a series of Transformer layers, and performs visual feature loss optimization for training; during pre-training, the position embeddings of the encoder and decoder are frozen.

[0017] Furthermore, in step (2), the feature transformation of the visual encoder and the text encoder are both implemented through a learnable feature adapter module, which consists of a series of layers: a projection head, a linear layer, followed by a ReLU activation layer, then another linear layer, and finally a ReLU activation layer.

[0018] Furthermore, in step (5), the contrastive learning loss adopts InfoNCE, and global visual features and text features are obtained through the visual encoder and the text encoder; the global visual features are used as the output of the cls token, and a projection is further used to align the dimensions of the text features extracted by the text encoder; the cosine similarity of the visual and text feature pairs of the same ship plate is maximized, while the similarity of the unmatched pairs is minimized, and the visual encoder is enhanced in the ship plate recognition scenario through the guidance of semantic labels.

[0019] Furthermore, in step (6), different arrangements of the reverse order correspond to different positions of character decoding, and the specific arrangement is realized by the matrix of the attention mask. In the reverse decoding process, the label is first decoded, and then the information is inferred by combining the label with the information output by the encoder and the attention mask matrix. The inference result is further inferred by combining the label, and finally the reverse decoding process is completed.

[0020] Furthermore, a pre-trained text encoder is used to iterate the correction of the result. Specifically, the character features of the decoded information are decoded, the decoded character string is input into the text encoder to obtain text features, this feature is merged with the features of the decoded information to obtain a new feature, and then the optimized decoding of the final text output is performed again.

[0021] In a second aspect, the present invention further provides a complex ship plate recognition system based on mask reconstruction and semantic enhancement learning, the system comprising:

[0022] The image block sequence preprocessing module is used to obtain the ship plate image, divide it into image block sequences, and input it into the mask pre-trained encoder to integrate features of different depths;

[0023] The cross-modal feature output module is used to convert the feature space of the visual vocabulary of the ship license plate image into the latent feature space of the visual semantic vector based on the visual encoder; extract the features of the visual semantic vector based on the text encoder and convert it to the same feature space as the mask network layer for feature fusion;

[0024] A gated feature fusion module is used to fuse the image features obtained by the image block sequence preprocessing module and the image features obtained by the cross-modal feature output module using a gated fusion operation;

[0025] The feature storage and global feature extraction module is used to store the gated fused features as feature memory units, and extract and spatially map the global features obtained by the cross-modal feature output module, so that the hidden feature dimension is consistent with the feature dimension of the memory unit;

[0026] The cross-modal feature contrast learning module is used to compare the features in the feature storage and global feature extraction module with the global text features output by the text encoder to enhance the semantics of the features output by the visual encoder.

[0027] The reverse decoding module is used to generate a mask matrix based on the memory unit. The decoder generates reverse pairs and decodes them in combination with the position information to obtain the ship license plate recognition result.

[0028] In the third aspect, the present invention also provides a complex license plate recognition device based on mask reconstruction and semantic enhancement learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the complex license plate recognition method based on mask reconstruction and semantic enhancement learning.

[0029] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the complex license plate recognition method based on mask reconstruction and semantic enhancement learning.

[0030] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning.

[0031] Beneficial effects of the present invention:

[0032] (1) Higher computational efficiency: Traditional low-quality text recognition algorithms usually require the use of image super-resolution algorithms for preprocessing, or embedding super-resolution modules within the model for text recognition, which increases the amount of computation. However, the ship license plate recognition algorithm based on self-supervised pre-training in the present invention does not require these additional steps, thereby significantly improving computational efficiency.

[0033] (2) Strong model transferability: The pre-trained model has strong transferability and can be applied to different low-quality license plate recognition tasks. Through fine-tuning and further optimization, the model can flexibly adapt to various application scenarios.

[0034] (3) Higher recognition accuracy: By designing two different learnable feature adapters, effective mapping between the visual feature space and the text feature space is achieved, which enhances the model's understanding of image semantics in complex scenes. This method significantly improves the accuracy of text recognition when processing ship license plates with complex or irregular text arrangements (such as reverse or random order). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 It is a schematic diagram of the structure of the ship license plate recognition technology based on the multimodal large model of the present invention.

[0037] Figure 2 The flowchart of the ship license plate recognition technology based on the multimodal large model of the present invention.

[0038] Figure 3 Schematic diagram of a ship license plate image with low imaging quality.

[0039] Figure 4 A complex schematic diagram for the ship sign text layout.

[0040] Figure 5 It is a structural diagram of a complex ship license plate recognition device based on mask reconstruction and semantic enhancement learning provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] The present invention provides a complex ship plate recognition method based on mask reconstruction and semantic enhancement learning, which adopts mask space modeling to solve the situation of irregular text layout, and adopts semantic enhancement to solve the situation of low imaging quality. In the pre-training stage, the mask pre-training mask visual encoder is performed by the image mask pre-training method, and the contrast learning pre-training of the ship plate image and the ship plate text pair is performed at the same time. In the training stage, the pre-trained cross-modal model CLIP is cascaded to combine the text branch and the visual branch, and the feature space mapping of the cross-modal visual feature to the cross-modal text feature is performed by the designed learnable feature adapter as the cross-modal feature extraction branch, and the mask visual encoder in the pre-training stage is used as the mask visual feature extraction branch, and the features extracted by the two branches are adaptively gated feature fusion to obtain a feature memory unit. For the extracted memory unit, the InfoNCE contrast loss function can be additionally used in the training stage to make the global features of the text features and the global features of the memory unit to be contrasted and learned, and the semantics of the feature memory unit is further improved. At this time, the text feature is extracted by freezing the pre-trained text encoder.

[0044] In the cross-modal feature fusion decoding stage, the reverse pair arrangement modeling language decoder is used for decoding. The decoder is essentially an autoregressive causal relationship decoder. In the model training stage, the real value of the ship license plate characters can be used to make a real value random shuffle. For example, the name of a ship license plate is - Zhe Xiao Shan Goods 23450 ZHEX IAO SHAN HUO, which is randomly shuffled to ZHEX Zhe Shan ANH Goods Xiao 25340 IAO SHUO, and it is reversed to get a reverse pair. We use three groups of reverse pairs to learn feature context decoding, so that it is not limited to traditional forward decoding and reverse decoding. Finally, the decoding is performed character by character. The decoding result is output through the normalized exponential function layer and the beam search algorithm, and the cross entropy is used as the loss function. In order to further improve the recognition results, an additional language model can be added to iteratively correct the recognition results.

[0045] The present invention proposes a complex license plate recognition method based on mask reconstruction and semantic enhancement learning, which is oriented to the license plate recognition scenario of waterway intelligent transportation and fully utilizes the information of multimodal large models and self-supervised mask pre-training information. It performs feature fusion of multiple pre-training information and feature space mapping. The present invention can make full use of additional unlabeled data for effective pre-training. Figure 1 and Figure 2 As shown, the specific steps include:

[0046] (1) Self-supervised mask pre-training

[0047] (1-1) In the visual feature encoding stage of self-supervised mask pre-training

[0048] This method uses ViT (Vision Transformer) as an encoder for feature extraction. The features extracted by ViT are more advantageous in ship license plate recognition. Since ship license plates are widely multi-line disordered, rather than single-line ordered, it is necessary to extract features with global and self-attention characteristics. The receptive field of CNN often has limitations in such problems. At the same time, before feature extraction, the image needs to be masked at a high ratio of pixels.

[0049] (1-2) Visual feature reconstruction decoding stage of self-supervised mask pre-training

[0050] The decoder's task is to reconstruct the occluded image parts based on the latent representation provided by the encoder. The decoder receives the latent representation as input and, through a series of Transformer layers, reconstructs the entire image, including the occluded parts of the original input. The goal of reconstruction is not only to accurately predict the original values ​​of the occluded pixels, but also to maintain the overall coherence and visual quality of the image.

[0051] (1-3) Visual feature loss optimization stage of self-supervised mask pre-training

[0052] Training process During the training phase, the goal of MAE is to minimize the difference between the reconstructed image and the original image, which is usually achieved by calculating the loss function mean square error MSE between them.

[0053]

[0054] Where n represents the total number of samples, y i is the pixel value of the i-th sample, y' i is the predicted value of the ith sample. The smaller the value of MSE, the closer the predicted value of the model is to the actual value, that is, the better the performance of the model. MSE directly measures the pixel-level difference between the reconstructed image and the original image. This difference is intuitive, easy to understand and calculate, making the goal of model training clear - that is, minimizing the squared difference between the predicted image and the actual image. MSE has good mathematical properties. It is a convex function. For linear regression models, this guarantees the possibility of finding the global minimum. In many image reconstruction tasks, although the deep learning model itself may be nonlinear, MSE as a loss function still helps to stabilize the training process and effectively minimize the error through optimization algorithms such as gradient descent.

[0055] In step (1), during the self-supervised mask pre-training, the position embeddings of the encoder and decoder are frozen and initialized using the sine-cosine embedding method. Although CLS_Token participates in the reconstruction of the image, it does not participate in the reconstruction loss calculation based on the mean square error loss. It is designed as a learnable parameter.

[0056] (2) Cross-modal contrastive learning pre-training

[0057] Through large-scale cross-modal contrastive learning, the correspondence between image content and natural language description is learned, and remarkable zero-shot performance can be achieved on multiple visual tasks. The success of this method provides new inspiration for text recognition, and the visual model can learn richer and more robust text feature representations. The present invention rationally utilizes image-text pairs and performs pre-training of contrastive learning in the manner of cross-modal contrastive learning. The initialization parameters are derived from the public CLIP model.

[0058] (3) Cascade of visual encoder and text encoder

[0059] (3-1) Image block sequence preprocessing

[0060] After completing steps (1) and (2), two encoding branches can be obtained, one for the mask pre-trained encoder feature extraction branch, and the other for the cross-modal encoder feature extraction branch. First, the image input is uniformly converted into images∈(B,C,H,W) through image preprocessing technology, where B represents the batch size during training, C represents the number of channels of the image, which is set to 3 by default, that is, RGB primary color channels, H represents the height of the image, and W represents the width of the image, which is set to 224 pixels by default. The image also needs to be divided into blocks, converting the two-dimensional image into a one-dimensional image block sequence, where the image block size is 16×16, a total of 196, and also contains a CLS_Token, which is the global feature of the image, and the image block and the image position code are calculated and added.

[0061] (3-2) Image block sequence input to mask pre-trained encoder

[0062] The image block sequence X is input into the network layer of the mask pre-trained encoder ViT. The output features of each layer are transmitted in sequence, and the residual connection technology is used to integrate the features of different depths.

[0063] (3-3) The visual encoder uses the visual branch of CLIP, and the text encoder uses the text branch of CLIP. The ship license image is input into the visual encoder of CLIP, where B is the training batch size, P is the size of the image block, and D clipmodelThe hidden layer dimension size of the CLIP vision branch output.

[0064] F v =CLIPEncoder vision (images), F v ∈(B,P 2 +1,D clipmodel ) (3-1)

[0065] (3-4) The feature space of visual vocabulary is transformed through the learnable feature adapter module Adapter v Transformed into the latent feature space of the visual semantic vector, proj v (F v ) is the projection of the feature.

[0066] F' v =Adapter v (proj v (F v )) (3-2)

[0067] (4) Cross-modal feature output

[0068] (4-1) The visual semantic text vector space of latent features is input into the text encoder of CLIP. The text encoder has a total of 24 layers. The embedding layer is deleted to accept the features of its visual semantic vector.

[0069] F t =CLIPEncoder textual (F' v ), F t ∈(B,T,D clipmodel ) (4-1)

[0070] (4-2) The features output by the text encoder are then input into the learnable feature adapter module Adapter for feature projection, which is projected into the output feature space of the mask pre-trained encoder for subsequent feature fusion.

[0071] F' t =Adapter t (proj t (F t )) (4-2)

[0072] In step (3) and step (4), the features output by the CLIP visual encoder are mapped from the vectors in the image space to the vectors in the text space through a mapping process of a learnable adapter, obtaining a vector sequence of text prompts, which is then further processed by the learnable adapter and then input into each layer of the text encoder, ultimately outputting a 768-dimensional text feature sequence. Since the visual vocabulary output by the visual encoder in CLIP is highly consistent with the text vocabulary features output by the text encoder, the feature semantic rearrangement feature of this method is reflected in processing the output of the CLIP visual encoder through the text encoder, and utilizing the normal human character reading order of the text string processed by CLIP during the pre-training process, the feature space of the visual vocabulary is further represented by the text vector feature space.

[0073] In the model of the present invention, the learnable feature adapter module is mainly composed of a series of layers: a projection head (the projection head can choose convolution or a learnable parameter), a linear layer, followed by a ReLU activation layer, then another linear layer, and finally a ReLU activation layer. Two different adapter modules are used in the entire network model architecture, and the strategy of reducing dimension and then increasing dimension is used for feature adjustment. Specifically, the first adapter module is used for feature conversion after CLIP visual encoder mapping. The second adapter is used for feature conversion after CLIP text encoder mapping.

[0074] This architecture enables the model to adjust features in a targeted manner at different stages, facilitating the effective integration and processing of information from different modalities. The use of adapter modules not only simplifies the process of feature conversion, but also introduces a flexible mechanism for refining the representation of information, which is crucial for tasks involving complex data types such as images and text.

[0075] (5) Gated feature fusion

[0076] The main operation used is gate fusion for feature fusion. Gate fusion is a common technique in the field of deep learning, especially when it is necessary to merge information from different sources. It is widely used in scenarios such as multimodal learning, feature fusion, and enhancing the model's ability to dynamically adjust the importance of input information. Gated fusion introduces a gating mechanism to automatically learn the importance weights of different information sources, and then effectively integrates this information to achieve the purpose of improving model performance. The basic idea behind gated fusion is that different information sources may contribute differently to the final task, some information may be more important, while others may be relatively less important. The gating mechanism can be used to assign weights to different information sources. These weights are learnable and can be automatically adjusted according to the task objectives to ensure that the model can focus more on important information. Finally, the memory that is finally sent to the autoregressive decoder is obtained through fusion through the gated fusion mechanism.

[0077] memory=σ1(W m F m +b m )*F m +σ2(W t F' t +b t )*F' t (5-1)

[0078] Among them, F m represents the output features of the mask pre-trained encoder, F′ t is the output feature of step (4-2), W m 、b m , W t 、b t Both are learnable variables, and σ1 and σ2 are two hyperparameters used to balance the importance of the two features.

[0079] Through the above process, gated fusion achieves dynamic weight allocation and effective fusion of different feature sources, thereby enhancing the model's ability to capture key information and improving the model's performance and generalization ability. This mechanism is particularly useful when processing complex multimodal data, and can adaptively adjust the focus on different modal information according to task requirements.

[0080] (6) Feature storage and global feature extraction

[0081] During the model processing, the features output from step (5) are saved and used as a "feature memory unit". These stored features are intended to support the decoder in subsequent steps, especially when recognizing ship license plate characters, where these features will be used as key information for the decoder. During the training phase of the model, the model outputs a cross-modal fusion feature (i.e., features that fuse different modalities of vision and text). For these fusion features, the model extracts representative global features. After extracting the global features, the model performs a "spatial relationship mapping" operation. The purpose of this step is to adjust the representation of the features so that the dimensions of these hidden features (i.e., the number and arrangement of features) can match the dimensions of the previously stored feature memory units. Through spatial relationship mapping, the hidden features generated by the model during training are ensured to be consistent in dimension with the previously stored memory unit features. This consistency is necessary because it ensures that the features can be correctly matched and interacted with each other, thereby improving the accuracy and efficiency of the decoder in ship license plate character recognition.

[0082] (7) Contrastive Learning of Cross-modal Features

[0083] The features output from step (6) are compared with the global text features output by the completely frozen CLIP text branch in step (2) for learning, and the contrast loss is InfoNCE to enhance the contextual semantics of the features output by the visual encoder. Although the visual encoder is pre-trained through self-supervised learning, the pre-training process is task-agnostic. Further retraining the visual encoder can enhance the feature extraction ability of the visual encoder in the license plate recognition scenario through the guidance of semantic labels. Since the license plate image and its corresponding text label are semantically related, constraints are added to ensure that they are closely aligned. At the same time, the contrastive learning mechanism between the visual and text encoders can maximize the cosine similarity of the visual and text feature pairs of the same license plate, while minimizing the similarity of mismatched pairs. Specifically, in the training phase, given a license plate image annotation with text labels, the global visual features I and text features T are first obtained through the visual encoder and text encoder. The global visual features I are output as the cls token, and a projection operation is further used to align the dimensions with the text features T extracted by the CLIP text encoder. For the i-th sample (global visual feature I i and text features T i ) and N samples in the same batch, the loss comparison learning is implemented using infoNCE loss, that is:

[0084]

[0085] Where T iis the text feature of the text label of the i-th sample, and the parameter τ is a temperature parameter used to adjust the distribution of similarity scores between samples. It is worth noting that the text encoder is not required in the inference stage, which ensures that no additional computational load is added during the inference process.

[0086] (8) Generation of reverse order pairs of training data

[0087] First, some permutation sequence modeling techniques are explained. The general sequence modeling technique considers the positive or reverse order relationship between characters, while the permutation sequence modeling technique randomly shuffles the order between characters, such as the order of apple, which is shuffled into papel. The shuffling process is completely random, and then the shuffled reverse order is taken as lepap as the reverse order pair. The method in the present invention is three reverse order pairs used in the training stage. Different permutations correspond to the positions of character decoding at different positions. The specific permutation is realized by the matrix of the attention mask. For example, the order of (3, 2, 1) is taken as an example. In the decoding process, the [B] label is first decoded, and then the information of y3 can be inferred through the combination of the [B] label and the information output by the encoder and the attention mask matrix. Then, y3 and [B] labels can be inferred to y2. Finally, [B], y3, and y2 are known to be able to infer the information of y1, and finally the reverse decoding process is formed. The same is true for other permutation sequences. The main advantage of this method is its ability to deeply learn contextual information, which shows superior performance compared to traditional forward and reverse decoding strategies. It achieves tighter information integration by strengthening the connections between characters and the associations with the visual features output by the corresponding image encoder.

[0088] (9) Feature decoding for permutation modeling

[0089] The memory unit after the above feature fusion is combined with the mask matrix generated by the image inversion pair in step (8) and the position query information to obtain the initial stage decoding information h0. Then, the feature hi of the fused attention relationship is obtained through the attention layer and the residual connection. Then, the multi-layer perceptron is used to perform inductive learning of the decoding information, and the decoding information is transcribed through the linear layer to finally obtain the decoding information.

[0090] After summarizing the above process, we can come up with the final formula, where T represents the longest length in the ship license plate recognition, and S represents the number of characters in the recognition dictionary. The reason for adding 1 is that the [E] tag needs to be used as the end recognition terminator.

[0091] y=Dec(z,p,c,m)∈R (T+1)×(S+1) (9-1)

[0092] Among them, z represents the memory unit derived from multimodal fusion and semantic guidance of the large model, which will be fused and residually connected through MHA. Among them, m represents the matrix of attention mask, c represents the label context information (context), and p represents the position information query. The way to achieve label disorder is to shuffle the attention mask matrix.

[0093] (10) Iterative Correction of Language Model

[0094] Step (10) is optional. If you want to further improve the recognition results and do not care about the inference speed, you can use a language model to iterate the results. The specific language model is the pre-trained text encoder in step (2). The actual steps are as follows: decode the character features output in step (9), input the decoded character string into the text decoder in step (2) to obtain text features, fuse this feature with the feature output in step (9) to obtain a new feature, and then input it into step (11) for decoding again.

[0095] (11) Optimized decoding of final text output

[0096] The Logits output from step (10) are decoded from features to text characters through the beam search algorithm and the greedy strategy algorithm to obtain the ship license plate recognition result. Compared with the pure greedy algorithm, the beam search considers more candidates, thereby increasing the probability of finding a better solution, but it is not as computationally intensive as the full search. By limiting the number of paths evaluated at any given time (i.e., the beam width), the beam search can quickly produce results in practical applications.

[0097] In summary, through a carefully designed two-stage training strategy, this study aims to improve the learning efficiency and performance of the model, while reducing resource consumption and improving the generalization ability of the model by adopting advanced data processing and optimization techniques.

[0098] Corresponding to the above-mentioned embodiment of a complex license plate recognition method based on mask reconstruction and semantic enhancement learning, the present invention also provides an embodiment of a complex license plate recognition system based on mask reconstruction and semantic enhancement learning. The implementation process of each module in the system can refer to the specific implementation steps of the above-mentioned embodiment of a complex license plate recognition method based on mask reconstruction and semantic enhancement learning. The system includes:

[0099] The image block sequence preprocessing module is used to obtain the ship plate image, divide it into image block sequences, and input it into the mask pre-trained encoder to integrate features of different depths;

[0100] The cross-modal feature output module is used to convert the feature space of the visual vocabulary of the ship license plate image into the latent feature space of the visual semantic vector based on the visual encoder; extract the features of the visual semantic vector based on the text encoder and convert them to the same feature space as the mask network layer for feature fusion;

[0101] A gated feature fusion module is used to fuse the image features obtained by the image block sequence preprocessing module and the image features obtained by the cross-modal feature output module using a gated fusion operation;

[0102] The feature storage and global feature extraction module is used to use the gated fused features as memory units, and extract and map the global features obtained by the cross-modal feature output module, so that the hidden feature dimension is consistent with the feature dimension of the memory unit;

[0103] The cross-modal feature contrast learning module is used to compare the features in the feature storage and global feature extraction module with the global text features output by the text encoder, enhance the semantics of the features output by the mask pre-trained encoder, and splice the global features and local features output by the mask pre-trained encoder into the decoder for decoding;

[0104] The reverse decoding module is used to generate a mask matrix based on the memory unit. The decoder generates reverse pairs and decodes them in combination with the position information to obtain the ship license plate recognition result.

[0105] Corresponding to the aforementioned embodiment of a complex license plate recognition method based on mask reconstruction and semantic enhancement learning, the present invention also provides an embodiment of a complex license plate recognition device based on mask reconstruction and semantic enhancement learning.

[0106] See also Figure 5 An embodiment of the present invention provides a complex license plate recognition device based on mask reconstruction and semantic enhancement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a complex license plate recognition method based on mask reconstruction and semantic enhancement learning in the above embodiment.

[0107] The embodiment of a complex license plate recognition device based on mask reconstruction and semantic enhancement learning provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5As shown in FIG. 1 , a hardware structure diagram of a complex ship plate recognition device based on mask reconstruction and semantic enhancement learning provided by the present invention is provided for any device with data processing capability, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0108] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0109] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0110] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning in the above embodiment is implemented.

[0111] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0112] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning is implemented.

[0113] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A complex ship plate recognition method based on mask reconstruction and semantic enhancement learning, characterized in that: The method comprises the following steps: (1) Obtain a ship plate image, divide it into a sequence of image blocks, input it into a mask pre-trained encoder, and integrate features of different depths; (2) Based on the visual encoder, the feature space of the visual vocabulary of the ship license plate image is converted into the latent feature space of the visual semantic vector; based on the text encoder, the features of the visual semantic vector are extracted and converted into the same feature space as the mask network layer for feature fusion; (3) fusing the image features obtained in step (1) and the image features obtained in step (2) using a gated fusion operation; (4) The gated fused features are saved as feature memory units, and the global features obtained in step (2) are extracted and spatially mapped so that the hidden feature dimension is consistent with the feature dimension of the memory unit; (5) Compare the features in step (4) with the global text features output by the text encoder to enhance the semantics of the features output by the visual encoder; (6) Based on the memory unit, the decoder generates a mask matrix by generating reverse pairs and decodes it in combination with the position information to obtain the license plate recognition result.

2. According to claim 1, a complex ship plate recognition method based on mask reconstruction and semantic enhancement learning is characterized in that: In step (1), the mask pre-training process is as follows: the ViT encoder is used to encode visual features, the decoder receives the latent representation as input, and reconstructs the entire image through a series of Transformer layers, and performs visual feature loss optimization for training; during pre-training, the position embeddings of the encoder and decoder are frozen.

3. The complex ship plate recognition method based on mask reconstruction and semantic enhancement learning according to claim 1 is characterized in that: In step (2), the feature transformation of the visual encoder and the text encoder is implemented by a learnable feature adapter module, which consists of a series of layers: a projection head, a linear layer, followed by a ReLU activation layer, then another linear layer, and finally a ReLU activation layer.

4. According to claim 1, a complex ship plate recognition method based on mask reconstruction and semantic enhancement learning is characterized in that: In step (5), InfoNCE is used as the loss for contrastive learning, and global visual features and text features are obtained through the visual encoder and the text encoder; the global visual features are used as the output of the cls token, and a projection is further used to align the dimensions of the text features extracted by the text encoder; the cosine similarity of the visual and text feature pairs of the same ship plate is maximized, while the similarity of the unmatched pairs is minimized, and the visual encoder is enhanced in the ship plate recognition scenario through the guidance of semantic labels.

5. The complex ship plate recognition method based on mask reconstruction and semantic enhancement learning according to claim 1 is characterized in that: In step (6), different reverse order arrangements correspond to different character decoding positions, and the specific arrangement is realized by the attention mask matrix. In the reverse order decoding process, the label is first decoded, and then the information is inferred by combining the label with the information output by the encoder and the attention mask matrix. The inference result is further inferred by combining the label, and finally the reverse order decoding process is completed.

6. The complex ship plate recognition method based on mask reconstruction and semantic enhancement learning according to claim 1 is characterized in that: The pre-trained text encoder is used to iterate the correction of the results. Specifically, the character features of the decoded information are decoded, and the decoded character string is input into the text encoder to obtain the text feature. This feature is merged with the feature of the decoded information to obtain a new feature, and then the optimized decoding of the final text output is performed again.

7. A complex ship plate recognition system based on mask reconstruction and semantic enhancement learning, characterized in that: The system includes: The image block sequence preprocessing module is used to obtain the ship plate image, divide it into image block sequences, and input it into the mask pre-trained encoder to integrate features of different depths; The cross-modal feature output module is used to convert the feature space of the visual vocabulary of the ship license plate image into the latent feature space of the visual semantic vector based on the visual encoder; extract the features of the visual semantic vector based on the text encoder and convert it to the same feature space as the mask network layer for feature fusion; A gated feature fusion module is used to fuse the image features obtained by the image block sequence preprocessing module and the image features obtained by the cross-modal feature output module using a gated fusion operation; The feature storage and global feature extraction module is used to store the gated fused features as feature memory units, and extract and spatially map the global features obtained by the cross-modal feature output module, so that the hidden feature dimension is consistent with the feature dimension of the memory unit; The cross-modal feature contrast learning module is used to compare the features in the feature storage and global feature extraction module with the global text features output by the text encoder to enhance the semantics of the features output by the visual encoder. The reverse decoding module is used to generate a mask matrix based on the memory unit. The decoder generates reverse pairs and decodes them in combination with the position information to obtain the ship license plate recognition result.

8. A complex ship plate recognition device based on mask reconstruction and semantic enhancement learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a complex ship license plate recognition method based on mask reconstruction and semantic enhancement learning as described in any one of claims 1 to 6 is implemented.

Images