Ship name intelligent identification method

Abstract

The invention provides a ship name intelligent identification method. The method comprises the following steps: acquiring a ship image, a ship name identification sample set in multiple languages and a ship name identification sample set in multiple languages; performing feature extraction in the final data set, designing a self-supervised learning model, and performing comparative learning pre-training on the self-supervised learning model by using the extracted features; a hybrid model based on the deep convolutional neural network and Transform is constructed; designing tasks, and then constructing a multi-task sharing model according to feature vectors output by the hybrid model; constructing a graph neural network by using ship name region features and character feature vectors; and taking the feature representation output by the graph neural network as the input of a lightweight detection model, and identifying the ship name. According to the method, the problems of identification precision, adaptability and real-time performance in ship name identification are solved, and the robustness and accuracy of the system under the complex environment and multi-language support are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the field of intelligent recognition, and in particular relates to a method for intelligent recognition of ship names. Background Art

[0002] In the modern shipping industry, accurate recognition of ship names plays an important role in port management, ship monitoring and navigation safety. However, the complexity and diversity of ship name recognition make it a technical challenge. Traditional ship name recognition methods mainly rely on manual monitoring and simple image processing technology, which show obvious limitations when facing challenges such as different ship name identification styles, complex backgrounds, changing environmental conditions and multi-language support. Specifically, traditional methods often have problems in the following aspects:

[0003] Diverse ship name identification styles: Ship names may use a variety of different fonts, colors, sizes and layouts. Traditional image processing methods are difficult to adapt to this diversity, resulting in low recognition rates.

[0004] Changing environmental conditions: The visibility and clarity of a ship’s name and logo will be affected in different seasons, time periods, and weather conditions. Traditional static image processing methods have difficulty coping with the challenges brought by changes in lighting, angles, and backgrounds, often resulting in false positives and false negatives.

[0005] Multi-language support: With the popularity of international shipping, ship names contain characters from multiple languages, such as Chinese, English and other languages. Traditional methods have significant deficiencies in multi-language recognition and it is difficult to accurately identify ship names in non-local languages.

[0006] Complex background: The background of the ship name logo may contain complex patterns, colors, stains and other interferences, which further increases the difficulty of recognition. Traditional methods have low recognition accuracy in complex backgrounds.

[0007] Real-time requirements: Real-time identification of ship names is crucial to improving port management efficiency and navigation safety during the process of ships entering and leaving ports and sailing. However, the processing speed of traditional methods often cannot meet real-time requirements.

[0008] In response to the above problems, the existing technologies mainly improve the recognition effect by improving image processing algorithms and enhancing computing power, but there are still obvious shortcomings. These methods still cannot guarantee high accuracy and high robustness when dealing with complex practical application scenarios. Specifically, the existing technologies mainly have the following shortcomings:

[0009] Insufficient recognition accuracy: When faced with diverse ship name logo styles and complex backgrounds, the recognition accuracy of traditional methods is low.

[0010] Poor adaptability: It has poor adaptability to different environmental conditions and multi-language support, and is difficult to apply in changing actual scenarios.

[0011] Lack of real-time performance: Traditional methods have a slow processing speed and cannot meet the needs of real-time recognition. Summary of the invention

[0012] The purpose of this invention is to propose a method for intelligent recognition of ship names. By innovatively combining deep convolutional neural networks, Transformers and graph neural networks, and introducing self-supervised learning, multi-task joint training, Attention mechanism, real-time processing and dynamic adjustment technologies, it not only solves the recognition accuracy, adaptability and real-time problems of existing technologies in ship name recognition, but also significantly improves the robustness and accuracy of the system in complex environments and multi-language support. This solution has broad prospects and potential commercial value in practical applications.

[0013] In order to achieve the above object, the present invention provides a method for intelligently identifying ship names, the method comprising:

[0014] S1. Collect ship images and a sample set of ship name identification in multiple languages ​​and a sample set of ship name identification in multiple languages, then perform data enhancement on the ship images and the sample set of ship name identification in multiple languages, perform quality control and annotation verification on the enhanced ship image data, perform multimodal data fusion on the verified ship image data and the enhanced sample set of ship name identification in multiple languages, and obtain the final data set and labels;

[0015] S2. Extract features from the final data set, design a self-supervised learning model, use the extracted features to perform comparative learning pre-training on the self-supervised learning model, and then perform multi-task training in combination with labels to output a high-quality feature representation data set after training;

[0016] S3. Build a hybrid model based on deep convolutional neural network and Transformer, train the hybrid model using a high-quality feature representation dataset, and obtain the hybrid model output feature vector;

[0017] S4. Design a task, which includes a ship name detection subtask and a character recognition subtask, then construct a multi-task sharing model according to the feature vector output by the hybrid model, input the output of the multi-task sharing model as input into the ship name detection network to extract the ship name region features, and then input the ship name region features in the ship name detection network into the character recognition network to extract the character feature vector;

[0018] S5. Use the regional features of the ship name and the character feature vector to construct a graph neural network, and optimize the relationship between the regional features of the ship name and the character feature vector through the graph neural network and the multi-head self-attention mechanism;

[0019] S6. The feature representation output by the graph neural network is used as the input of the lightweight detection model. The lightweight detection model is used to perform preliminary detection of the real-time ship name area and the preliminary detected ship name area is added to the feature representation output by the graph neural network. The feature weights are then dynamically adjusted according to environmental conditions to perform feature enhancement. The enhanced features are input into the character recognition network for character recognition to obtain the recognized ship name characters.

[0020] Further, in S1, the data enhancement includes illumination change enhancement, angle change enhancement, and background replacement enhancement;

[0021] The multimodal data fusion combines each image I i and its corresponding metadata M i The fusion is expressed as follows:

[0022] F i =concat(φ(I i ),ψ(M i ))

[0023] Among them, F i represents the joint feature representation, φ(I i ) represents the features extracted from the image, ψ(M i ) represents the features extracted from metadata, concat represents the concatenation operation to obtain the joint feature representation; using the joint feature representation F i Combined with high-quality annotated datasets Perform fusion data enhancement, where the fusion formula is expressed as follows:

[0024]

[0025] Generate a rich set of augmented samples

[0026] Furthermore, the extracted features are used to perform contrastive learning pre-training on the self-supervised learning model, specifically including:

[0027] For each pair of enhanced images (I i ,I′ i ), define the contrast loss function so that the feature representations of different enhanced images of the same sample are as similar as possible, and the feature representations of different samples are as different as possible. The specific formula is:

[0028]

[0029] Among them, z i and z′ i Represents the same image I iThe feature representation of different enhanced versions, sim(·,·) represents the similarity function, and τ represents the temperature parameter;

[0030] For each original image I i , generating two different data augmentation versions I i,a and I i,b , use the ResNet-based encoder f(·) to extract feature representation, obtain feature vector, and perform L2 normalization on the feature vector;

[0031] Using contrastive loss function Train the encoder f(·) and optimize it to minimize the loss function, which is expressed as follows:

[0032]

[0033] Perform feature extraction on all enhanced images to generate feature representations, and cluster the feature representations using the K-means clustering algorithm to generate pseudo labels;

[0034] A multi-task learning framework is used to combine the pseudo-label classification task and the ship name recognition task for joint training. The multi-task loss function of the multi-task learning framework is:

[0035]

[0036] in, represents the pseudo-label classification loss, represents the ship name recognition loss, λ represents the weight hyperparameter;

[0037] Among them, the pseudo-label classification loss is expressed as follows:

[0038]

[0039] Among them, σ k represents the k-th class output probability of the classifier, K represents the total number of categories, and N represents the total number of samples;

[0040] Vessel name recognition loss, expressed as follows:

[0041]

[0042] Among them, P(R i |I i ) represents a given image I i The predicted probability of the ship name area.

[0043] Furthermore, the pre-trained ResNet50 is used as the basic model of the deep convolutional neural network to extract the local feature map F of the image. i , the local feature map F iExpand it into a two-dimensional matrix and then flatten it to get the feature sequence S i , define the position encoding matrix P∈R (H×W)×C , by taking the position encoding matrix P∈R (H×W)×C Add to feature sequence S i In the above equation, we get the feature sequence S′ i , using a multi-head self-attention mechanism and a feedforward neural network to construct a Transformer encoder for the feature sequence S′ i Processing, get the output T i , the output T of the Transformer encoder layer i After the fully connected layer, the final output feature vector O is obtained i .

[0044] Furthermore, the multi-task shared model uses the hybrid model trained in step S3 to extract image features, and the output of the shared feature extraction network is F i , the shared feature F i Input to the ship name detection subnetwork, predict the bounding box coordinates and category of the ship name area, and then use the ship name area feature R predicted by the ship name detection subnetwork i Input to the character recognition sub-network to predict the ship name character sequence.

[0045] Furthermore, the output D of the ship name detection subnetwork i , which is expressed as follows:

[0046] D i =DetectionHead(F i )

[0047] The loss functions for bounding box regression and category prediction are and It is expressed as follows:

[0048]

[0049]

[0050] Among them, b j and denote the true and predicted bounding box coordinates, c j and Represent the true and predicted class probabilities respectively;

[0051] The output C of the character recognition subnetwork i , which is expressed as follows:

[0052] C i =RecognitionHead(R i ))

[0053] The loss function for character recognition is the connectivity temporal classification loss It is expressed as follows:

[0054]

[0055] Among them, y i represents the true character sequence label, P(y i |C i ) represents a given feature C i The predicted probability of a character sequence.

[0056] Furthermore, the loss function of the ship name detection subnetwork and the loss of the character recognition subnetwork are combined to obtain the total multi-task loss function after training: It is expressed as follows:

[0057]

[0058] Among them, α, β, and γ represent the weight hyperparameters of the task loss, and the optimal values ​​are determined by hyperparameter tuning.

[0059] Furthermore, the characters detected in the ship name area are constructed into a graph neural network, with the characters as nodes of the graph and the relationships between characters as edges of the graph. Then, for each character node v i , use the output feature vector of the character recognition subnetwork to initialize the node features, use the graph convolutional network to update the node features, and update the node features through information propagation of adjacent nodes; the update rule of the graph convolutional network is expressed as follows:

[0060]

[0061] in, Represents node v i The feature vector at level 1, Represents node v i The set of neighbor nodes, d i and d j Respectively represent the node v i and v j Degree, W (l) represents the weight matrix of the first layer, σ represents the activation function;

[0062] On the basis of the graph convolution layer, a multi-head self-attention mechanism is used to further fuse node features and capture the global correlation between characters. The multi-head self-attention mechanism is defined as follows:

[0063]

[0064] in, represents the learnable weight matrix;

[0065] Among them, the attention calculation formula is:

[0066]

[0067] Multiple heads The output of is concatenated and linearly transformed to get the final output:

[0068]

[0069] Among them, W O is the output weight matrix.

[0070] Furthermore, using the cross entropy loss function The graph neural network is trained to minimize the difference between the predicted output and the true label, the cross entropy loss function It is expressed as follows:

[0071]

[0072] Among them, y i Represents image I i The true label, O G Represents the model prediction output;

[0073] In order to prevent the model from overfitting, the L2 regularization term is introduced It is expressed as follows:

[0074]

[0075] Among them, λ is the regularization coefficient, W k are learnable parameters in the model.

[0076] Defining the total loss function of the graph neural network for:

[0077]

[0078] The Adam optimizer is used to optimize the parameters of the graph neural network. The optimization goal is to minimize the total loss function, which is expressed as follows:

[0079]

[0080] Among them, θ represents all learnable parameters of the graph neural network.

[0081] Furthermore, during real-time recognition, a dynamic weight adjustment mechanism is designed to adjust feature weights according to real-time lighting, angle, and background changes, and define the weight adjustment function T adjust :

[0082] (α,β,γ)=T adjust (L i,j ,A i,j ,C i,j )

[0083] Among them, T adjust It is implemented through a multi-layer perceptron MLP, which is expressed as follows:

[0084] T adjust (L i,j ,A i,j ,C i,j )=MLP(concat(L i,j ,A i,j ,C i,j ))

[0085] Among them, α, β, γ represent the feature weights adjusted by real-time illumination, angle, and background changes, and L i,j ,A i,j ,C i,j Features that represent real-time lighting, angle, and background changes.

[0086] The beneficial technical effects of the present invention are at least as follows:

[0087] (1) The present invention combines a hybrid model of CNN and Transformer, uses CNN to extract local features, and then uses Transformer to capture global features and contextual relationships. This innovative architecture can not only process the details of ship name identification, but also enhance the recognition ability of complex backgrounds and diverse ship names, thereby greatly improving the recognition accuracy.

[0088] (2) The present invention uses self-supervised learning technology to perform pre-training on unlabeled data, generate pseudo labels, and reduce data annotation costs. At the same time, a multi-task joint training model is designed to combine the three tasks of ship name detection, character recognition, and background segmentation to improve the generalization ability and accuracy of the model, especially in different environmental conditions and multi-language support.

[0089] (3) The present invention uses GNN to capture the spatial relationship and structural information between characters, and enhances the relevance and contextual information between characters through the Attention mechanism. This design not only improves the accuracy of character detection in complex backgrounds, but also enhances the adaptability of the model in multilingual environments.

[0090] (4) In order to meet the real-time requirements, the present invention uses lightweight YOLO or SSD to perform preliminary detection of the ship name area to ensure rapid response of the recognition process. At the same time, an adaptive feature enhancement module is designed to dynamically adjust the feature weight according to real-time lighting, angle and background, optimize the recognition effect, and ensure high-precision recognition under different environmental conditions.

[0091] (5) The present invention designs a multi-language character recognition module that supports ship name recognition in Chinese, English and other major languages. Through a character normalization algorithm, multiple lines of printed ship name identification characters are normalized to the same line, improving the recognizability of the characters and thus enhancing the accuracy of multi-language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0093] Figure 1 This is a flow chart of the method for intelligently identifying ship names of the present invention. DETAILED DESCRIPTION

[0094] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0095] like Figure 1 As shown, the method for intelligently identifying ship names provided by an embodiment of the present invention comprises the following steps S1-S6:

[0096] S1. Collect ship images and a sample set of ship name identification in multiple languages ​​and a sample set of ship name identification in multiple languages, then perform data enhancement on the ship images and the sample set of ship name identification in multiple languages, perform quality control and annotation verification on the enhanced ship image data, perform multimodal data fusion on the verified ship image data and the enhanced sample set of ship name identification in multiple languages, and obtain the final data set and labels.

[0097] Specifically, we obtain ship images from surveillance cameras, drones, satellite images and other sources at different ports around the world, covering a variety of ship name identification styles, resolutions, backgrounds, seasons, time periods, lighting, angles and other factors. Each image is captured using I i Indicates that the overall sample set is express.

[0098] Record metadata M for each image i, including shooting time, weather conditions, shooting angle, etc., for subsequent data enhancement and model training. Collection formula:

[0099]

[0100] Among them I i For the image, M i For metadata.

[0101] Furthermore, the StyleGAN2 generator G is used to generate new images. For each original image I i , generate the corresponding enhanced image I′ i , generate an image set Where M≥N.

[0102] Furthermore, the generator G and discriminator D are trained by minimizing the following loss function:

[0103]

[0104] Among them, p data is the actual data distribution, p z is the potential space distribution.

[0105] Only then use the perceptual loss function L p Calculate the perceived similarity between the generated image and the real image to ensure the quality of the generated image. The formula is:

[0106]

[0107] Among them, φ represents the pre-trained feature extraction network, usually selecting a layer output of the VGG network.

[0108] Generate formula:

[0109]

[0110] Furthermore, the multilingual ship name samples are expanded.

[0111] Specifically, a sample set of ship name identifications in multiple languages ​​(such as Chinese, English, French, etc.) is collected. Each sample L j Contains language annotations and image information, and the set is represented as

[0112] These samples were preprocessed using OCR (optical character recognition) technology to extract the text content and ship name area and record them as R j .

[0113] Mark the specific language and content of each sample to ensure the accuracy and availability of multilingual samples.

[0114]

[0115] Furthermore, data enhancement is performed for lighting, angle, and background changes.

[0116] Specifically, for each image I i , apply the lighting transformation function T light , generate enhanced image I i,light , whose brightness, contrast and color balance are different. The formula is:

[0117] I i,light (x,y)=βI i (x,y)+γ(I i (x,y)-μ i )+δ

[0118] Among them, β is the brightness adjustment parameter, γ is the contrast adjustment parameter, δ is the color balance parameter, μ i For image I i The average pixel value.

[0119] Apply the angle transformation function T angle , simulate different shooting angles and generate image I i,angle The formula is:

[0120] I i,angle (x′,y′)=I i (cos(θ)x+sin(θ)y,-sin(θ)x+cos(θ)y)

[0121] Where θ is the rotation angle.

[0122] Replace the background of the ship name area to generate a complex background image I i,background The formula is:

[0123] I i,background =αI i +(1-α)B i

[0124] Among them, B i is the background image, and α is the mixing coefficient matrix, which is used to control the degree of fusion between the ship name area and the background. Enhancement formula:

[0125]

[0126] Furthermore, a semi-automatic annotation tool is used to annotate the ship name area of ​​all images. The annotation information R of each image i Includes the coordinates and dimensions of the ship name area, expressed as: R i={(x,y,w,h)}, where (x,y) is the coordinate of the upper left corner, w is the width, and h is the height.

[0127] Furthermore, manual verification and cross-validation are performed to ensure the accuracy and consistency of the labeled data. Manual corrections are made to the labeling errors to form a high-quality labeled data set. Marking formula:

[0128]

[0129] Furthermore, for each image I i and its corresponding metadata M i , using multimodal fusion technology to fuse visual information and metadata information. Define the joint feature representation F i , the formula is:

[0130] F i =concat(φ(I i ),ψ(M i ))

[0131] Among them, φ(I i ) represents the features extracted from the image, ψ(M i ) represents the features extracted from metadata, and the joint feature representation is obtained through concatenation operation.

[0132] Using the joint feature representation F i Perform data enhancement to generate a richer set of enhanced samples Fusion formula:

[0133]

[0134] S2. Perform feature extraction in the final data set, design a self-supervised learning model, pre-train the self-supervised learning model with comparative learning using the extracted features, and then perform multi-task training in combination with the labels to output a high-quality feature representation data set after training.

[0135] Specifically, the self-supervised learning task uses contrastive learning to learn feature representation through instance discrimination tasks. i ,I′ i ), define the contrast loss function so that the feature representations of different enhanced images of the same sample are as similar as possible, and the feature representations of different samples are as different as possible. The specific formula is:

[0136]

[0137] Among them, z i and z′ i For the same image Ii The feature representation of different enhanced versions of , sim(·,·) is the similarity function (such as cosine similarity), and τ is the temperature parameter.

[0138] Furthermore, for each original image I i , generating two different data augmentation versions I i,a and I i,b The enhancement methods include random cropping, color jitter, Gaussian blur, grayscale transformation, etc. The data enhancement function T acts on the image I i , generate an enhanced image:

[0139] I i,a =T a (I i ),I i,b =T b (I i )

[0140] Among them, T a and T b are different data augmentation functions.

[0141] The ResNet-based encoder f(·) is used to extract the feature representation and obtain the feature vector:

[0142] z i,a =f(I i,a ),z i,b =f(I i,b )

[0143] Perform L2 normalization on the feature vector:

[0144]

[0145] Furthermore, using the contrast loss function The encoder f(·) is trained to minimize the loss function so that the feature representations of different enhanced versions of the same image are as close as possible, and the feature representations of different images are as far apart as possible. The optimization process is as follows:

[0146]

[0147] Use stochastic gradient descent (such as Adam) for optimization and update the encoder parameters.

[0148] Furthermore, based on the contrastive learning pre-training, all enhanced images are feature extracted to generate feature representations. The feature representations are clustered using the K-means clustering algorithm to generate pseudo labels. Define the pseudo label set where p i For image I iPseudo labels. K-means clustering algorithm:

[0149]

[0150] Among them, K is the number of cluster centers, z i For image I i The feature representation of .

[0151] Use a multi-task learning framework to combine the pseudo-label classification task and the ship name recognition task for joint training. Define the multi-task loss function as:

[0152]

[0153] in, is the pseudo-label classification loss, is the ship name recognition loss, and λ is the weight hyperparameter.

[0154] Pseudo-label classification loss:

[0155]

[0156] Among them, σ k Output probability for the kth class of the classifier.

[0157] Vessel name identification loss:

[0158]

[0159] Among them, P(R i |I i ) is a given image I i The predicted probability of the ship name area.

[0160] S3. Build a hybrid model based on deep convolutional neural network and Transformer, train the hybrid model using a high-quality feature representation dataset, and obtain the output feature vector of the hybrid model.

[0161] Specifically, the pre-trained ResNet50 is used as the deep convolutional neural network (CNN) basic model to extract the local features of the image. Input image I i The feature map F is obtained through the CNN model i :

[0162] F i =ResNet50(I i )

[0163] Feature map F i The dimension is (C, H, W), where C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0164] Furthermore, the feature map F i Convert to feature sequence S i , in order to input into the Transformer model. First, the feature map is expanded into a two-dimensional matrix and then flattened:

[0165] S i = reshape(F i ,(H×W,C))

[0166] Feature sequence S i The dimension is (H×W,C), and the length of the feature vector at each position is C.

[0167] Furthermore, for the feature sequence S i Add position encoding to preserve spatial position information. Define the position encoding matrix By adding the positional encoding matrix to the feature sequence:

[0168] S′ i =S i +P

[0169] Each row of the position encoding matrix P represents the encoding information of a specific position, and the position encoding is calculated using sine and cosine functions:

[0170]

[0171] Among them, pos represents the position and i represents the feature dimension index.

[0172] Furthermore, a Transformer encoder is constructed using a multi-head self-attention mechanism and a feedforward neural network to process the feature sequence. The output of the Transformer encoder layer is defined as T i :

[0173] T i = TransformerEncoder(S′ i )

[0174] The calculation formula of the multi-head self-attention mechanism is:

[0175]

[0176] Where Q, K, and V represent query, key, and value matrices, respectively, and d k Represents the dimension of the key vector. Multiple self-attention heads are used in parallel to capture information from different subspaces:

[0177] MultiHead(Q,K,V)=concat(head1,head2,…,head h )WO

[0178]

[0179] in, and W O is the learnable weight matrix.

[0180] Furthermore, the output T of the Transformer encoder layer i After the fully connected layer, the final output feature vector O is obtained i , which is used for the subsequent ship name recognition task. The weight matrix of the fully connected layer is defined as W o , with a bias of b o :

[0181] O i =FC(T i )=T i W o +b o

[0182] Output feature vector O i The dimension corresponds to the number of categories in the ship name recognition task.

[0183] Furthermore, in order to prevent the model from overfitting, a regularization term is introduced. The L2 regularization term is defined as:

[0184]

[0185] Among them, λ is the regularization coefficient, W k are learnable parameters in the model.

[0186] The model is trained using the cross entropy loss function to minimize the difference between the predicted output and the true label. Define the total loss function for the hybrid model for:

[0187]

[0188] Among them, the cross entropy loss function for:

[0189]

[0190] Among them, y i For image I i The true label, O i Predict output for the model.

[0191] Furthermore, the Adam optimizer is used to optimize the model parameters and update the weights of the CNN and Transformer encoders. The optimization objective is defined as minimizing the total loss function:

[0192]

[0193] Among them, θ is all the learnable parameters of the model.

[0194] S4. Design a task, which includes a ship name detection subtask and a character recognition subtask, and then build a multi-task sharing model based on the feature vector output by the hybrid model, input the output of the multi-task sharing model as input into the ship name detection network to extract the ship name area features, and then input the ship name area features in the ship name detection network into the character recognition network to extract the character feature vector.

[0195] Specifically, the ship name recognition task is decomposed into two subtasks: ship name detection and character recognition. Through the multi-task learning framework, the models of the two tasks are jointly trained to improve the overall performance of the model.

[0196] Task 1: Ship name detection, the goal is to detect the ship name area in the image.

[0197] Task 2: Character recognition, the goal is to recognize the characters in the detected ship name area.

[0198] Furthermore, the multi-task model architecture is to share the underlying feature extraction network, using the deep convolutional neural network and Transformer hybrid model trained in step 3 to extract image features. The output of the shared feature extraction network is defined as F i :

[0199] F i = Transformer(I i )

[0200] Furthermore, task-specific subnetworks are designed for each task. The ship name detection subnetwork is used to predict the bounding box of the ship name area, and the character recognition subnetwork is used to recognize the characters in the ship name area.

[0201] Specifically, the ship name detection subnetwork is to share the feature F i Input to the ship name detection subnetwork to predict the bounding box coordinates and category of the ship name area. Define the output of the ship name detection subnetwork as D i :

[0202] D i =DetectionHead(F i )

[0203] The loss functions for bounding box regression and category prediction are and

[0204]

[0205]

[0206] Among them, b j and are the true and predicted bounding box coordinates, c j and are the true and predicted class probabilities, respectively.

[0207] The character recognition sub-network is to transform the ship name region feature R predicted by the ship name detection sub-network into i Input to the character recognition sub-network to predict the ship name character sequence. Define the output of the character recognition sub-network as C i :

[0208] C i =RecognitionHead(R i )

[0209] The loss function for character recognition is the Connectivity Temporal Classification (CTC) loss

[0210]

[0211] Among them, y i is the true character sequence label, P(y i |C i ) is a given feature C i The predicted probability of a character sequence.

[0212] Furthermore, the ship name detection loss and character recognition loss are combined to define the total loss function of the multi-task loss after training: for:

[0213] Among them, α, β, and γ are the weight hyperparameters of the task loss, and the optimal values ​​are determined by hyperparameter tuning.

[0214] Furthermore, stochastic gradient descent (such as Adam optimizer) is used for multi-task joint training, and the optimization goal is to minimize the total loss function

[0215]

[0216] Here, θ is all the learnable parameters of the shared feature extraction network and the task-specific sub-network.

[0217] S5. Use the ship name regional features and character feature vectors to construct a graph neural network, and optimize the relationship between the ship name regional features and character feature vectors through the graph neural network and multi-head self-attention mechanism.

[0218] Specifically, the characters in the ship name region detected in step 4 are constructed into a graph structure, with the characters as nodes of the graph and the relationships between the characters as edges of the graph. Let V = {v1, v2, ..., v n} is a node set, each node v i Represents a character; E = {e ij} is an edge set, each edge e ij Represents the character v i and v j The relationship between them. The graph is represented by G = (V, E). For each character node v i , use the output feature vector of the character recognition subnetwork in step 4 to initialize the node features. Define the node feature vector as h i :

[0219] h i =C i

[0220] Among them, C i is the output feature vector of the character recognition sub-network.

[0221] Furthermore, a graph convolutional network (GCN) is used to update node features, and node features are updated by propagating information from adjacent nodes. The update formula for the graph convolution layer is defined as:

[0222]

[0223] in, For node v i The feature vector at level 1, For node v i The set of neighbor nodes, d i and d j The nodes v i and v j Degree, W (l) is the weight matrix of the first layer, and σ is the activation function (such as ReLU).

[0224] Furthermore, based on the graph convolution layer, a multi-head self-attention mechanism is used to further fuse node features and capture the global correlation between characters. The calculation formula for the multi-head self-attention mechanism is defined as:

[0225]

[0226] in, is a learnable weight matrix. The attention calculation formula is:

[0227]

[0228] The outputs of multiple heads are concatenated and linearly transformed to obtain the final output:

[0229]

[0230] Among them, W O is the output weight matrix.

[0231] Furthermore, the node features updated by the multi-head self-attention mechanism Perform global aggregation to obtain the graph-level feature representation h G :

[0232]

[0233] Among them, GlobalPool can be global average pooling or global maximum pooling.

[0234] The graph-level features are represented as h G Input to the fully connected layer for ship name recognition and classification. Define the output of the fully connected layer as O G :

[0235] O G =FC(h G )

[0236] Among them, FC is the fully connected layer operation.

[0237] Furthermore, using the cross entropy loss function The model is trained to minimize the difference between the predicted output and the true label:

[0238]

[0239] Among them, y i For image I i The true label, O G Predict output for the model.

[0240] To prevent the model from overfitting, the L2 regularization term is introduced

[0241]

[0242] Among them, λ is the regularization coefficient, W k are learnable parameters in the model.

[0243] Define the total loss function for:

[0244]

[0245] The Adam optimizer is used to optimize the model parameters, and the optimization goal is to minimize the total loss function:

[0246]

[0247] Among them, θ is all the learnable parameters of the model.

[0248] S6. The feature representation output by the graph neural network is used as the input of the lightweight detection model. The lightweight detection model is used to perform preliminary detection of the real-time ship name area and the preliminary detected ship name area is added to the feature representation output by the graph neural network. The feature weights are then dynamically adjusted according to environmental conditions to perform feature enhancement. The enhanced features are input into the character recognition network for character recognition to obtain the recognized ship name characters.

[0249] Specifically, a lightweight detection model (such as YOLO or SSD) is used to detect the input image I i Perform preliminary detection of the ship name area to ensure a fast response in the recognition process. The model outputs the bounding box set B of the ship name area. i :

[0250] B i =DetectionModel(I i )

[0251] Bounding Box B i Contains the coordinates (x i,j ,y i,j ), width w i,j , height h i,j and the confidence level c i,j , expressed as:

[0252]

[0253] Where M is the number of detected ship name regions.

[0254] Furthermore, the detection loss By bounding box regression loss and classification loss composition.

[0255] Bounding Box Regression Loss:

[0256]

[0257] Among them, b i,j represents the true bounding box coordinates, Represents the predicted bounding box coordinates.

[0258] Classification loss:

[0259]

[0260] Among them, c i,j is the true category label, is the predicted class probability.

[0261] Feature enhancement loss Used to measure the feature F′ adjusted by the adaptive feature enhancement module i,j With the original feature F i,j The perceived difference between:

[0262]

[0263] Here, φ represents the feature map extracted by the pre-trained convolutional neural network.

[0264] Character recognition loss Connectivity Temporal Classification (CTC) loss is used to ensure the accuracy of character sequence recognition.

[0265]

[0266] Among them, y i,j is the true character sequence label, P(y i,j |C i,j ) is a given input feature F′ i,j The probability of the character sequence predicted when .

[0267] Furthermore, for the detected ship name area B i Perform feature extraction and use the CNN-Transformer hybrid model in step 3 to generate preliminary features F i,j :

[0268] F i,j =CNN-Transformer(I i [B i ])

[0269] On this basis, an adaptive feature enhancement module is introduced to dynamically adjust the feature weight according to environmental conditions (such as lighting, angle, background). Define the lighting feature L i,j , Angle feature A i,j and background features C i,j , total feature F′ i,j for:

[0270] F′ i,j =αL i,j +βA i,j +γC i,j +δF i,j

[0271] Among them, α, β, γ, and δ are weight parameters learned through training.

[0272] Furthermore, a dynamic weight adjustment mechanism is designed to adjust the feature weights according to real-time lighting, angle and background changes. Define the weight adjustment function T adjust :

[0273] (α,β,γ)=T adjust (L i,j ,A i,j ,C i,j )

[0274] Among them, T adjust This is achieved through a multi-layer perceptron (MLP):

[0275] T adjust (L i,j ,A i,j ,C i,j )=MLP(concat(L i,j ,A i,j ,C i,j ))

[0276] Furthermore, using the enhanced feature F′ i,j Perform character recognition and output character sequence C i,j :

[0277] C i,j =CharacterRecognition(F′ i,j )

[0278] Using CTC loss function Ensure accurate recognition of character sequences.

[0279] Furthermore, the overall loss function is defined as To detect loss Feature enhancement loss and character recognition loss The weighted sum of:

[0280]

[0281] Among them, λ1, λ2, λ3 are weight hyperparameters, which are determined by hyperparameter tuning.

[0282] The Adam optimizer is used to optimize the model parameters θ, with the goal of minimizing the overall loss function:

[0283]

[0284] Here, θ is the set of all learnable parameters in the model.

[0285] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0286] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, and will not be repeated here.

[0287] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0288] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0289] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0290] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for intelligently identifying ship names, characterized in that: The method comprises: S1. Collect ship images and a sample set of ship name identification in multiple languages ​​and a sample set of ship name identification in multiple languages, then perform data enhancement on the ship images and the sample set of ship name identification in multiple languages, perform quality control and annotation verification on the enhanced ship image data, perform multimodal data fusion on the verified ship image data and the enhanced sample set of ship name identification in multiple languages, and obtain the final data set and labels; S2. Extract features from the final data set, design a self-supervised learning model, use the extracted features to perform comparative learning pre-training on the self-supervised learning model, and then perform multi-task training in combination with labels to output a high-quality feature representation data set after training; S3. Build a hybrid model based on deep convolutional neural network and Transformer, train the hybrid model using a high-quality feature representation dataset, and obtain the hybrid model output feature vector; S4. Design a task, which includes a ship name detection subtask and a character recognition subtask, then construct a multi-task sharing model according to the feature vector output by the hybrid model, input the output of the multi-task sharing model as input into the ship name detection network to extract the ship name region features, and then input the ship name region features in the ship name detection network into the character recognition network to extract the character feature vector; S5. Use the regional features of the ship name and the character feature vector to construct a graph neural network, and optimize the relationship between the regional features of the ship name and the character feature vector through the graph neural network and the multi-head self-attention mechanism; S6. The feature representation output by the graph neural network is used as the input of the lightweight detection model. The lightweight detection model is used to perform preliminary detection of the real-time ship name area and the preliminary detected ship name area is added to the feature representation output by the graph neural network. The feature weights are then dynamically adjusted according to environmental conditions to perform feature enhancement. The enhanced features are input into the character recognition network for character recognition to obtain the recognized ship name characters.

2. The method for intelligently identifying ship names according to claim 1, characterized in that: In S1, the data enhancement includes illumination change enhancement, angle change enhancement, and background replacement enhancement; The multimodal data fusion combines each image I i and its corresponding metadata M i The fusion is expressed as follows: F i =concat(φ(I i ),ψ(M i )) Among them, F i represents the joint feature representation, φ(I i ) represents the features extracted from the image, ψ(M i ) represents the features extracted from metadata, concat represents the concatenation operation to obtain the joint feature representation; using the joint feature representation F i Combined with high-quality annotated datasets Perform fusion data enhancement, where the fusion formula is expressed as follows: Generate a rich set of augmented samples 3. The method for intelligently identifying ship names according to claim 1, characterized in that: The extracted features are used to perform contrastive learning pre-training on the self-supervised learning model, specifically including: For each pair of enhanced images (I i ,I′ i ), define the contrast loss function so that the feature representations of different enhanced images of the same sample are as similar as possible, and the feature representations of different samples are as different as possible. The specific formula is: Among them, z i and z′ i Represents the same image I i The feature representation of different enhanced versions of , sim(·,·) represents the similarity function, and τ represents the temperature parameter; For each original image I i , generating two different data augmentation versions I i,a and I i,b , use the ResNet-based encoder f(·) to extract feature representation, obtain feature vector, and perform L2 normalization on the feature vector; Using contrastive loss function Train the encoder f(·) and optimize it to minimize the loss function, which is expressed as follows: Perform feature extraction on all enhanced images to generate feature representations, and cluster the feature representations using the K-means clustering algorithm to generate pseudo labels; A multi-task learning framework is used to combine the pseudo-label classification task and the ship name recognition task for joint training. The multi-task loss function of the multi-task learning framework is: in, represents the pseudo-label classification loss, represents the ship name recognition loss, λ represents the weight hyperparameter; Among them, the pseudo-label classification loss is expressed as follows: Among them, σ k represents the k-th class output probability of the classifier, K represents the total number of categories, and N represents the total number of samples; Vessel name recognition loss, expressed as follows: Among them, P(R i |I i ) represents a given image I i The predicted probability of the ship name area.

4. The method for intelligently identifying ship names according to claim 1, characterized in that: Use the pre-trained ResNet50 as the base model of the deep convolutional neural network to extract the local feature map F of the image i , the local feature map F i Expand it into a two-dimensional matrix and then flatten it to get the feature sequence S i , define the position encoding matrix P∈R (H×W)×C , by taking the position encoding matrix P∈R (H×W)×C Add to feature sequence S i In the above equation, we get the feature sequence S′ i , using a multi-head self-attention mechanism and a feedforward neural network to construct a Transformer encoder for the feature sequence S′ i Processing, get the output T i , the output T of the Transformer encoder layer i After the fully connected layer, the final output feature vector O is obtained i .

5. The method for intelligently identifying ship names according to claim 4, characterized in that: The multi-task shared model uses the hybrid model trained in step S3 to extract image features, and the output of the shared feature extraction network is F i , the shared feature F i Input to the ship name detection subnetwork, predict the bounding box coordinates and category of the ship name area, and then use the ship name area feature R predicted by the ship name detection subnetwork i Input to the character recognition sub-network to predict the ship name character sequence.

6. The method for intelligently identifying ship names according to claim 4, characterized in that: The output D of the ship name detection subnetwork i , which is expressed as follows: D i =DetectionHead(F i ) The loss functions for bounding box regression and category prediction are L bbox and L cls , which is expressed as follows: Among them, b j and denote the true and predicted bounding box coordinates, c j and Represent the true and predicted class probabilities respectively; The output C of the character recognition subnetwork i , which is expressed as follows: C i =RecognitionHead(R i )) The loss function for character recognition is the connectivity temporal classification loss It is expressed as follows: Among them, y i represents the true character sequence label, P(y i |C i ) represents a given feature C i The predicted probability of a character sequence.

7. The method for intelligently identifying ship names according to claim 6, characterized in that: The loss function of the ship name detection subnetwork and the loss of the character recognition subnetwork are combined to obtain the total multi-task loss function after training: It is expressed as follows: Among them, α, β, and γ represent the weight hyperparameters of the task loss, and the optimal values ​​are determined by hyperparameter tuning.

8. The method for intelligently identifying ship names according to claim 1, characterized in that: The characters in the detected ship name area are constructed into a graph neural network, with the characters as nodes of the graph and the relationships between characters as edges of the graph. Then, for each character node v i , use the output feature vector of the character recognition subnetwork to initialize the node features, use the graph convolutional network to update the node features, and update the node features through information propagation of adjacent nodes; the update rule of the graph convolutional network is expressed as follows: in, Represents node v i The feature vector at level 1, Represents node v i The set of neighbor nodes, d i and d j Respectively represent the node v i and v j Degree, W (l) represents the weight matrix of the first layer, σ represents the activation function; On the basis of the graph convolution layer, a multi-head self-attention mechanism is used to further fuse node features and capture the global correlation between characters. The multi-head self-attention mechanism is defined as follows: in, represents the learnable weight matrix; The attention calculation formula is: Multiple heads The output of is concatenated and linearly transformed to get the final output: Among them, W O is the output weight matrix.

9. The method for intelligently identifying ship names according to claim 1, characterized in that: Using the cross entropy loss function The graph neural network is trained to minimize the difference between the predicted output and the true label, the cross entropy loss function It is expressed as follows: Among them, y i Represents image I i The true label, O G Represents the model prediction output; In order to prevent the model from overfitting, the L2 regularization term is introduced It is expressed as follows: Among them, λ is the regularization coefficient, W k is the learnable parameter in the model; Defining the total loss function of the graph neural network for: The Adam optimizer is used to optimize the parameters of the graph neural network. The optimization goal is to minimize the total loss function, which is expressed as follows: Among them, θ represents all learnable parameters of the graph neural network.

10. The method for intelligently identifying ship names according to claim 1, characterized in that: When performing real-time recognition, a dynamic weight adjustment mechanism is designed to adjust the feature weights according to real-time lighting, angle and background changes, and define the weight adjustment function T adjust : Among them, T adjust It is implemented through a multi-layer perceptron MLP, which is expressed as follows: T adjust (L i,j ,A i,j ,C i,j )=MLP(concat(L i,j ,A i,j ,C i,j )) Among them, α, β, γ represent the feature weights adjusted by real-time illumination, angle, and background changes, and L i,j ,A i,j ,C i,j Features that represent real-time lighting, angle, and background changes.

Images