Inland ship name identification method combining autoregression and non-autoregression decoding

Abstract

The invention discloses an inland ship name identification method combining autoregression and non-autoregression decoding. The method specifically comprises the following steps: 1, constructing a data set; 2, pre-training a feature extractor; 3, constructing a ship name recognition model: respectively sending ship name image features extracted by the feature extractor into autoregression and non-autoregression decoder branches to obtain two types of character representations; 4, adaptively fusing the two types of character representations through a gating mechanism; 5, constructing three linear classifiers by taking the character representations extracted by the autoregression decoder branch and the non-autoregression decoder branch and the fused character representations as input to obtain three groups of predictions; and 6, by fusing the loss of the autoregression decoder branch and the loss of the non-autoregression decoder branch, respectively constructing character-level and instance-level reweighting weights, and dynamically adjusting the contribution of batch samples to the total loss of the current batch in the back propagation process. According to the method, a tedious training process and dependence on an external language model are eliminated.

Description

Technical Field

[0001] The invention belongs to visual-linguistic multimodal ship name recognition involving dynamic fusion of autoregressive and non-autoregressive character representations, and specifically is an inland river ship name recognition method combining autoregressive and non-autoregressive decoding. Background Art

[0002] Inland waterway transport has the characteristics of low pollution, large transport volume and low energy consumption, and has always had a profound impact on the economic development and cultural dissemination of human society. However, with the increase in the number of inland river ships and the increase in the navigation volume of inland river transport, the regulatory pressure has become more serious, mainly manifested in the attempt of some inland river transport ships to evade supervision by turning off the ship's automatic identification system (Automatic Identification System, AIS). This has placed a great burden on the maintenance of inland river shipping order.

[0003] At the same time, there is also a waste of information resources in the current shipping system. Specifically, although a large number of high-definition cameras have been installed at docks, ports or both sides of the canal, these devices currently only serve to provide law enforcement evidence for regulatory authorities in most cases. These images or video data contain a large amount of information useful for ship identification, such as the ship's name. This patent proposes an efficient ship name recognition algorithm.

[0004] The current research on ship name recognition technology is limited, and the challenges in image quantity, image quality and language semantics have not been fully addressed. Specifically, these challenges are reflected in the following aspects:

[0005] 1. Complexity of the training process:

[0006] Current ship license plate recognition systems rely on complex training procedures and additional corpora to improve recognition performance during construction. This reliance not only increases the difficulty of system implementation, but also limits their applicability and flexibility in a variety of practical application scenarios.

[0007] 2. Dependence on external language models:

[0008] Existing methods usually integrate pre-trained language models to correct the errors made by the visual model during the recognition process. However, this approach that relies on an external language model may cause the performance of the recognizer to be overly dependent on the accuracy of the language model, which is inconsistent with the weak semantic relevance characteristics of ship license plate text images, and may affect the overall performance and generalization ability of the recognizer.

[0009] 3. Efficiency and adaptability to actual application scenarios:

[0010] Since the semantic relevance of the text image of the ship license plate in the character sequence is weak, the existing recognition method based on the language model has limitations in processing efficiency and adaptability in practical applications. In particular, when faced with irregular layouts and appearance changes, the existing technology may not be able to complete the recognition task efficiently and accurately.

[0011] In order to meet the above challenges, overcome the difficulties of ship name recognition, and provide solid technical support for the construction of an intelligent integrated management system for inland waterway shipping, this paper proposes a new type of ship name identifier. By adaptively merging double-branch character representations, the tedious training process is avoided and the dependence on external language models is eliminated. Summary of the invention

[0012] The purpose of the present invention is to provide a method for recognizing the name of an inland ship by combining autoregressive and non-autoregressive decoding, which improves the cumbersome training process and dependence on external language models of previous visual-language multimodal ship name recognition models. These previous methods, due to the use of external corpus to train language models, have generated a vocabulary dependency problem, and the correct result predicted by the visual model has been overcorrected. The technology proposed in this patent adaptively fuses the representations from the non-autoregressive visual model branch and the autoregressive language model branch at the character-level representation level, eliminating the cumbersome training process and the dependence on external language models. In addition, the present invention introduces a confidence-based dynamic sample loss reweighting strategy, including two confidence-based difficult sample mining losses at the character level and instance level, which enables the model training process to benefit more from difficult or challenging samples. At the same time, considering the variability of the scale of the ship name text image and the problem of fuzzy samples, the present invention constructs a visual feature extractor pre-training process based on mask image modeling, which is used to initialize the feature extraction module of the post-ship name recognizer.

[0013] To achieve the above object, the technical solution of the present invention mainly includes the following steps:

[0014] Step 1: Construct a ship name recognition dataset: Use a camera to collect images covering the ship name image, crop the ship name text line, use labelme annotation software to annotate the data, and use the text content as annotation information to create a recognition training set.

[0015] Step 2: Pre-training of the visual encoding model: Model the network based on the mask image and pre-train the feature extractor in a self-supervised learning manner.

[0016] Step 3: Build a ship name recognition model: Initialize the feature extractor pre-trained in step 2, and the extracted ship name image features are sent to the autoregressive decoder branch and the non-autoregressive decoder branch respectively to obtain two types of character representations.

[0017] Step 4: Adaptively fuse the representations of the autoregressive decoder branch and the non-autoregressive decoder branch through a gating mechanism.

[0018] Step 5: Model training: Take the character representations extracted by the autoregressive decoder branch and the non-autoregressive decoder branch and the fused character representation as input, respectively, and construct three linear classifiers to obtain three sets of predictions. The model is trained under the supervision of cross entropy loss.

[0019] Step 6: Implement difficult sample mining by fusing the losses of the autoregressive decoder branch and the non-autoregressive decoder branch, respectively constructing character-level and instance-level reweighted weights, and dynamically adjusting the contribution of batch samples to the total loss of the current batch during the back-propagation process.

[0020] The specific steps of step 1 are as follows:

[0021] Step 1-1: Take an image of a ship and its name by the Qiantang River with a camera. Adjust the focal length and shooting angle to make the image cover as many different properties as possible, such as scale, background, lighting conditions, shooting location, etc.

[0022] Step 1-2: Perform a rough correction on the text line area based on the perspective transformation of the coordinates of the four corner points of the ship name text line to obtain a small-scale ship name text line image.

[0023] Steps 1-3: On the basis of maintaining the aspect ratio of the image, a series of data enhancement techniques, including but not limited to random rotation, random contrast adjustment, random scale adjustment, random resolution adjustment, and random blurring, are used to process the small-scale image dataset. Subsequently, the processed images are precisely placed on a gray template of size 100×32. Data enhancement is used to improve the generalization of the optimized model during subsequent model training.

[0024] Step 1-4: Ship name text recognition model annotation: Save the cropped and expanded ship name text line image in the "file name annotation" format as .txt format as annotation information. Thus, the ship name text recognition model training data set is created.

[0025] The specific steps of step 2 are as follows:

[0026] Step 2-1: The selected network is VIT, and the network is modified, that is, the class token used for classification is deleted, and only the embedding vector corresponding to the image block is retained as the extracted image feature.

[0027] Step 2-3: By randomly selecting and masking a certain proportion of the area in the ship name image, the network is prompted to learn to predict the features of the masked area from the remaining area, and the mean squared error (MSE) or contrast loss is used to measure the difference between the predicted features and the true features to ensure that the network can effectively restore the masked area.

[0028] Step 2-4: Configure the training process. Select Adam as the optimizer, set the initial learning rate, and dynamically adjust the learning rate according to the training progress. Introduce L2 regularization and Dropout technology to reduce the risk of model overfitting and enhance the model's generalization ability to new data.

[0029] Steps 2-5: Regularly evaluate the performance of the model during training. Evaluate the performance of the model by calculating indicators such as the loss function value on the validation set, and fine-tune it as needed to complete and achieve a satisfactory performance level on the self-supervised learning task.

[0030] The specific steps of step 3 are as follows:

[0031] Step 3-1: Input the ship name image from the input end The image features of the ship name are obtained by outputting the feature extractor trained in step 2 Represents a real number set, C represents the number of output image channels, and H×W represents the height and width of the output image.

[0032] Step 3-2, build an autoregressive decoder (AR). The autoregressive decoder acts as a language model to capture the semantic relationship between the ship name character sequence. Therefore, the Transformer decoder is used as the autoregressive decoder, which includes three key modules: masked multi-head self-attention module, multi-head cross attention module and feed-forward network module (FFN). The autoregressive decoder branch uses right-shifted text annotation (is a known parameter)(y T The character representing the time step T) and the ship name image feature X obtained in step 3-1 are used as input, and are processed layer by layer through the masked multi-head self-attention module, the multi-head cross-attention module and the feedforward network module in turn, and finally the context-aware autoregressive character representation feature is extracted The predicted probability of a character at time step t is expressed as That is, the probability of predicting time step t The output from input 1 to t-1 is required And the ship name image feature X, where the softmax function is used to convert a set of real numbers into a probability distribution so that the output value is in the range of [0,1] and the sum of all output values ​​is 1.θ represents a linear classifier, l t is the character representation feature at the t-th time step.

[0033] Step 3-3: Construct a non-autoregressive decoder (NAR). The NAR decoder is used, which first performs a multi-attention dynamic fusion mechanism to refine the decoder input features. For the ship name image feature X obtained in step 3-1, the enhanced feature is obtained through three independent components, including channel attention, spatial attention and self-attention. Then, the enhanced features are integrated through a gated full fusion mechanism Dynamic Fusion Improved Features Finally, by querying the attention of the character sequence, the feature is improved Convert to non-autoregressive branch character representation feature F NAR =[v1,...,v T ]. Generate the complete target transcript, where the argmax function is used to find the value of the independent variable where the function reaches the maximum value on the given domain. Characters predicted for the non-autoregressive decoder.

[0034] The specific steps of step 4 are as follows:

[0035] Step 4-1: Adopt the gating mechanism to perform adaptive character representation fusion on the outputs of step 3-2 and step 3-3 to obtain the fusion feature F. The formula is as follows:

[0036] G=σ([F AR ,F NAR ]W)

[0037]

[0038] in, represents the learnable weight, σ represents the nonlinear activation function sigmoid, represents element-wise multiplication, Yes F AR and F NAR The adaptive weight matrix.

[0039] The final output fusion feature F is determined by the following formula:

[0040]

[0041] in represents the character predicted at the tth time step, represents the predicted character within the input 1-t time step, g t represents the adaptive weight at time step t, lt represents the autoregressive character representation feature at the tth time step, v t represents the non-autoregressive character representation features at the t-th time step.

[0042] The specific steps of step 5 are as follows:

[0043] Step 5-1: Represent F the extracted characters using the autoregressive decoder branch and the non-autoregressive decoder branch of step 3-3 and step 3-4 AR and F NAR , and the fused feature F after step 4-1 is used as input to construct three linear classifiers c θ , and get three sets of predictions. θ The main function is to divide the input space into different regions, each region maps a category, and learn the weights of features through training data to determine which features are more important for classification decisions. The specific form is f(x) = w x + b, x is the input feature vector; w is the weight vector, which represents the direction in the feature space and determines the decision boundary of the classification; b is the bias term, which controls the position of the decision boundary.

[0044] The specific steps of step 6 are as follows:

[0045] Step 6-1: During training, predictions for ship license plate text images with incomplete visual information (such as occlusion or blur) show lower confidence scores, so adding these challenging ship license plate text images can enhance the robustness of model recognition. The loss contribution of difficult and misidentified samples is inversely proportional to the prediction probability, that is, In order to refine the learnable weight W to better handle the data information generated when learning difficult data, two weights are designed: instance level weight w i and character-level weight w i , in order to dynamically adjust the contribution of batch samples to the total loss of the current batch during the back propagation process.

[0046] Step 6-2: Design instance-level weight w i . Instance level weight It is generated by taking the product of the confidence scores of T time steps and re-weighting the recognition loss at the granularity of the text image. The specific implementation formula is The sigmoid function compresses any real value into the range of 0 to 1, that is, performs the operation of the activation function.

[0047] Step 6-3: Design character level weight w i . Character Level Weight The character level It means that different weights are assigned to the recognition loss of the tth time step of the ith character. The specific implementation formula is: In order to balance learning from all data and learning from difficult data, a warm-up mechanism is adopted and the re-weighted working period is defined as t*E, where E represents the predefined period. i Or character level weight w i is based on the fusion loss L Fused Generated.

[0048] Step 6-4: Combine steps 6-1 to 6-3 to design the following loss function:

[0049]

[0050] The model is trained under the supervision of this cross entropy loss, where B represents the number of training batches and the CrossEntropy function represents the cross entropy loss function, which measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. The autoregressive decoder branch loss L is obtained by solving the cross entropy loss function AR , non-autoregressive decoder branch loss L NAR and the fusion loss L Fused ; Finally, the overall optimization goal is determined to be L = αL AR +βL NAR +γL Fused , where α, β, and γ are used as balancing factors. This loss reweighting method is to improve the robustness of the model for difficult-to-process samples, thereby improving the recognition performance of ship name text.

[0051] The beneficial effects of the present invention are as follows:

[0052] In view of the inefficiency of existing ship name recognition methods in practical applications, the present invention constructs a simple and efficient integrated architecture for ship name recognition by combining AR and NAR decoding mechanisms. The fused character representation dynamically integrates visual and language clues to adaptively adjust their contributions in identifying ship names with weak semantic relevance, and uses instance-level and character-level confidence-based reweighting mechanisms to explicitly mine difficult data, prompting the model to learn more from difficult ship name text images. This greatly solves the problem of recognizing ship name images that are not clearly photographed, blurred, or unevenly illuminated in practical applications, and constructs a more accurate and practical ship name recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Specific flow chart of the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Step 1: Construct a ship name recognition dataset: Use a camera to collect images covering the ship name image, crop the ship name text line, use labelme annotation software to annotate the data, and use the text content as annotation information to create a recognition training set.

[0056] Step 2: Pre-training of the visual encoding model: Model the network based on the mask image and pre-train the feature extractor in a self-supervised learning manner.

[0057] Step 3: Build a ship name recognition model: Initialize the feature extractor pre-trained in step 2, and the extracted ship name image features are sent to the autoregressive decoder branch and the non-autoregressive decoder branch respectively to obtain two types of character representations.

[0058] Step 4: Adaptively fuse the representations of the autoregressive decoder branch and the non-autoregressive decoder branch through a gating mechanism.

[0059] Step 5: Model training: Take the character representations extracted by the autoregressive decoder branch and the non-autoregressive decoder branch and the fused character representation as input, respectively, and construct three linear classifiers to obtain three sets of predictions. The model is trained under the supervision of cross entropy loss.

[0060] Step 6: Implement difficult sample mining by fusing the losses of the autoregressive decoder branch and the non-autoregressive decoder branch, respectively constructing character-level and instance-level reweighted weights, and dynamically adjusting the contribution of batch samples to the total loss of the current batch during the back-propagation process.

[0061] The specific steps of step 1 are as follows:

[0062] Step 1-1: Take an image of a ship and its name by the Qiantang River with a camera. Adjust the focal length and shooting angle to make the image cover as many different properties as possible, such as scale, background, lighting conditions, shooting location, etc.

[0063] Step 1-2: Perform a rough correction on the text line area based on the perspective transformation of the coordinates of the four corner points of the ship name text line to obtain a small-scale ship name text line image.

[0064] Steps 1-3: On the basis of maintaining the aspect ratio of the image, a series of data enhancement techniques, including but not limited to random rotation, random contrast adjustment, random scale adjustment, random resolution adjustment, and random blurring, are used to process the small-scale image dataset. Subsequently, the processed images are precisely placed on a gray template of size 100×32. Data enhancement is used to improve the generalization of the optimized model during subsequent model training.

[0065] Step 1-4: Ship name text recognition model annotation: Save the cropped and expanded ship name text line image in the "file name annotation" format as .txt format as annotation information. Thus, the ship name text recognition model training data set is created.

[0066] The specific steps of step 2 are as follows:

[0067] Step 2-1: The selected network is VIT, and the network is modified, that is, the class token used for classification is deleted, and only the embedding vector corresponding to the image block is retained as the extracted image feature, such as Figure 1 As shown in the "VIT encoder specific structure" section.

[0068] Step 2-3, such as Figure 1 As shown in the "Pre-training of Ship Name Feature Extractor Based on Mask Image Modeling" section, by randomly selecting and masking a certain proportion of areas in the ship name image, the network is prompted to learn to predict the features of the masked area from the remaining area, and the mean squared error (MSE) or contrast loss is used to measure the difference between the predicted features and the true features, ensuring that the network can effectively restore the masked area.

[0069] Step 2-4: Configure the training process. Select Adam as the optimizer, set the initial learning rate, and dynamically adjust the learning rate according to the training progress. Introduce L2 regularization and Dropout technology to reduce the risk of model overfitting and enhance the model's generalization ability to new data.

[0070] Steps 2-5: Regularly evaluate the performance of the model during training. Evaluate the performance of the model by calculating indicators such as the loss function value on the validation set, and fine-tune it as needed to complete and achieve a satisfactory performance level on the self-supervised learning task.

[0071] The specific steps of step 3 are as follows:

[0072] Step 3-1: Input the ship name image from the input end The image features of the ship name are obtained by outputting the feature extractor trained in step 2 represents a real number set, C represents the number of output image channels, H×W represents the height and width of the output image, and constructs Figure 1 The model structure shown in “Constructing a ship name identifier by combining autoregressive and non-autoregressive decoding”.

[0073] Step 3-2, build an autoregressive decoder (AR). The autoregressive decoder acts as a language model to capture the semantic relationship between the ship name character sequence. Therefore, the Transformer decoder is used as the autoregressive decoder, which includes three key modules: masked multi-head self-attention module, multi-head cross attention module and feed-forward network module (Feed-Forward Network, FFN). The autoregressive decoder branch uses right-shifted text annotation (is a known parameter)(y T The character representing the time step T) and the ship name image feature X obtained in step 3-1 are used as input, and are processed layer by layer through the masked multi-head self-attention module, the multi-head cross-attention module and the feedforward network module in turn, and finally the context-aware autoregressive character representation feature is extracted The predicted probability of a character at time step t is expressed as That is, the probability of predicting time step t The output from input 1 to t-1 is required And the ship name image feature X, where the softmax function is used to convert a set of real numbers into a probability distribution so that the output value is in the range of [0,1] and the sum of all output values ​​is 1. θ represents a linear classifier, l t is the character representation feature at the t-th time step.

[0074] Step 3-3: Construct a non-autoregressive decoder (NAR). The NAR decoder is used, which first performs a multi-attention dynamic fusion mechanism to refine the decoder input features. For the ship name image feature X obtained in step 3-1, the enhanced feature is obtained through three independent components, including channel attention, spatial attention and self-attention. Then, the enhanced features are integrated through a gated full fusion mechanism Dynamic Fusion Improved Features Finally, by querying the attention of the character sequence, the feature is improved Convert to non-autoregressive branch character representation feature F NAR =[v1,...,v T ]. Generate the complete target transcript, where the argmax function is used to find the value of the independent variable where the function reaches the maximum value on the given domain. Characters predicted for the non-autoregressive decoder.

[0075] The specific steps of step 4 are as follows:

[0076] Step 4-1: Adopt the gating mechanism to perform adaptive character representation fusion on the outputs of step 3-2 and step 3-3 to obtain the fusion feature F. The formula is as follows:

[0077] G=σ([F AR ,F NAR ]W)

[0078]

[0079] in, represents the learnable weight, σ represents the nonlinear activation function sigmoid, represents element-wise multiplication, Yes F AR and F NAR The adaptive weight matrix.

[0080] The final output fusion feature F is determined by the following formula:

[0081]

[0082] in represents the predicted character at the t-th time step, represents the predicted character within the input 1-t time step, g t represents the adaptive weight at time step t, l t represents the autoregressive character representation feature at the tth time step, v t Represents the non-autoregressive character representation feature at time step t

[0083] The specific steps of step 5 are as follows:

[0084] Step 5-1: Represent F the extracted characters using the autoregressive decoder branch and the non-autoregressive decoder branch of step 3-3 and step 3-4 AR and F NAR , and the fused feature F after step 4-1 is used as input to construct three linear classifiers c θ , and get three sets of predictions. θ The main function is to divide the input space into different regions, each region maps a category, and learn the weights of features through training data to determine which features are more important for classification decisions. The specific form is f(x) = w x + b, x is the input feature vector; w is the weight vector, which represents the direction in the feature space and determines the decision boundary of the classification; b is the bias term, which controls the position of the decision boundary.

[0085] The specific steps of step 6 are as follows:

[0086] Step 6-1: During training, predictions for ship license plate text images with incomplete visual information (such as occlusion or blur) show lower confidence scores, so adding these challenging ship license plate text images can enhance the robustness of model recognition. The loss contribution of difficult and misidentified samples is inversely proportional to the prediction probability, that is, In order to refine the learnable weight W to better handle the data information generated when learning difficult data, two weights are designed: instance level weight w i and character-level weight w i , in order to dynamically adjust the contribution of batch samples to the total loss of the current batch during the back propagation process.

[0087] Step 6-2: Design instance-level weight w i . Instance level weight It is generated by taking the product of the confidence scores of T time steps and re-weighting the recognition loss at the granularity of the text image. The specific implementation formula is The sigmoid function compresses any real value into the range of 0 to 1, that is, performs the operation of the activation function.

[0088] Step 6-3: Design character level weight w i . Character Level Weight The character level It means that different weights are assigned to the recognition loss of the tth time step of the ith character. The specific implementation formula is: In order to balance learning from all data and learning from difficult data, a warm-up mechanism is adopted and the re-weighted working period is defined as t*E, where E represents the predefined period. i Or character level weight w i is based on the fusion loss L Fused Generated.

[0089] Step 6-4: Combine steps 6-1 to 6-3 to design the following loss function:

[0090]

[0091] The model is trained under the supervision of this cross entropy loss, where B represents the number of training batches and the CrossEntropy function represents the cross entropy loss function, which measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. The autoregressive decoder branch loss L is obtained by solving the cross entropy loss function AR , non-autoregressive decoder branch loss L NAR and the fusion loss L Fused ; Finally, the overall optimization goal is determined to be L = αL AR +βLNAR +γL Fused , where α, β, and γ are used as balancing factors. This loss reweighting method is to improve the robustness of the model for difficult-to-process samples, thereby improving the recognition performance of ship name text.

[0092] Table 1: Ablation experiment of ship plate text recognition algorithm based on multimodal learning of representation fusion

[0093]

[0094] After experiments, the present invention achieved a recognition accuracy of 92.74% on the SLPR-R dataset, as shown in Table 1. A recognition accuracy of 89.44% was achieved on the SLPR-P dataset, and the specific comparison results are shown in Table 1 below. Therefore, the ship name text recognition model with an integrated architecture of autoregressive and non-autoregressive decoding branches proposed in the present invention utilizes visual modality and language modality, interacts the two through a multi-attention mechanism, and uses the complementarity between autoregressive decoding and non-autoregressive decoding to further improve the recognition accuracy of ship name text images with distorted appearance.

Claims

1. A method for identifying inland river vessel names by combining autoregressive and non-autoregressive decoding, characterized in that: At the character-level representation level, the autoregressive decoder branch and the non-autoregressive decoder branch are adaptively fused, and a confidence-based dynamic sample loss reweighting strategy is introduced, including two confidence-based difficult sample mining losses at the character level and instance level. The specific steps are as follows: Step 1: Constructing a ship name recognition dataset: Use a camera to collect images covering the ship name image, crop the ship name text line, use labelme annotation software to annotate the data, and use the text content as annotation information to create a recognition training set; Step 2: Pre-training the visual encoding model: Model the network based on the mask image and pre-train the feature extractor in a self-supervised learning manner; Step 3: Build a ship name recognition model: Initialize the feature extractor pre-trained in step 2, and the extracted ship name image features are sent to the autoregressive decoder branch and the non-autoregressive decoder branch respectively to obtain two types of character representations; Step 4: Adaptively fuse the representations of the autoregressive decoder branch and the non-autoregressive decoder branch through a gating mechanism; Step 5: Model training: Take the character representations extracted by the autoregressive decoder branch and the non-autoregressive decoder branch and the fused character representations as input, respectively, and construct three linear classifiers to obtain three sets of predictions; The model is trained under the supervision of cross entropy loss; Step 6: Implement difficult sample mining by fusing the losses of the autoregressive decoder branch and the non-autoregressive decoder branch, respectively constructing character-level and instance-level reweighted weights, and dynamically adjusting the contribution of batch samples to the total loss of the current batch during the back-propagation process.

2. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1-1, use a camera to capture images with ships and their names; adjust the focal length and shooting angle to make the captured images cover more diverse properties; Step 1-2, roughly correct the text line area based on the perspective transformation of the coordinates of the four corner points of the ship name text line to obtain a small-scale ship name text line image; Step 1-3: On the basis of maintaining the aspect ratio of the image, the image dataset is processed using data augmentation technology, including but not limited to random rotation, random contrast adjustment, random scale adjustment, random resolution adjustment, and random blur processing; the processed image is accurately placed on a gray template with a size of 100×32; Step 1-4, the ship name text recognition model saves the cropped and expanded ship name text line image in .txt format in the "file name annotation" format as annotation information; Thus, the ship name text recognition model training data set is created.

3. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2-1, select the VIT network and modify it, delete the category mark used for classification in the VIT network, and only retain the embedding vector corresponding to the image block as the extracted image feature; Step 2-3: By randomly selecting and masking a certain proportion of the ship name image, the network is prompted to learn to predict the features of the masked area from the remaining area, and the mean square error or contrast loss is used to measure the difference between the predicted features and the true features to ensure that the network can effectively restore the masked area; Step 2-4: Configure the training process; select Adam as the optimizer, set the initial learning rate, and dynamically adjust the learning rate according to the training progress; introduce L2 regularization and Dropout technology to reduce the risk of model overfitting and enhance the model's generalization ability to new data; Steps 2-5. Evaluate the performance of the model regularly during training; evaluate the performance of the model by calculating the loss function value on the validation set and fine-tune it as needed.

4. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3-1: Input the ship name image from the input end The image features of the ship name are obtained by outputting the feature extractor trained in step 2 represents a real number set, C represents the number of output image channels, and H×W represents the height and width of the output image; Step 3-2: Construct an autoregressive decoder. Use the Transformer decoder as the autoregressive decoder, which includes three key modules: a masked multi-head self-attention module, a multi-head cross-attention module, and a feedforward network module. The autoregressive decoder branch uses right-shifted text annotation. The ship name image feature X obtained in step 3-1 is used as input, and is processed layer by layer through the masked multi-head self-attention module, the multi-head cross-attention module and the feedforward network module, and finally the context-aware autoregressive character representation feature is extracted. y T represents the character at time step T; the predicted probability of the character in time step t is expressed as Predict the probability at time step t The output from input 1 to t-1 is required and the ship name image feature X, where the softmax function is used to convert a set of real numbers into a probability distribution so that the output value is in the range of [0,1] and the sum of all output values ​​is 1; c θ represents a linear classifier, l t is the character representation feature at the tth time step; Step 3-3: Construct a non-autoregressive decoder; use a NAR decoder and first perform a multi-attention dynamic fusion mechanism to refine the decoder input features; For the ship name image feature X obtained in step 3-1, the enhanced feature is obtained through three independent components, including channel attention, spatial attention and self-attention Then, the enhanced features are integrated through a gated full fusion mechanism Dynamic Fusion Improved Features Finally, by querying the attention of the character sequence, the feature is improved Convert to non-autoregressive branch character representation feature F NAR =[v1,...,v T ]; at the same time through Generate a complete target transcript, where the argmax function is used to find the value of the independent variable when the function reaches the maximum value on a given domain; Characters predicted for the non-autoregressive decoder.

5. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 4, characterized in that: The specific steps of step 4 are as follows: Step 4-1: Adopt the gating mechanism to perform adaptive character representation fusion on the outputs of step 3-2 and step 3-3 to obtain the fusion feature F. The formula is as follows: G=σ([F AR ,F NAR ]W) in, represents the learnable weight, σ represents the nonlinear activation function sigmoid, represents element-wise multiplication, Yes F AR and F NAR The adaptive weight matrix of The final output fusion feature F is determined by the following formula: in represents the character predicted at the tth time step, represents the predicted character within the input 1-t time step, g t represents the adaptive weight at time step t, l t represents the autoregressive character representation feature at the tth time step, v t represents the non-autoregressive character representation features at the t-th time step.

6. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 5, characterized in that: The specific steps of step 5 are as follows: Step 5-1: Represent F the extracted characters using the autoregressive decoder branch and the non-autoregressive decoder branch of step 3-3 and step 3-4 AR and F NAR , and the fused feature F after step 4-1 is used as input to construct three linear classifiers c θ , and get three sets of predictions; among them, c θ The main function is to divide the input space into different areas, each area maps a category, and learn the weights of the features through training data to determine which features are more important for classification decisions; the specific form is f(x) = w·x+b, x is the input feature vector, w is the weight vector, and b is the bias term.

7. The method for recognizing inland river vessel names by combining autoregressive and non-autoregressive decoding according to claim 6, characterized in that: The specific steps of step 6 are as follows: Step 6-1: In order to refine the learnable weight W and better handle the data information generated when learning difficult data, two weights are designed: instance-level weight w i and character-level weight w i , to dynamically adjust the contribution of batch samples to the total loss of the current batch during the back-propagation process; Step 6-2: Design instance-level weight w i ; Instance level weight It is generated by taking the product of the confidence scores of T time steps and re-weighting the recognition loss at the granularity of the text image. The specific implementation formula is The sigmoid function compresses any real value into the range of 0 to 1, that is, performs the operation of the activation function; Step 6-3: Design character level weight w i ; Character level weight The character level It means that different weights are assigned to the recognition loss of the tth time step of the ith character. The specific implementation formula is: In order to balance learning from all data and learning from difficult data, a warm-up mechanism is adopted, and the reweighted working period is defined as t*E, where E represents the predefined period; the instance-level weight w i Or character level weight w i is based on the fusion loss L Fused generated; Step 6-4: Combine steps 6-1 to 6-3 to design the following loss function: The model is trained under the supervision of this cross entropy loss, where B represents the number of training batches and the CrossEntropy function represents the cross entropy loss function, which measures the difference between the probability distribution predicted by the model and the probability distribution of the true label; the autoregressive decoder branch loss L is obtained by solving the cross entropy loss function AR , non-autoregressive decoder branch loss L NAR and the fusion loss L Fused ; Finally, the overall optimization goal is determined to be L = αL AR +βL NAR +γL Fused , where α, β, and γ serve as balancing factors.

Images