A method for recognizing characters in a customs document image

Abstract

The application discloses a method for recognizing characters in a customs document image, which comprises constructing a model Swin-TR for character recognition, wherein the Swin-TR is composed of three parts of a Swin-T backbone, a feature pyramid network, and a self-adaptive addressing and aggregation module; training the Swin-TR; in the training process, respectively calculating cross-entropy of prediction output of a character A 3 module, a subword A 3 module and a whole word A 3 module and real labels; and recognizing characters in a document image by using the qualified Swin-TR. The method for recognizing characters in a customs document image can accurately predict English, Chinese words and punctuation marks, can adapt to different text recognition scenes, and through comparison with other existing text recognition models by experiments, it is proved that the model Swin-TR proposed in the method has higher character recognition accuracy than existing models.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and in particular to a method for recognizing text in images of customs documents. Background Technology

[0002] With the rapid development of international trade, customs is currently facing challenges such as a surge in data processing volume and an increasing number of business rules. Customs-attached documents contain information not found in other business data. Extracting information from these documents is of significant strategic importance and practical necessity for improving customs' risk assessment capabilities and regulatory efficiency, enhancing tax revenue quality, and ensuring information security. However, customs-attached documents are unstructured data, often exhibiting drawbacks such as diverse table styles, low image quality, skewed document images, and irregular key-value pair distribution. Although OCR (optical character recognition) technology has made progress, character recognition errors can still occur, especially with poor image quality or complex fonts. This can lead to errors in the recognized text, requiring manual correction. OCR technology also struggles to recognize characters in uncommon or non-standard languages, making accurate recognition and translation of text in these languages ​​difficult. Furthermore, OCR technology may have difficulty recognizing text in complex layouts or designs, or text containing tables or graphics, potentially resulting in formatting errors in the recognized text. The data deficiencies in customs accompanying documents and the limitations of existing OCR technology make it difficult for the customs' existing system to efficiently extract information for risk identification from accompanying documents. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method for recognizing text in customs document images, so as to solve the technical problem of automatically extracting and recognizing character information from customs accompanying documents, and to solve the technical problem of character recognition accuracy.

[0004] The method for recognizing text in customs document images according to the present invention includes:

[0005] 1) Construct a model for character recognition called Swin-TR, which consists of three parts: a Swin-T backbone, a feature pyramid network, and an adaptive addressing and aggregation module;

[0006] The Swin-T backbone includes a Swin transformer and an SK unit. The SK unit is added after Patch Partition in the first stage of the Swin transformer and after Patch Merging in the second, third and fourth stages, respectively. The input of the first stage of the Swin transformer is a document image, and the output of the first stage of the Swin transformer is used as the input of the second stage, the output of the second stage is used as the input of the third stage, and the output of the third stage is used as the input of the fourth stage.

[0007] The feature pyramid network includes three sampling layers and one concat layer. The outputs of the third and fourth stages of the Swin transformer are merged and input into the first sampling layer of the feature pyramid network. The output of the second stage of the Swin transformer is merged with the output of the first sampling layer of the feature pyramid network and input into the second sampling layer of the feature pyramid network. The output of the first stage of the Swin transformer is merged with the output of the second sampling layer of the feature pyramid network and input into the third sampling layer of the feature pyramid network. The outputs of each sampling layer are connected through the concat layer and then output.

[0008] The adaptive addressing and aggregation module consists of character A. 3 Module, Sub-word A 3 Module, Whole Word A 3 The network consists of modules and fusion layers. The outputs of each layer of the feature pyramid network are concatenated using the concat function and then input to the character A. 3 Module, Sub-word A 3 Modules and whole words A 3 Module, character A 3 Module, Sub-word A 3 Modules and whole words A 3 The module is used to output the prediction results for the character. The fusion layer adjusts the first three A values ​​according to the confidence level. 3 The prediction results of the module are selected, and the prediction result with the highest confidence is selected as the output of Swin-TR;

[0009] 2) Train Swin-TR, calculating the character A during the training process. 3 Module, Sub-word A 3 Modules and whole words A 3 The cross-entropy between the module's predicted output and the true label is calculated, and the three cross-entropies are summed to form the final loss function of Swin-TR.

[0010] The formula for calculating cross-entropy is as follows:

[0011]

[0012] Where, p m This represents the predicted output of module m, where m∈{Character,BPE,WP}, and Character is character A. 3 Module, BPE is sub-word A 3 Module, WP is the whole word A 3 The module is defined as follows: t represents the true label of the target sample, with a length of N; dict represents a dictionary set, and w represents a character in the dictionary; t iw Indicates whether the i-th position in the actual label is the character w, when t iw A value of 0 indicates that it is not, when t iw A value of 1 indicates that it is; This represents the probability that the predicted result at the i-th position in the prediction output of module m is the character w;

[0013] The loss function is:

[0014] L=C(p Character ,t)+C(p BPE ,t)+C(p WP ,t)

[0015] 3) Use the qualified Swin-TR trained in step 2) to recognize the text in the document image.

[0016] The beneficial effects of this invention are:

[0017] 1. Experiments have demonstrated that the method for recognizing text in customs document images according to the present invention can accurately predict English and Chinese words and punctuation marks, and can adapt to different text recognition scenarios.

[0018] 2. The ablation experiment results demonstrate that the proposed model Swin-TR for the method of recognizing text in customs document images has the best accuracy compared to models Swin-T, Swin-T-SK, Swin-T-Parallel, Swin-T-FPN, and Swin-T-Parallel-SK.

[0019] 3. Through experiments and comparisons with other existing text recognition models, it is demonstrated that the Swin-TR model proposed in this invention has higher text recognition accuracy than existing models TBRA, DAN, SE-ASTER, SATRN, VisionLAN, ABINet, MGP-STR, SVTR, and SVTR-LCNet. Attached Figure Description

[0020] Figure 1 This is a structural diagram of the Swing transformer.

[0021] Figure 2 This is a structural diagram of the SK unit.

[0022] Figure 3 This is a structural diagram of the Swin-TR.

[0023] Figure 4 This is a structural diagram of the Swin-T backbone.

[0024] Figure 5 The architecture diagram for replacing the FPN in Swin-TR with a parallel structure (Swin-T-parallel or Swin-T-parallel-SK).

[0025] Figure 6 The character categories contained in the samples in the test set, validation set, and training set.

[0026] Figure 7 The loss curve is for training.

[0027] Figure 8 This represents the accuracy on the test set.

[0028] Figure 9 This is a schematic diagram of the attention output by Swin-TR when predicting characters. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] The method for recognizing text in customs document images in this embodiment includes:

[0031] 1) Construct a model for character recognition called Swin-TR, which consists of three parts: the Swin-T backbone, the feature pyramid network, and the adaptive addressing and aggregation module.

[0032] The Swin-T backbone includes a Swin transformer and an SK unit. The SK unit is added after Patch Partition in the first stage of the Swin transformer and after Patch Merging in the second, third and fourth stages, respectively. The input of the first stage of the Swin transformer is a document image, and the output of the first stage of the Swin transformer is used as the input of the second stage, the output of the second stage is used as the input of the third stage, and the output of the third stage is used as the input of the fourth stage.

[0033] The structure of the existing Swin transformer is as follows: Figure 1As shown, the Swin Transformer consists of four stages. The image is first sliced ​​into small patches, and then patch merging is performed at each stage to reduce the number of tokens and increase the feature dimension, thus forming four features at different scales. The forward pass of the model is as follows: the image is sliced ​​into 4x4 patches by the patchpartition module, each patch acting as a token, and projected onto any dimension through a linear embedding layer. Each stage consists of patch merging and several Swin Transformer Blocks. Patch merging reduces the number of tokens. The patch embedding module implements patch partitioning and linear embedding, used to slice patches and embed patch features into a specified dimension. This is achieved directly using a convolution with kernel_size and stride equal to patch_size. Patch merging performs a function similar to pooling or convolution with stride=2 in CNNs, used to reduce image resolution and increase the number of channels, thereby forming hierarchical features. This uses a pixel unshuffle-like operation, sampling at intervals along rows and columns, downsampling by a factor of 2, increasing the number of channels to 4 times, and then using a fully connected (FC) layer to double the number of channels again. Each block in the Swin Transformer Block consists of two consecutive sub-blocks, one based on W-MSA and the other on SW-MSA. The Swin Transformer is built by replacing the standard multi-head self-attention (MSA) module in the Transformer block with a shift-window-based module, while keeping other layers unchanged. Figure 1 As shown in (b), the Swin transformer block consists of a shift-window-based MSA module, followed by a two-layer MLP with intermediate GELU nonlinearity. LayerNorm (LN) layers are applied before each MSA module and each MLP, and residual connections are applied after each module. For computational efficiency, non-overlapping window-based attention is used, reducing the computational complexity from being proportional to the square of the image size to being proportional to the image size itself.

[0034] In this embodiment, the Swin-TR adds an SK unit to the existing Swin transformer structure. The structure of the existing SK unit is as follows: Figure 2 As shown, SK convolution is mainly implemented through three operators: Split, Fuse, and Select. The Split operation operates on the input feature map X∈R. H′*W′*C′Perform two convolution transformations respectively. During the process, the convolution kernel size is 3*3, and the feature map X is transformed by convolution to... During the process, the convolution kernel size is 5*5, and the feature map X is transformed by convolution to... The purpose of Fuse is to enable neurons to adaptively adjust the size of their receptive fields based on the stimulus content. Therefore, gates are needed to control the flow of information from multiple branches carrying information of different scales to neurons in the next layer. The purpose of Select is to adaptively select different spatial scales of information using cross-channel soft attention, guided by a compact feature descriptor z.

[0035] Subsequent ablation experiments confirmed that introducing the SK unit into the existing Swing transformer structure can improve accuracy, thus making this improvement meaningful.

[0036] Because customs documents vary in layout, font size, style, and image quality, a Feature Pyramid Network (FPN) was incorporated into the Swin-TR architecture to address these image variations encountered during customs document recognition. For example... Figure 3 As shown, the feature pyramid network includes three sampling layers and one concat layer. The outputs of the third and fourth stages of the Swin transformer are merged and input into the first sampling layer of the feature pyramid network. The output of the second stage of the Swin transformer is merged with the output of the first sampling layer of the feature pyramid network and input into the second sampling layer of the feature pyramid network. The output of the first stage of the Swin transformer is merged with the output of the second sampling layer of the feature pyramid network and input into the third sampling layer of the feature pyramid network. The outputs of each sampling layer are connected through the concat layer and then output. The feature pyramid network is used to capture multi-scale features.

[0037] Following the pyramid structure, the output resolution of the four stages in the Swin-T backbone gradually decreases from high (4x) to low (32x). For the coarser resolution feature maps, we increase the spatial resolution by 4x. Then, the upsampled feature maps are merged with the feature maps output from the previous stage by element-wise addition. This process is repeated until the finest resolution map is generated. The feature maps output from all stages are converted to the same size. Finally, these feature maps are concatenated together in the connection layer of the FPN.

[0038] The Adaptive Addressing and Aggregation (A3) module consists of the character A. 3 Module, Sub-word A 3Module, Whole Word A 3 The network consists of modules and fusion layers. The outputs of each layer of the feature pyramid network are concatenated using the concat function and then input to the character A. 3 Module, Sub-word A 3 Modules and whole words A 3 Module, character A 3 Module, Sub-word A 3 Modules and whole words A 3 The module is used to output the prediction results for the character. The fusion layer adjusts the first three A values ​​according to the confidence level. 3 The prediction results from the modules are selected, and the prediction result with the highest confidence is used as the output of Swin-TR. Introducing adaptive addressing and aggregation modules into Swin-TR to process multi-granular (character, sub-word, and even word) semantic information of customs documents enables the model to better understand and recognize complex text.

[0039] 2) Train Swin-TR, calculating the character A during the training process. 3 Module, Sub-word A 3 Modules and whole words A 3 The cross-entropy between the module's predicted output and the true label is calculated, and the sum of the three cross-entropies is used as the final loss function of Swin-TR.

[0040] The formula for calculating cross-entropy is as follows:

[0041]

[0042] Where, p m This represents the predicted output of module m, where m∈{Character,BPE,WP}, and Character is character A. 3 Module, BPE is sub-word A 3 Module, WP is the whole word A 3 The module is defined as follows: t represents the true label of the target sample, with a length of N; dict represents a dictionary set, and w represents a character in the dictionary; t iw Indicates whether the i-th position in the actual label is the character w, when t iw A value of 0 indicates that it is not, when t iw A value of 1 indicates that it is; This represents the probability that the predicted result at the i-th position in the prediction output of module m is the character w;

[0043] The loss function is:

[0044] L=C(p Character ,t)+C(p BPE ,t)+C(p WP ,t)

[0045] 3) Use the qualified Swin-TR trained in step 2) to recognize the text in the document image.

[0046] The advantages of the method for recognizing text in customs document images described in this embodiment are verified through experiments below:

[0047] Experimental setup

[0048] The experiment used a dataset called CustomsInvoice. CustomsInvoice consists of 3873 images (samples), all collected from invoices. Each image is labeled with a text tag. The dataset contains 423 characters (classes), mainly including Chinese characters, English letters, numbers, and punctuation marks. Typically, in text recognition, each distinct character is considered a separate class. Detailed information about this dataset is shown in Table 1.

[0049] Table 1. Statistics of the CustomsInvoice dataset used for text recognition.

[0050]

[0051] Figure 6 This displays the features covered by the training, validation, and test sets, respectively. The numbers represent the categories of characters contained in the samples. Figure 6 As we can see, the class sets of both the training and validation sets are subsets of the classes in the test set. Therefore, some characters in the test set (mainly Chinese characters) will be characters the model has never seen before. The class set of the training set is also a subset of the classes in the validation set, which makes the model's performance evaluation on the validation set similar to what it will encounter during inference on the test set.

[0052] In our experiments, sentence accuracy (SA), GPU inference speed (ms / image), parameter count, and FLOPS (floating-point operations per second) were used to evaluate the text recognition performance of different models. SA is the ratio of correctly recognized sentences to the total number of sentences. Here, a sentence refers to a single sample. To ensure a fair comparison, the average results of five runs were reported to reduce bias caused by random factors during testing. We saved the model that achieved the highest validation accuracy and reported its inference performance on the test set. The hyperparameter settings for the Swin-TR model are listed in Table 2.

[0053] Table 2. Hyperparameter settings for Swin-TR

[0054]

[0055] All experiments were implemented using Python and PyTorch and run on an NVIDIA GTX 3090Ti graphics card with 10752 cores and 24GB of global memory.

[0056] Swin-TR Evaluation

[0057] First, ablation experiments were conducted to validate the different components of our proposed model. The Swin-T transformer (Swin-T) backbone in the model was initialized with weights pre-trained on the ImageNet dataset, while the other components were initialized with random uniform distributions. For fair comparison, no pre-training was performed. Table 3 records the results of text recognition using the Swin-T backbone with different components. Due to A 3 The module is an important module for generating predictions, so it is included in all the comparative models in Table 3.

[0058] Table 3 Ablation test results of Swin-STR

[0059]

[0060] As shown in Table 3, the Swin-TR model achieved the best accuracy, while its inference speed was slightly slower than other models. This is because we introduced two components into our model: the Feature Pyramid Network (FPN) and the Selective Kernel (SK) unit. On the one hand, the addition of the FPN and SK units improved the model's accuracy. The FPN provides multi-scale feature representations, which we utilize to capture features at different levels of detail, helping the model better understand the input data and improve its accuracy. The SK unit allows each neuron in the CNN to adjust its receptive field size based on multiple scales of the input information; this adaptability further enhances the model's ability to capture relevant features. On the other hand, the introduction of the FPN and SK units increased the model's complexity, thus slightly decreasing the inference speed, but the improved accuracy justifies the use of these techniques.

[0061] We compared our proposed Swin-TR model with other state-of-the-art (SOTA) text recognition models, including TBRA, DAN, SE-ASTER, SATRN, VisionLAN, ABINet, MGP-STR, SVTR, and SVTR-LCNet. All models were initialized with pre-trained weights and trained for 200 epochs on the same dataset, CustomsInvoice, for a fair comparison. Table 4 summarizes the comparison results for CustomsInvoice. From Table 4, we can see that among all the compared methods, Swin-TR achieved the highest accuracy, while Swin-TR was in the middle in terms of inference speed.

[0062] Table 4 Performance comparison of different methods on the CustomsInvoice dataset

[0063]

[0064] The loss curve of Swin-TR during training and the accuracy curve on the test set are shown in Figure 7 and Figure 8 respectively. From Figure 7 , it can be seen that the loss curve gradually converges and stabilizes after the 25th epoch; from Figure 8 , it can be seen that the accuracy on the test set gradually increases, stabilizes around the 200th epoch, and remains stable at 30.56.

[0065] Table 5 lists several samples, labels, and prediction values of Swin-TR and several other models. It can be seen that Swin-TR can accurately recognize English words, Chinese words and punctuation marks, and can adapt to different text recognition scenarios.

[0066] Table 5 Comparison of predicted values of different models on the CustomsInvoice dataset

[0067]

[0068] Figure 9 shows how the attention layer works when Swin-TR performs text prediction. It can be seen from the figure that when predicting a single character such as the Chinese character "个", the model focuses on the main body of the character; when predicting multiple characters such as "编号:" (Number:) and "唛头" (Shipping Mark), the model not only focuses on the main body of each character, but also focuses on the boundary of the entire word; when there are many characters, the model notices the internal connection between multiple characters. For example, when predicting "电子签章" (electronic signature), the model's attention is respectively focused on the two parts "电子" (electronic) and "签章" (signature), and focuses on the boundary between them (the edges of "电子" and the edges of "签章"); similarly, when recognizing the English word "Quantity", the model's attention is also respectively focused on the word boundary and the phonetically continuous parts "Qu", "an", and "tity". This helps us understand how the Swin-TR model recognizes samples in the CustomsInvoice dataset.

[0069] Byte-Pair Encoding (BPE) tends to generate short segments, therefore the attention mask for BPE is divided into multiple regions. This is likely because performing sub-word segmentation and character addressing simultaneously is difficult. Furthermore, WordPiece typically generates a complete word, and the attention map should be the entire feature map. Since the attention map produced by the softmax function is usually sparse, the attention map for WordPiece is not as good as that for Character A. 3 The attention mapping of the module is so precise.

[0070] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for recognizing text in customs document images, characterized in that: include: 1) Construct a model for character recognition called Swin-TR, which consists of three parts: a Swin-T backbone, a feature pyramid network, and an adaptive addressing and aggregation module; The Swin-T backbone includes a Swin transformer and an SK unit. The SK unit is added after Patch Partition in the first stage of the Swin transformer and after Patch Merging in the second, third and fourth stages, respectively. The input of the first stage of the Swin transformer is a document image, and the output of the first stage of the Swin transformer is used as the input of the second stage, the output of the second stage is used as the input of the third stage, and the output of the third stage is used as the input of the fourth stage. The feature pyramid network includes three sampling layers and one concat layer. The outputs of the third and fourth stages of the Swin transformer are merged and input into the first sampling layer of the feature pyramid network. The output of the second stage of the Swin transformer is merged with the output of the first sampling layer of the feature pyramid network and input into the second sampling layer of the feature pyramid network. The output of the first stage of the Swin transformer is merged with the output of the second sampling layer of the feature pyramid network and input into the third sampling layer of the feature pyramid network. The outputs of each sampling layer are connected through the concat layer and then output. The adaptive addressing and aggregation module is composed of character A 3 Module, subword A 3 Module, whole word A 3 Module and fusion layer, the output of each layer of the feature pyramid network is connected by a concat function and then input into character A 3 Module, subword A 3 Module and whole word A 3 Module, character A 3 Module, subword A 3 Module and whole word A 3 Module is used for outputting the prediction result of the character, and the fusion layer selects the prediction results of the above three A 3 Module according to the confidence level, and selects the prediction result with the highest confidence as the output of Swin-TR; 2) Train Swin-TR, calculating the character A during the training process. 3 Module, Sub-word A 3 Modules and whole words A 3 The cross-entropy between the module's predicted output and the true label is calculated, and the three cross-entropies are summed to form the final loss function of Swin-TR. The formula for calculating cross-entropy is as follows: Where, p m This represents the predicted output of module m, where m ∈ {Character, BPE, WP}, and Character is character A. 3 Module, BPE is sub-word A 3 Module, WP is the whole word A 3 The module is defined as follows: t represents the true label of the target sample, with a length of N; dict represents a dictionary set, and w represents a character in the dictionary; t iw Indicates whether the i-th position in the actual label is the character w, when t iw A value of 0 indicates that it is not, when t iw A value of 1 indicates that it is; This represents the probability that the predicted result at the i-th position in the prediction output of module m is the character w; The loss function is: L=C(p Character ,t)+C(p BPE ,t)+C(p WP ,t) 3) Use the qualified Swin-TR trained in step 2) to recognize the text in the document image.

Images