Certificate image processing method based on FaPN, Swin Transformer and RRPN

Abstract

The invention discloses a certificate image processing method based on FaPN, Swin Transformer and RRPN, and relates to the technical field of certificate image detection. According to the method, the SwinTransform network is combined with the feature alignment pyramid FaPN network and the rotation detection suggestion network RRPN for application, texts of different sizes in the certificate image can be better processed, the advantages of the two networks can be fully utilized, the overall performance of certificate detection is improved, the detection precision is higher, the reasoning speed is higher, and the detection efficiency is improved. In order to solve the problem that various certificates all have rotary and inclined photos, an RRPN (Region Proportional by Rotation Anchor) is added on the basis of an original network, a region proposal used for generating rotation invariant is specially designed, rotation detection frames are generated through the RRPN, feature extraction is carried out on the rotation detection frames through Region of Interest Alignment (ROIAlignment to obtain feature vectors, and the feature vectors are extracted through the ROIAlignment to obtain the rotation detection frames of the original network, so that the rotation detection frames of the original network of the original network and the rotation detection frames of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network of the original network. And finally, a rotation detection result is obtained through the full connection layer, so that the operation can be simplified, the cost is reduced, and the safety and reliability of dangerous chemical transportation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of document image detection, and in particular to a document image processing method based on FaPN, Swin Transformer and RRPN. Background Art

[0002] In today's society, with the continuous advancement of industrialization and globalization, the transportation of dangerous goods has become an indispensable part of the modern logistics system. The transportation of hazardous chemicals (hazardous chemicals) is a highly specialized and extremely risky industry. The correct identification, classification, packaging, labeling and document management of hazardous chemicals are crucial to ensuring transportation safety. The transportation of hazardous chemicals by vehicle is directly related to the personal and property safety of the country and the people. It is particularly important to strictly manage and supervise the professionals who transport dangerous goods [1]. Among them, the operation process of hazardous chemicals includes checking whether the relevant procedures are complete and valid [2], and verifying the professional certificates of the transport personnel and their validity period is one of the key steps to ensure transportation safety.

[0003] With the rapid development of computer vision technology, it has become possible to use this technology to improve the automation and accuracy of hazardous chemicals transportation documents. Computer vision technology can quickly and accurately extract key information from hazardous chemicals transportation documents, such as name, name, validity period, transportation mark, etc., through image recognition and analysis technology, thereby greatly improving work efficiency and safety.

[0004] Certificate information extraction is done manually, and the information contains a large amount of numbers, so its efficiency and accuracy cannot meet business needs.

[0005] Deep learning-based text detection plays a vital role in document image detection and analysis, and has achieved impressive performance in many types of document tasks [3], [4], [5], such as document image detection in complex scenes [6], [7], and document authenticity and validity detection [8], [9],

[10] . Accurate and efficient document image detection can not only accurately locate key information areas, but also provide a reliable basis for document information recognition research

[11] ,

[12] , and has potential application value in identity authentication and security monitoring.

[0006] There are many different studies in the field of image detection for various types of certificates. In 2016, TianZ et al. improved and proposed CTPN

[13] based on fast-rcnn for horizontal text detection. In 2017, Minghui Liao proposed the Textbox method

[14] based on the SSD framework. EAST

[15] and SegLink

[16] were published at the 2017 CVPR (Conference on Computer Vision and Pattern Recognition) conference to achieve scene text detection. In 2018, PixelLink

[17] combined pixel-level segmentation and link prediction to detect text. In 2020, DBNet

[18] achieved efficient and accurate scene text detection, especially in processing text of arbitrary shapes. The above emphasizes the enhancement of the network architecture to better detect the target area. These studies have jointly promoted the accuracy and efficiency of text detection.

[0007] In the process of document detection, there are two reasons: First, the information in the document image may contain text and seals of different sizes. Second, the document may be rotated, tilted, or partially occluded. Commonly used networks include R-CNN

[19] , Fast R-CNN

[20] , and FasterR-CNN

[21] . These networks have achieved remarkable results in the field of target detection. In particular, FasterR-CNN has significantly improved the detection speed and accuracy by introducing the region proposal network. However, compared with more modern networks YOLO

[22] and SSD

[23] , its speed is still relatively slow. These networks may not perform well when dealing with small targets. Mask R-CNN

[24] performs target detection and can better handle rotated, tilted, and partially occluded targets, but the computational cost is high.

[0008] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0009] The purpose of the present invention is to provide a document image processing method based on FaPN, Swin Transformer and RRPN, aiming to improve the performance of the document detection method, propose an efficient and accurate detection method, promote the intelligent process in related fields, and solve the technical problems raised in the background technology.

[0010] To achieve the above object, the present invention provides the following technical solution: a document image processing method based on FaPN, Swin Transformer and RRPN, comprising at least the following steps:

[0011] S1: Build a backbone network based on the Swin Transformer network, using the window-based W-MSA to solve the computational efficiency problem by limiting the calculation range of self-attention within the local window, and solve the interaction problem by using the shifted window-based SW-MSA. Use two consecutive Swin Transformer modules to alternately use W-MSA and SW-MSA to achieve an effective combination of local and global features;

[0012] S2: Add the FaPN network on the basis of S1, enhance the feature expression ability through a more sophisticated feature aggregation strategy, and improve the accuracy and robustness of detection;

[0013] S3: Based on S2, a RRPN network is added to generate rotation-invariant candidate regions to improve the accuracy of detecting tilted or rotated text on documents.

[0014] Furthermore, the application of the backbone network of the Swin Transformer network in S1 at least comprises the following steps:

[0015] By adopting the shift window partitioning method, where Z l , Z l+1 and The calculation of can be expressed as, expressed by the following equation:

[0016]

[0017] Where: H and W represent the number of pixels of the image in the height and width directions respectively, MSA is multi-head self-attention, and LN is layer normalization;

[0018] and Z l represents the output features of the (S)WMSA module and the MLP module of block l;

[0019] and Z l+1 represents the output features of the (S)WMSA module and the MLP module of block l+1;

[0020] When processing an image of size 1024×1024×3, the image is first split into 4×4 patches (small blocks) through the PatchPartition module. Each patch is regarded as a new feature vector with a dimension of 4×4×3=48. After this processing, the image is converted into a tensor of size 256×256×48.

[0021] In the first stage, the patch feature vector first passes through a linear encoding layer and is converted into 96-dimensional tokens, which are then input into the SwinTransformerBlock for further processing; starting from the second stage, each stage consists of a Patch Merging module and a Swin TransformerBlock;

[0022] In the second stage, the Patch Merging module splits and merges the 256×256×96 tensor to generate a new tensor of 128×128×384.

[0023] To maintain consistency with other models, this tensor is then converted to a size of 128×128×192 using a 1×1 convolution kernel;

[0024] Therefore, the tensor size in the second stage changes from 256×256×96 to 128×128×192;

[0025] In the third and fourth stages, the size of the tensor is further reduced to 64×64×384 and 32×32×768, respectively, to meet the subsequent processing requirements;

[0026] The entire process ensures that image data can be effectively processed and analyzed at different stages.

[0027] Furthermore, the S2 at least includes the following steps:

[0028] By introducing a feature alignment module to replace the 1×1 convolutional layer in the traditional feature pyramid network, the module can learn pixel-level offsets to achieve alignment of high-resolution features with context information during upsampling;

[0029] A feature selection module is added after upsampling, which can focus on low-resolution features containing rich spatial details to enhance the feature expression ability and detection performance.

[0030] Furthermore, the introducing a feature alignment module at least comprises the following steps:

[0031] Before feature fusion, and Features are aligned. Feature alignment can be expressed mathematically as:

[0032]

[0033] in yes and The concatenation of provides the spatial difference between the upsampled and corresponding bottom-up features;

[0034] f o () and f a () respectively represent the offset Δ learned from the spatial difference i and a function to align features according to the offset;

[0035] Adding a feature selection module at least includes the following steps:

[0036] To extract the feature map of spatial details, first extract the feature map C through the global average pooling operation i Global information of

[0037] Then it is input into the 1×1 convolution layer and the sigmoid activation function, that is, f m ;

[0038] Get the important eigenvector u and convert C i Use u to scale;

[0039] Then the scaled C i Add to original C i middle;

[0040] Finally, through a 1×1 convolutional layer, that is, f s , keep important channels and discard unimportant channels.

[0041] Furthermore, the RRPN module includes a rotated bounding box, a rotated anchor point, candidate region learning, optimization of a region extraction network, and a RRoI pooling layer.

[0042] Further, the rotation detection frame is represented as (x, y, h, w, θ);

[0043] Where (x, y) represents the geometric center of the detection box, h is the short side of the detection box, w is the long side of the detection box, and θ is the angle between the positive x-axis and the long side of the rotated bounding box.

[0044] Further, the application of the rotating anchor point comprises at least the following steps:

[0045] The RRPN module uses 6 different angles to control the extraction of the target suspected area, namely:

[0046] The length-to-width ratio adopts 3 groups (1:2, 1:5, 1:8);

[0047] The scale is 3 groups (8, 16, 32);

[0048] This generates 6×3×3=54 5-dimensional (x, y, h, w, θ) anchor points.

[0049] For a feature map with a width and height of W×H, a number of anchor points of W×H×54 will be generated.

[0050] Furthermore, the application of the candidate region learning includes at least the following steps:

[0051] For the regression of the detection box, the background RoI is ignored and the smooth-L1 loss is used for the target RoI:

[0052]

[0053] Scale-invariant parameterized tuple v and v * The calculation of is as follows:

[0054]

[0055] Among them, x, x a and x * They are used to represent the predicted box, anchor box and real box respectively, and the same is true for y, h, w and θ;

[0056] Function f(a,b)=a-b+kπ, where k∈Z, such that

[0057] Furthermore, the region extraction network is optimized so that rotational region proposals can be generated in any orientation;

[0058] Therefore, the IoU calculation for the horizontal box may cause the IoU detection of the tilted detection box to be distorted. Therefore, the implementation of the tilted IoU calculation considering triangulation is used. Given a set of skew rectangles R1, R2, ..., R N , the goal is to calculate each pair of R i and R j IoU.

[0059] Furthermore, the RROI pooling layer uses a rotation RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction. The traditional RoI pooling layer may have difficulties in processing those rotated or tilted text regions because it is mainly designed to process horizontally or vertically aligned regions.

[0060] The process of using the RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction is:

[0061] Set the hyperparameters of the RRol layer to H r and W r ;

[0062] Input the representation of the proposed region (x, y, h, w, θ);

[0063] Pooling size (H r ,W r ), input feature map and spatial scale;

[0064] For a proposal region with height h and width w, the rotated proposal region can be divided into H r ×W r sub-regions, each with the same orientation as the proposed one;

[0065] Traverse all sub-regions of the candidate region and transform the sub-regions into standard rectangles through affine transformation;

[0066] Traverse each sub-region and obtain The maximum value, due to the use of floor rounding when interpolating the elements in the standard rectangle.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention can better process texts of different sizes in document images by using a Swin Transformer network combined with a feature alignment pyramid FaPN network and a rotation detection proposal network RRPN, and can make full use of the advantages of both to improve the overall performance of document detection, including higher detection accuracy and faster reasoning speed. In addition, for the problem of rotated and tilted photos in various types of documents, RRPN (Region Proposal by Rotated Anchor) is added to the original network. It is specially designed to generate rotation-invariant region proposals, and a rotation detection frame is generated by RRPN. Then, feature extraction is performed on these rotation detection frames through region of interest alignment (ROI Align) to obtain feature vectors. Finally, the rotation detection result is obtained through a fully connected layer, thereby simplifying the operation, reducing costs, and improving the safety and reliability of transporting hazardous chemicals.

[0069] 2. According to the diversity of transport documents, the network model proposed by the present invention uses Swin Transformer as the backbone network for target detection. It performs well as a backbone network in target detection tasks and can provide richer contextual information and stronger feature representation capabilities, which is especially important for detecting different targets in documents or targets in complex backgrounds.

[0070] 3. The Swin Transformer in the present invention can capture visual features from low-level to high-level through a hierarchical structure, involving window division and shift operations, effectively combining local and global information, so that the detection task improves the expression ability and efficiency of features. Since there are rotated or tilted documents in the target documents, the RRPN network is added to detect areas in any direction, which improves the overall accuracy. The accuracy of the improved Swin Transformer is verified through experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0072] Figure 1 It is a schematic diagram of the SwinTransformer module in the prior art;

[0073] Figure 2 It is a FSM structure diagram in the prior art;

[0074] Figure 3 It is a structural diagram of RRPN in the prior art;

[0075] Figure 4 This is a structural diagram of the model of the present invention;

[0076] Figure 5 This is a schematic diagram of the loss comparison curve of the model of the present invention;

[0077] Figure 6 This is a schematic diagram of the accuracy comparison curve of the model of the present invention;

[0078] Figure 7 It is a schematic diagram of the total loss curve of FPN and FaPN of the present invention;

[0079] Figure 8 Schematic diagram of loss_bbox curve of FPN and FaPN of the present invention;

[0080] Fig. 9 It is a schematic diagram of the accuracy comparison between FPN and FaPN of the present invention;

[0081] Fig.10 This is a schematic diagram for positioning comparison of the present invention. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0083] A document image processing method based on FaPN, Swin Transformer and RRPN, comprising at least the following steps:

[0084] S1: Build a backbone network based on the Swin Transformer network, using the window-based W-MSA. By limiting the calculation range of self-attention within the local window, the amount of calculation is significantly reduced, because the size of the window is usually much smaller than the entire image, thereby solving the problem of computational efficiency. By using the shifted window-based SW-MSA, the model can establish cross-window connections between different layers through the movement of the window, thereby capturing a wider range of global dependencies, thereby solving the interaction problem. Two consecutive Swin Transformer modules are used to alternately use W-MSA and SW-MSA to achieve an effective combination of local and global features.

[0085] S2: Add the FaPN network on the basis of S1, enhance the feature expression ability through a more sophisticated feature aggregation strategy, and improve the accuracy and robustness of detection;

[0086] S3: Based on S2, a RRPN network is added to generate rotation-invariant candidate regions to improve the accuracy of detecting tilted or rotated text on documents.

[0087] The application of the backbone network of the Swin Transformer network includes at least the following steps:

[0088] By adopting the shift window partitioning method, such as Figure 1 As shown. l , Z l+1 and The calculation of can be expressed as, expressed by the following equation:

[0089]

[0090] Where: H and W represent the number of pixels of the image in the height and width directions respectively, MSA is multi-head self-attention, and LN is layer normalization;

[0091] and Z l represents the output features of the (S)WMSA module and the MLP module of block l;

[0092] and Z l+1represents the output features of the (S)WMSA module and the MLP module of block l+1;

[0093] When processing an image of size 1024×1024×3, the image is first split into 4×4 patches (small blocks) through the PatchPartition module. Each patch is regarded as a new feature vector with a dimension of 4×4×3=48. After this processing, the image is converted into a tensor of size 256×256×48.

[0094] In the first stage, the patch feature vector first passes through a linear encoding layer and is converted into 96-dimensional tokens, which are then input into the SwinTransformerBlock for further processing; starting from the second stage, each stage consists of a Patch Merging module and a Swin TransformerBlock;

[0095] In the second stage, the Patch Merging module splits and merges the 256×256×96 tensor to generate a new tensor of 128×128×384.

[0096] For consistency with other models (such as ResNet), this tensor is then converted to a size of 128×128×192 through a 1×1 convolution kernel;

[0097] Therefore, the tensor size in the second stage changes from 256×256×96 to 128×128×192;

[0098] In the third and fourth stages, the size of the tensor is further reduced to 64×64×384 and 32×32×768, respectively, to meet the subsequent processing requirements;

[0099] The entire process ensures that image data can be effectively processed and analyzed at different stages.

[0100] FaPN (Feature Aggregation Pyramid Network)

[27] is called feature alignment pyramid, which has the characteristics of multi-scale feature fusion and feature alignment. By introducing a feature alignment module (FAM) to replace the 1×1 convolution layer in the traditional feature pyramid network (FPN), the module can learn pixel-level offsets, thereby achieving alignment of high-resolution features and context information during upsampling;

[0101] A feature selection module (FSM) is added after upsampling, which can focus on low-resolution features containing rich spatial details to enhance the feature expression ability and detection performance.

[0102] Introducing a feature alignment module includes at least the following steps:

[0103] Before feature fusion, and Features are aligned. Feature alignment can be expressed mathematically as:

[0104]

[0105] in yes and The concatenation of provides the spatial difference between the upsampled and corresponding bottom-up features;

[0106] f o () and f a () respectively represent the offset Δ learned from the spatial difference i and a function to align features according to the offset;

[0107] Adding a feature selection module includes at least the following steps:

[0108] To extract the feature map of spatial details, first extract the feature map C through the global average pooling operation i Global information of

[0109] Then it is input into the 1×1 convolution layer and the sigmoid activation function, that is, f m ;

[0110] Get the important eigenvector u and convert C i Use u to scale;

[0111] Then the scaled C i Add to original C i middle;

[0112] Finally, through a 1×1 convolutional layer, that is, f s , keep important channels and discard unimportant channels. Figure 2 .

[0113] RRPN

[28] (Arbitrary-Oriented Scene Text Detection via Rotation Proposals) achieves the purpose of detection by giving a rotation detection box. Its structure is as follows: Figure 3 As shown in Figure 2, the RRPN module includes a rotated bounding box, a rotated anchor point, a candidate region learning, an optimization of the region extraction network, and an RRoI pooling layer.

[0114] Furthermore, the rotation detection box is represented as (x, y, h, w, θ);

[0115] Where (x, y) represents the geometric center of the detection box, h is the short side of the detection box, w is the long side of the detection box, and θ is the angle between the positive x-axis and the long side of the rotated bounding box;

[0116] Using five variables like this for representation has three benefits: (1) the angular difference between two rotated bounding boxes can be directly calculated using θ; (2) compared to the traditional bounding box representation using eight points (i.e., the coordinates of four vertices), the five-tuple representation is more concise; and (3) this representation makes it more efficient to calculate the rotated ground truth.

[0117] The application of the rotation anchor point includes at least the following steps:

[0118] The RRPN module uses 6 different angles to control the extraction of the target suspected area, namely:

[0119] The length-to-width ratio adopts 3 groups (1:2, 1:5, 1:8);

[0120] The scale is 3 groups (8, 16, 32);

[0121] This generates 6×3×3=54 5-dimensional (x, y, h, w, θ) anchor points.

[0122] For a feature map with a width and height of W×H, a number of anchor points of W×H×54 will be generated.

[0123] The application of candidate region learning includes at least the following steps:

[0124] For the regression of the detection box, the background RoI is ignored and the smooth-L1 loss is used for the target RoI:

[0125]

[0126] Scale-invariant parameterized tuple v and v * The calculation of is as follows:

[0127]

[0128] Among them, x, x a and x * They are used to represent the predicted box, anchor box and real box respectively, and the same is true for y, h, w and θ;

[0129] Function f(a,b)=a-b+kπ, where k∈Z, such that

[0130] The region extraction network is optimized so that rotational region proposals can be generated in any orientation;

[0131] Therefore, the IoU calculation for the horizontal box may cause the IoU detection of the tilted detection box to be distorted. Therefore, the implementation of the tilted IoU calculation considering triangulation is used. Given a set of skew rectangles R1, R2, ..., R N , the goal is to calculate each pair of R i and R j IoU.

[0132] The RROI pooling layer uses the rotation RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction. The traditional RoI pooling layer may have difficulties in processing those rotated or tilted text regions because it is mainly designed to process horizontally or vertically aligned regions;

[0133] The process of using the RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction is:

[0134] Set the hyperparameters of the RRol layer to H r and W r ;

[0135] Input the representation of the proposed region (x,u,h,w,θ);

[0136] Pooling size (H r ,W r ), input feature map and spatial scale;

[0137] For a proposal region with height h and width w, the rotated proposal region can be divided into H r ×W r sub-regions, each with the same orientation as the proposed one;

[0138] Traverse all sub-regions of the candidate region and transform the sub-regions into standard rectangles through affine transformation;

[0139] Traverse each sub-region and obtain The maximum value, due to the use of floor rounding when interpolating the elements in the standard rectangle.

[0140] Based on the above embodiment, the present invention proposes the following experiment:

[0141] 1. Dataset

[0142] The experimental data set contains 1967 training set images of 1024×1024 and 490 test set images of the same size, including 6 types of certificates, namely dangerous goods driving qualification certificate, road transport certificate, special equipment use mark, vehicle registration certificate, driver's license, and vehicle gas cylinder registration certificate. Due to the design of privacy security, the number of photos in 3 categories is relatively small. There are many types of images in the data set, the target scales vary, and the key information is different, which poses a great challenge to the accuracy, multi-scale extraction ability, and robustness of the algorithm.

[0143] 2. Evaluation indicators

[0144] The evaluation index used in this experiment is the average precision mAP50 (meanAveragedPrecision, IoU=0.5) value of all categories. The relevant calculation formula is as follows:

[0145]

[0146] Among them, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, and FN is the number of unidentified targets. When the intersection over union (IoU) of the predicted box and the true box area is greater than the threshold of 0.5 and the categories are consistent, the predicted box is marked as a positive sample. AP is the area under the PR curve surrounded by precision and recall, and mAP is the average AP of all categories.

[0147]

[0148] 3. Experimental environment and hyperparameter settings

[0149] The model training environment is divided into: the hardware environment is a computer CPU with a main frequency of 3.00Ghz, 8 cores and 16 threads. The GPU is NVIDIA RTXA6000 48G video memory. The software environment is CUDA11.8, Python3.9.16PyTorch2.0.0.

[0150] The model training uses the default loss function; the optimizer uses AdamW; the learning rate is initially 10-3 and eventually decays to 10-5, the learningscheduler uses LamdaLR, the image size is resized to 1024×1024 during training, the batch size is 32, the number of iterations is 100, and the training is terminated early if the performance improvement within 100 times is less than.

[0151] 4. Results and Analysis

[0152] The present invention verifies the effectiveness of the improved Swin Transformer in document detection. It can be clearly seen from the provided Table 1 that our enhanced Swin Transformer model shows excellent performance in the field in several key indicators, including the average accuracy of all categories (mPA50), inference speed (fps), number of parameters (M), number of floating-point operations (GFLOPs), and average training time (s / iter).

[0153] Table 1 Comparison of training results of different backbone networks combined with rotation detection modules

[0154]

[0155] The latest research of this paper introduces an improved SwinTransformer model, and compares various popular target detection networks Rotated RetinaNet

[29] , R3Det

[30] , Faster RCNN

[31] , ReDet

[32] , S2ANet

[33] , and RoI Transformer

[34] on a unified dataset combined with a rotation detection module for comparative experiments. The training parameters remain consistent. The comparison of the training results of different backbone networks combined with the rotation detection module is shown in Table 1. These data highlight the advantages of our model. The model has achieved good results in document detection tasks, especially in terms of key performance indicators. The convergence results of different network training are shown in Figure 1. Figure 5 As shown, the accuracy is Figure 6 shown.

[0156] pass Figure 5 It can be seen that the improved SwinTransformer has good convergence during the training process, and all networks have finally completed convergence. In the detection results of the same data set using different networks, it can be seen that the improved SwinTransformer has the highest mAP, reaching 90.03%, and has the best performance. The reason for the analysis is that the Transformer structure of the improved SwinTransformer can effectively capture the long-distance dependencies in the image, and W-MSA and SW-MSA can fully extract local information and its contextual information, thereby improving the recognition ability of the model. In addition, after obtaining multi-scale information of the image through PatchMerging, FaPN can effectively fuse features at different levels and improve the expression ability of the model. It uses deformable convolution to provide the deviation of the convolution kernel, and uses the ideas of the convolution kernel attention mechanism and the channel attention mechanism to improve the detection effect of the target. In summary, the improved SwinTransformer model not only improves the overall detection accuracy, but also maintains a high level of accuracy balance, which is particularly important in the field of image detection.

[0157] Further ablation experiments and analysis are proposed, as follows:

[0158] FqV

[0159] (1) Comparison of total training loss

[0160] Evaluate the performance of the network during training, conduct comparative analysis based on loss curves, conduct ablation experiments on the same dataset, and conduct experiments based on the FPN module and the FaPN module respectively. Figure 7 The loss of the model over a series of epochs on the dataset is plotted. The blue line represents the total loss of FaPN and the yellow line represents the total loss of FPN.

[0161] The total loss curves of FaPN and FPN both start from a relatively high initial loss value, and the loss value gradually decreases as the training iterations proceed. The loss curve of FaPN is always lower than that of FPN throughout the training process, especially in the early stages of training, when the loss of FaPN decreases significantly faster than that of FPN. As the number of iterations increases, the loss curves of the two models tend to stabilize, but the loss of FaPN always remains at a low level.

[0162] The reason for the analysis is that feature alignment reduces the loss of information during feature fusion by ensuring accurate matching of key information between feature maps of different scales. This mechanism optimizes the transmission and utilization of features, improves the model's ability to identify and locate targets, and thus outperforms traditional FPN in terms of loss function, which fails to fully utilize information of all scales during feature fusion.

[0163] (2) loss_bbox comparison

[0164] Loss_bbox is a commonly used loss function in object detection tasks, which is used to measure the difference between the predicted bounding box and the true bounding box. Figure 8 The loss_bbox curves of FaPN and FPN are shown in Figure 2. They show the performance of FaPN and FPN in terms of bounding box loss. Similar to the total loss curve, the bounding box loss of FaPN is also significantly lower than that of FPN, and this advantage is maintained throughout the training process. This shows that FaPN not only has an advantage in initial convergence speed, but also has more accurate bounding box positioning accuracy in the later stage.

[0165] Bounding box loss is directly related to the accuracy of the model in locating the target. FaPN improves the quality of feature transfer through feature alignment, allowing the model to more accurately predict the location of the target. Feature alignment ensures that features of different scales can work better together, especially when locating small or partially occluded targets, providing more accurate information. In addition, feature alignment also improves the model's ability to handle edge information and frame the target boundary.

[0166] (3) Accuracy

[0167] In order to evaluate the performance of the network during training, the following comparative analysis is conducted based on accuracy. Ablation experiments are conducted on the same dataset, and experiments based on the FPN module and the FaPN module are conducted separately. The chart depicts the accuracy of the model in a series of periods of the dataset. The blue line represents the accuracy of FaPN, and the yellow line represents the accuracy of FPN.

[0168] As the training iterations proceed, the accuracy of FaPN and FPN gradually increases. The loss curve of FaPN is always higher than that of FPN during the entire training process, especially in the early stage of training, when the accuracy of FaPN is higher than that of FPN.

[0169] As the number of iterations increases, the accuracy curves of the two models tend to stabilize, but the accuracy of FaPN always remains at a high level.

[0170] RRPN

[0171] The technology of Rotated Region Proposal Network (RRPN) can more accurately identify and locate tilted or rotated text regions in documents. RRPN generates rotated region proposals, which enables the model to adapt to texts at different angles and provide text regions with the correct direction for the OCR stage, thus greatly improving the accuracy and efficiency of text recognition. Fig.10 As shown in the figure, traditional detection methods may not be able to accurately select or miss some rotated texts, while RRPN can accurately cover and recognize these texts.

[0172] When FaPN is combined with the Swin Transformer model for object detection, the mAP50 is 79.61% when applying standard horizontal detection (Horizontal), and the mAP50 is as high as 90.03% when applying the rotation region proposal network (RRPN). As shown in Table 2.

[0173] Table 2 Comparison of horizontal rotation

[0174]

[0175] The reason for the analysis is that a large number of images in the dataset contain rotated objects, and traditional horizontal detection methods may not be able to effectively locate and identify these objects. Because standard detection frameworks usually assume that objects are placed horizontally or vertically, this will lead to inaccurate detection of rotated or tilted objects. RRPN allows the model to more accurately align and identify rotated objects in the dataset by allowing the detection box to capture the target at any angle, thereby significantly improving the mAP indicator. In actual application scenarios, rotation detection can significantly improve the robustness of the model without the user having to strictly align the horizontality of the photo.

[0176] in conclusion

[0177] In view of the problems that the existing target recognition methods have poor detection effects on multiple types of certificates and key information, there are large redundant areas in the detection results, and the detection results do not include key information, the present invention has carried out research on target detection of multiple types of certificate key information, proposed a method based on multi-scale feature extraction and rotation target detection, combined with FaPN to improve SwinTransformer, enhance its detection capabilities of multiple types of certificates and key information, and achieve accurate positioning of tilted and rotated detection targets through the RRPN rotation detection module. By evaluating the recognition accuracy of different models under the same data set and conducting ablation experiments on specific network modules, the feasibility and effectiveness of the optimized model were verified, with mAP50 reaching 90.03%, inference speed (fps) of 21.1, parameter count (M) of 58.75, floating point operations (GFLOPs) of 66.32, and average training time (s / iter) of 1.3567. The references involved in the entire specification are as follows:

[0178] [1] Yin Zhenhai. Suggestions on strengthening the management of hazardous chemicals transportation vehicles[J]. Transport Manager World, 2020, (18): 116-117.

[0179] [2] Yu Tao. Research on safety precautions for road transportation of hazardous chemicals[D]. Southwest Jiaotong University, 2009.

[0180] [3] Gao Shang, Li Yanling, Ge Fengpei, et al. Identity card information recognition based on improved convolutional neural network [J]. Computer Engineering and Design, 2023, 44(11): 3447-3454. DOI: 10.16208 / j.issn1000-7024.2023.11.032.

[0181] [4] Zhang Qingqing. Research on compliance detection of college entrance examination ID photos based on deep learning[D]. Nanchang University, 2023. DOI: 10.27232 / d.cnki.gnchu.2023.004322.

[0182] [5] Yu Jiangjian, Wu Qianqian. Design and implementation of card recognition applet based on Baidu AI[J]. Science and Technology Innovation and Application, 2020, (06): 80-83.

[0183] [6] Lin Hanyang, Zhan Yongzhao, Chen Yuzhong. Rapid detection and recognition of motor vehicle driving license in complex scenes[J]. Small and Micro Computer Systems, 2019, 40(05): 1076-1082.

[0184] [7] He Shuchi. Research on document image text detection and recognition algorithm based on EAST and CRNN[D]. Suzhou University, 2022.DOI:10.27351 / d.cnki.gszhu.2022.001949.

[0185] [8] Zhao Weidong, Huang Jian, Zhang Rui, et al. Application of multi-directional stacked memory network in document image tampering detection [J / OL]. Small and Micro Computer Systems, 1-12 [2024-07-13]. http: / / kns.cnki.net / kcms / detail / 21.1106.tp.20240202.0929.006.html.

[0186] [9] Jiang Baoxiang. Research on document and certificate image tampering detection algorithm and system application based on deep learning[D]. Xiamen University, 2022.DOI:10.27424 / d.cnki.gxmdu.2022.003245.

[0187]

[10] Zhang Zhida, Huang Peng, Liu Yang, et al. Document quality detection system based on YOLO algorithm and deep learning[J]. Microcontrollers and Embedded Systems Applications, 2021, 21(07): 67-70.

[0188]

[11] Huang Wenlu. Design and implementation of certificate information recognition system based on multi-scale feature fusion[D]. Beijing University of Posts and Telecommunications, 2022.DOI:10.26969 / d.cnki.gbydu.2022.002911.

[0189]

[12] Lu Jianxiao. Design and implementation of travel document image text recognition algorithm based on deep learning[D]. Huazhong University of Science and Technology, 2020. DOI: 10.27157 / d.cnki.ghzku.2020.002396.

[0190]

[13] Tian Z,Huang W,He T,et al.Detecting text in natural image withconnectionist text proposal network[C] / / Computer Vision-ECCV 2016:14thEuropean Conference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part VIII 14.Springer International Publishing,2016:56-72.

[0191]

[14] Liao M,Shi B,Bai X,et al.Textboxes:A fast text detector with asingledeep neural network[C] / / Proceedings of the AAAI conference onartificial intelligence.2017,31(1).

[0192]

[15] Zhou X,Yao C,Wen H,et al.East:an efficient and accurate scenetext detector[C] / / Proceedings of the IEEE conference on Computer Vision andPattern Recognition.2017:5551-5560.

[0193]

[16] Shi B,Bai X,Belongie S.Detecting oriented text in natural imagesby linking segments[C] / / Proceedings of the IEEE conference on computer visionand pattern recognition.2017:2550-2558.

[0194]

[17] Deng D,Liu H,Li X,et al.Pixellink:Detecting scene text viainstance segmentation[C] / / Proceedings of the AAAI conference on artificialintelligence.2018,32(1).

[0195]

[18] Liao M,Wan Z,Yao C,et al.Real-time scene text detection withdifferentiable binarization[C] / / Proceedings of the AAAI conference onartificial intelligence.2020,34(07):11474-11481.

[0196]

[19] Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies foraccurate object detection and semantic segmentation[C] / / Proceedings oftheIEEE conference on computer vision andpattern recognition.2014:580-587.

[0197]

[20] Girshick R.Fast r-cnn[C] / / Proceedings ofthe IEEE internationalconference on computer vision.2015:1440-1448.

[0198]

[21] Ren S,He K,Girshick R,et al.Faster r-cnn:Towards real-time objectdetection with region proposal networks[J].Advances in neural informationprocessing systems,2015,28.

[0199]

[22] Jiang P,Ergu D,Liu F,et al.A Review of Yolo algorithmdevelopments[J].Procedia computer science,2022,199:1066-1073.

[0200]

[23] Liu W,Anguelov D,Erhan D,et al.Ssd:Single shot multibox detector[C] / / Computer Vision-ECCV 2016:14th European Conference,Amsterdam,TheNetherlands,October 11-14,2016,Proceedings,Part I 14.Springer InternationalPublishing,2016:21-37.

[0201]

[24] He K,Gkioxari G,Dollár P,et al.Mask r-cnn[C] / / Proceedings of theIEEE international conference on computer vision.2017:2961-2969.

[0202]

[25] Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.

[0203]

[26] Krizhevsky A,Sutskever I,Hinton G E.Imagenet classification withdeep convolutional neural networks[J].Advances in neural informationprocessing systems,2012,25.

[0204]

[27] Huang S,Lu Z,Cheng R,et al.Fapn:Feature-aligned pyramid networkfor dense image prediction[C] / / Proceedings of the IEEE / CVF internationalconference on computervision.2021:864-873.

[0205]

[28] J,Shao W,Ye H,et al.Arbitrary-oriented scene text detection viarotation proposals[J].IEEE transactions on multimedia,2018,20(11):3111-3122.

[0206]

[29] Lin T Y,Goyal P,Girshick R,et al.Focal Loss for Dense ObjectDetection[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,PP(99):2999-3007.DOI:10.1109 / TPAMI.2018.2858826.

[0207]

[30] Yang X,Liu Q,Yan J,et al.R3Det:Refined Single-Stage Detector withFeature Refinement for Rotating Object[J].2019.DOI:10.48550 / arXiv.1908.05612.

[0208]

[31] Ren S,He K,Girshick R,et al.Faster R-CNN:Towards Real-Time ObjectDetection with Region Proposal Networks[J].IEEE Transactions on PatternAnalysis&Machine Intelligence,2017,39(6):1137-1149.DOI:10.1109 / TPAMI.2016.2577031.

[0209]

[32] Jiaming Han; Jian Ding; Nan Xue; Gui-Song Xia.ReDet: A Rotation-equivariant Detector for Aerial Object Detection[A].2021IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR)[C], 2021

[0210]

[33] Dosovitskiy A,Beyer L,Kolesnikov A,et al.An Image is Worth 16x16Words:Transformers for Image Recognition at Scale[C] / / InternationalConference on Learning Representations.2021.

[0211]

[34] Ding J,

[0212] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A document image processing method based on FaPN, Swin Transformer and RRPN, characterized in that: At least the following steps are included: S1: Build a backbone network based on the Swin Transformer network, using the window-based W-MSA to solve the computational efficiency problem by limiting the calculation range of self-attention within the local window, and solve the interaction problem by using the shifted window-based SW-MSA. Use two consecutive Swin Transformer modules to alternately use W-MSA and SW-MSA to achieve an effective combination of local and global features; S2: Add the FaPN network on the basis of S1, enhance the feature expression ability through a more sophisticated feature aggregation strategy, and improve the accuracy and robustness of detection; S3: Based on S2, a RRPN network is added to generate rotation-invariant candidate regions to improve the accuracy of detecting tilted or rotated text on documents.

2. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 1, characterized in that: The application of the backbone network of the Swin Transformer network in S1 at least comprises the following steps: By adopting the shift window partitioning method, where Z l , Z l+1 and The calculation of can be expressed as, expressed by the following equation: Where: H and W represent the number of pixels of the image in the height and width directions respectively, MSA is multi-head self-attention, and LN is layer normalization; and Z l represents the output features of the (S)WMSA module and the MLP module of block l; and Z l+1 represents the output features of the (S)WMSA module and the MLP module of block l+1; When processing an image of size 1024×1024×3, the image is first divided into 4×4 patches (small blocks) through the Patch Partition module. Each patch is regarded as a new feature vector with a dimension of 4×4×3=48. After this processing, the image is converted into a tensor of size 256×256×48. In the first stage, the patch feature vector first passes through a linear encoding layer and is converted into 96-dimensional tokens, which are then input into the SwinTransformerBlock for further processing; starting from the second stage, each stage consists of a Patch Merging module and a Swin TransformerBlock; In the second stage, the Patch Merging module splits and merges the 256×256×96 tensor to generate a new tensor of 128×128×384. To maintain consistency with other models, this tensor is then converted to a size of 128×128×192 using a 1×1 convolution kernel; Therefore, the tensor size in the second stage changes from 256×256×96 to 128×128×192; In the third and fourth stages, the size of the tensor is further reduced to 64×64×384 and 32×32×768, respectively, to meet the subsequent processing requirements; The entire process ensures that image data can be effectively processed and analyzed at different stages.

3. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 1, characterized in that: The S2 at least comprises the following steps: By introducing a feature alignment module to replace the 1×1 convolutional layer in the traditional feature pyramid network, the module can learn pixel-level offsets to achieve alignment of high-resolution features with context information during upsampling; A feature selection module is added after upsampling, which can focus on low-resolution features containing rich spatial details to enhance the feature expression ability and detection performance.

4. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 3, characterized in that: The introducing of a feature alignment module comprises at least the following steps: Before feature fusion, and Features are aligned. Feature alignment can be expressed mathematically as: in yes and The concatenation of provides the spatial difference between the upsampled and corresponding bottom-up features; f o ( ) and f a ( ) represent the offset Δ learned from the spatial difference i and a function to align features according to the offset; Adding a feature selection module at least includes the following steps: To extract the feature map of spatial details, first extract the feature map C through the global average pooling operation i Global information of Then it is input into the 1×1 convolution layer and the sigmoid activation function, that is, f m ; Get the important eigenvector u and convert C i Use u to scale; Then the scaled C i Add to original C i middle; Finally, through a 1×1 convolutional layer, that is, f s , keep important channels and discard unimportant channels.

5. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 1, characterized in that: The RRPN module includes a rotated bounding box, a rotated anchor point, a candidate region learning, an optimization of a region extraction network, and an RRoI pooling layer.

6. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 5, characterized in that: The rotation detection box is represented as (x, y, h, w, θ); Where (x, y) represents the geometric center of the detection box, h is the short side of the detection box, w is the long side of the detection box, and θ is the angle between the positive x-axis and the long side of the rotated bounding box.

7. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 6, characterized in that: The application of the rotating anchor point comprises at least the following steps: The RRPN module uses 6 different angles to control the extraction of the target suspected area, namely: The length-to-width ratio adopts 3 groups (1:2, 1:5, 1:8); The scale is 3 groups (8, 16, 32); This generates 6×3×3=54 5-dimensional (x, y, h, w, θ) anchor points. For a feature map with a width and height of W×H, a number of anchor points of W×H×54 will be generated.

8. The document image processing method based on FaPN, Swin Transformer and RRPN according to claim 7, characterized in that: The application of the candidate region learning comprises at least the following steps: For the regression of the detection box, the background RoI is ignored and the smooth-L1 loss is used for the target RoI: Scale-invariant parameterized tuple v and v * The calculation of is as follows: Among them, x, x a and x * They are used to represent the predicted box, anchor box and true box respectively, and the same is true for y, h, w and θ; Function f(a,b)=a-b+kπ, where k∈Z, such that 9. The document image processing method based on FaPN, SwinTransformer and RRPN according to claim 8, characterized in that: The region extraction network is optimized so that rotational region proposals can be generated in any orientation; Therefore, the IoU calculation for the horizontal box may cause the IoU detection of the tilted detection box to be distorted. Therefore, the implementation of the tilted IoU calculation considering triangulation is used. Given a set of skew rectangles R1, R2, ..., R N , the goal is to calculate each pair of R i and R j IoU.

10. The document image processing method based on FaPN, SwinTransformer and RRPN according to claim 9, characterized in that: The RROI pooling layer uses a rotation RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction; The process of using the RRoI pooling layer to adjust the region proposal box generated by RRPN facing any direction is: Set the hyperparameters of the RRol layer to H r and W r ; Input the representation of the proposed region (x, y, h, w, θ); Pooling size (H r ,W r ), input feature map and spatial scale; For a proposal region with height h and width w, the rotated proposal region can be divided into H r ×W r sub-regions, each with the same orientation as the proposed one; Traverse all sub-regions of the candidate region and transform the sub-regions into standard rectangles through affine transformation; Traverse each sub-region and obtain The maximum value, due to the use of floor rounding when interpolating the elements in the standard rectangle.

Images