A DRRG algorithm optimization method
By improving the feature extraction network and adding attention module in the DRRG algorithm, combined with the FFM feature fusion module, the problem of DRRG algorithm not paying enough attention to small receptive texts is solved, the detection accuracy and speed is improved, and the algorithm detection in any shape text is improved. performance in.
Patent Information
- Application Number
- CN202110416706.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-19
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-04-19
AI Technical Summary
Traditional text detection methods and deep learning-based methods do not work well when detecting arbitrary shapes and bent text. Although the DRRG algorithm improves bent text detection, it has little effect on skewed text detection and the algorithm structure is too complex, which affects speed.
Based on the DRRG algorithm, the feature extraction network is improved, attention module and feature fusion module are added, and different receptive field features are fused through the FFM feature fusion module to improve the accuracy and speed of text detection.
It improves attention to small receptive texts, enhances detection accuracy, reduces computational burden, and improves the performance of the algorithm on ICDAR2015 and ICDAR2017 datasets.
Smart Images
Figure CN113159070B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of natural scene text detection algorithms, and in particular relates to a DRRG algorithm optimization method. Background Art
[0002] Traditional text detection methods and some deep learning-based text detection methods are mostly aimed at long texts and texts with regular shapes, but the detection effect on arbitrary shape texts and curved texts is not good, and the accuracy is also limited. In response to the above problems, the DRRG (Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection) algorithm was proposed in 2020. It is one of the two new text detection algorithms proposed at CVPR in 2020. It was jointly launched by the Beijing University of Science and Technology, the Joint Laboratory of Artificial Intelligence of the University of Science and Technology of China, and Tencent Technology. It uses a deep relational reasoning graph network for arbitrary shape text detection. The local graph establishes the relationship between the CNN-based textproposal model and the GCN (Graph Convolutional Network)-based deep relational reasoning network, so that the model can be trained end-to-end. DRRG adopts an FCN structure similar to the EAST algorithm, integrates 5 layers of features, and uses the GCN method to achieve a new linking method for the lack of sequence features in the EAST algorithm. It not only adds text sequence features to better predict text, but also realizes the prediction of curved text. DRRG achieved 86.56% and 67.31% in the Hmean index of ICDAR2015 and ICDAR2017 respectively, far exceeding previous text detection algorithms. Although DRRG has good results, it has little advantage in cases where there are few tilted texts and curved texts, and the algorithm structure is too complex, so it has little advantage in speed. In response to the above problems, this paper takes the DRRG algorithm as the basis, combines the characteristics of text in natural scenes, improves the feature extraction network, adds attention modules and feature fusion modules to complete the text detection task that fits the characteristics of text.
[0003] The general idea of the DRRG algorithm is as follows: the text component candidate box network shares convolutional features with the deep relational reasoning graph network, and the shared convolution uses VGG16 and FPN as the backbone network. The text candidate box network uses shared features to estimate the geometric properties of the text components. After obtaining the geometric properties, the local graph can roughly establish the links between different text components. Based on the local graph, the relational reasoning network will further infer the deep possibility of the link between the component and its adjacent components. Finally, according to the reasoning results, the text components will be summarized into a whole text instance. The network structure of the backbone network is as follows Figure 1As shown, where f i (i=1, 2, 3, 4, 5) represents the output feature maps of feature pyramids of different sizes, h i (i=1, 2, 3, 4, 5) represents the output feature map after the fusion of different receptive fields. The text candidate box network is similar to the feature pyramid structure of the EAST algorithm. The difference is that the feature pyramid in the EAST algorithm is 4 layers, while the DRRG algorithm uses 5 layers, which makes it easier to extract information at different levels. The shared features of the output are obtained through a similar feature pyramid structure. After extracting the shared features, 3*3 convolutional layers and 1*1 convolutional layers are used to predict the geometric properties of the text components. The threshold of the positive sample is set, and the threshold is applied to each prediction area, where the geometric shapes with a score exceeding the predetermined threshold are considered valid and saved for subsequent local perceptual non-maximum suppression (LNMS). The result after LNMS processing is used as the final result of the network structure.
[0004] It can be seen from the FPN structure that the small receptive field feature map is located at the bottom layer of the feature extraction network and the top layer of the feature merging network. This structure causes the small receptive field feature map to undergo only a few convolution calculations, and insufficient attention is paid to the small receptive field text. The situation with large receptive fields is the opposite. During the feature fusion process, the proportion of large receptive fields gradually decreases. All of the above situations are not conducive to subsequent processing. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a DRRG algorithm optimization method. After the feature fusion output, the FFM feature fusion module is used to fuse the bottom-layer large receptive field feature fusion result with the middle-layer medium receptive field feature fusion result, and then further fuse them with the top-layer small receptive field feature fusion output result. This solves the problems that the DRRG algorithm does not pay enough attention to small receptive field texts, the proportion of large receptive field feature maps gradually decreases during the fusion process, which is not conducive to subsequent processing, and at the same time will not bring too much burden to the network's computational workload.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A DRRG algorithm optimization algorithm includes the following implementation steps:
[0008] Step 1: Image preprocessing.
[0009] Step 101: Pre-process all images and process them to a size of 512*512 pixels for subsequent processing.
[0010] Step 2: Input the feature extraction network for feature extraction.
[0011] Step 201: The original feature extraction network of the DRRG algorithm is VGG16. In order to improve the network accuracy, increase the number of network layers, extract deep semantic information, and solve the problem that the feature map processing is less and is not conducive to subsequent processing, the original feature extraction network is replaced by a lightweight feature extraction network ResNet, and in order to extract deep semantic information, the ResNet network is deepened to 50 layers.
[0012] Step 202: In order to integrate the channel domain attention mechanism and the spatial domain attention mechanism, a lightweight hybrid attention mechanism module CBAM (Convolutional Block Attention Module) is added to the ResNet50 network.
[0013] Step 203: Use the improved ResNet50 as the feature extraction network to extract 5 layers of feature maps, which are denoted as f 5 、f 4 、f 3 、f 2 、f 1 .
[0014] Step 3: Feature fusion.
[0015] Step 301: f 1 After the deconvolution operation, the number of channels of the feature map becomes 1 / 2 of the original one, and the size of the feature map is doubled, which is recorded as h 0 . With f 2 Perform feature fusion and send it to the Up block module for convolution to collect spatial and channel information. The output is recorded as h 1 .
[0016] Step 302: f 3 and h 1 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 2 .
[0017] Step 303: f 4 and h 2 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 3 .
[0018] Step 304: f 5 and h 3 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 4 .
[0019] Step 4: FFM feature fusion.
[0020] Step 401: h0 and h 2 It is sent to the FFM feature fusion module and the output result is recorded as h5.
[0021] Step 402: h 4 And h5 are sent to the FFM feature fusion module, and the output result is recorded as h6.
[0022] Step 5: Enter the BLSTM network.
[0023] Step 501: Input the FFM feature fusion result h6 into the BLSTM network.
[0024] Step 6: Input Text-GCN deep relational reasoning network.
[0025] Step 601: The output result of the BLSTM network is processed and sent to the Text-GCN deep relationship reasoning network to perform relationship reasoning between nodes, and the output result is output as the final result.
[0026] Step 7: Output the results.
[0027] Compared with the prior art, the present invention provides a DRRG algorithm optimization method, which has the following beneficial effects:
[0028] The present invention provides a DRRG algorithm optimization method. After the feature fusion output, the FFM feature fusion module is used to fuse the bottom layer large receptive field feature fusion result with the middle layer medium receptive field feature fusion result, and then further fuse with the top layer small receptive field feature fusion output result. The problem that the DRRG algorithm does not pay enough attention to the small receptive field text, the proportion of the large receptive field feature map gradually decreases during the fusion process, which is not conducive to subsequent processing, etc. is solved, and it will not bring too much burden to the network's calculation amount, thereby improving the detection accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is the structure diagram of the original DRRG algorithm;
[0030] Figure 2 This is the structure diagram of the optimized DRRG algorithm;
[0031] Figure 3 This is a comparison chart of the results of the original algorithm and the optimized algorithm on the ICDAR2015 dataset. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] Combination Figure 1 to Figure 2 As shown, a DRRG algorithm optimization method under this implementation mode includes the following steps:
[0034] Step 1: Preprocessing of training set.
[0035] Step 101: Change the size of all ICDAR2015 training set images to 512*512 pixels before feeding them into the network, and convert the labels into a format suitable for network processing.
[0036] Step 2: Input the feature extraction network for feature extraction.
[0037] Step 201: The original feature extraction network of the DRRG algorithm is VGG16. In order to improve the network accuracy, increase the number of network layers, extract deep semantic information, and solve the problem that the feature map processing is less and is not conducive to subsequent processing, the original feature extraction network is replaced by a lightweight feature extraction network ResNet, and in order to extract deep semantic information, the ResNet network is deepened to 50 layers.
[0038] Step 202: In order to integrate the channel domain attention mechanism and the spatial domain attention mechanism, a lightweight hybrid attention mechanism module CBAM (Convolutional Block Attention Module) is added to the ResNet50 network.
[0039] Step 203: Use the improved ResNet50 as the feature extraction network to extract 5 layers of feature maps, which are denoted as f 5 、f 4 、f 3 、f 2 、f 1 , the size of the feature map decreases successively to 1 / 2 of the previous feature map, and the number of channels is: 64, 128, 256, 512, 512.
[0040] Step 3: Feature fusion.
[0041] Step 301: f 1 After the deconvolution operation, the number of channels of the feature map becomes 1 / 2 of the original one, and the size of the feature map is doubled, which is recorded as h 0 At this time, the number of feature map channels is 256, and the feature map size is 1 / 16 of the original image. Channel fusion is used with f2 Perform feature fusion and send it to the Up block module (the module structure is 3*3 depth convolution + Relu activation function + 1*1 point-by-point convolution + Relu activation function + 3*3 depth convolution), perform convolution to collect spatial and channel information, and the output is recorded as h 1 , at this time the number of channels is 128 and the feature map size is 1 / 8 of the original image.
[0042] Step 302: Using channel fusion, f 3 and h 1 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 2 , at this time the number of channels is 64 and the feature map size is 1 / 4 of the original image.
[0043] Step 303: Adopt channel fusion method, f 4 and h 2 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 3 , at this time the number of channels is 32 and the feature map size is 1 / 2 of the original image.
[0044] Step 304: Adopt channel fusion method, f 5 and h 3 After fusion, it is sent to the Up block module for convolution to collect spatial and channel information, denoted as h 4 , at this time the number of channels is still 32, and the feature map size is the same as the original image size.
[0045] Step 4: FFM feature fusion.
[0046] Step 401: Deconvolve h twice 0 and h 2 It is sent to the FFM feature fusion module, and the output result is recorded as h5. At this time, the number of feature channels is 64 and the size of the feature map is 1 / 4 of the original image.
[0047] Step 402: h 4 After two deconvolutions, it is sent to the FFM feature fusion module together with h5, and the output result is recorded as h6. At this time, the number of channels is 32, and the size of the feature map is the same as the original image size.
[0048] Step 5: Enter the BLSTM network.
[0049] Step 501: h6 is input into the BLSTM network. The BLSTM network can bring sequence features to the entire algorithm, make preliminary preparations for the next step of node relationship reasoning, and thus improve the algorithm accuracy.
[0050] Step 6: Enter the Text-GCN deep relational reasoning network.
[0051] Step 601: The output result of the BLSTM network is processed and sent to the Text-GCN deep relationship reasoning network to perform relationship reasoning between nodes, and the output result is output as the final result.
[0052] Step 7: Output the results, feed them into the loss function, and update the network parameters.
[0053] Step 701: Use the output results of the deep relational reasoning network and the processed training set labels as inputs to the loss function, calculate the loss, and use the optimization function to update the entire network parameters.
[0054] Step 8: Repeat the training process to train the DRRG algorithm to improve accuracy.
[0055] Step 801: Repeat training steps 2-7, train the entire network, and update network parameters until the model reaches the best effect.
[0056] Step 9: Use the evaluation function to calculate the network accuracy and compare it with other algorithms.
[0057] Step 901: Use the evaluation function to calculate the precision P, recall R and F value of the algorithm and compare them with other algorithms. All of them are improved. The comparison results are as follows: Figure 3 .
Claims
1. A DRRG algorithm optimization method, characterized in that: The following steps are involved: Step 1: training set preprocessing; Step 2: Input the processed training set images into the main network and replace the feature extraction network of VGG16 with ResNet50 for feature extraction; Step 201: extract 5 layers of feature maps, which are denoted as f5, f4, f3, f2, and f1 in sequence; Step 3: Feature fusion; Step 301: After the deconvolution operation, the number of channels of the feature map f1 becomes 1 / 2 of the original one, and the size of the feature map is doubled, denoted as h0, and the feature is fused with f2 and sent to the Up block module. The Up block module structure is 3*3 depth convolution + Relu activation function + 1*1 point-by-point convolution + Relu activation function + 3*3 depth convolution. Convolution is performed to collect spatial and channel information, and the output is denoted as h1; Step 302: f3 and h1 are fused and sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h2; Step 303: f4 and h2 are fused and sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h3; Step 304: f5 and h3 are fused and sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h4; Step 4: FFM feature fusion; Step 401: h0 and h2 are sent to the FFM feature fusion module, and the output result is recorded as h5; Step 402: h4 and h5 are sent to the FFM feature fusion module, and the output result is recorded as h6; Step 5: Input BLSTM network; Step 501: Input the FFM feature fusion result h6 into the BLSTM network; Step 6: Input the Text-GCN deep relational reasoning network; Step 601: The output result of the BLSTM network is processed and sent to the Text-GCN deep relationship reasoning network to perform relationship reasoning between nodes, and the output result is output as the final result.
2. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 201, the optimized feature extraction network ResNet50 is used to extract five layers of feature maps, which are denoted as f5, f4, f3, f2, and f1 respectively. The size of the feature maps decreases successively to 1 / 2 of the previous feature map, and the number of channels is 64, 128, 256, 512, and 512 respectively.
3. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 301, the number of channels of the f1 feature map is reduced to 1 / 2 of the original after the deconvolution operation, and the size of the feature map is doubled, which is recorded as h0. At this time, the number of channels of the feature map is 256, and the size of the feature map is 1 / 16 of the original image. The feature is fused with f2 using the channel fusion method, and sent to the Upblock module for convolution to collect spatial and channel information. The output is recorded as h1. At this time, the number of channels is 128, and the size of the feature map is 1 / 8 of the original image.
4. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 302, a channel fusion method is adopted, f3 and h1 are fused, and the fusion is sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h2. At this time, the number of channels is 64, and the size of the feature map is 1 / 4 of the original image.
5. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 303, a channel fusion method is adopted, f4 and h2 are fused, and the fusion is sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h3. At this time, the number of channels is 32, and the size of the feature map is 1 / 2 of the original image.
6. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 304, a channel fusion method is adopted, f5 and h3 are fused, and the fusion is sent to the Up block module for convolution to collect spatial and channel information, which is recorded as h4. At this time, the number of channels is still 32, and the size of the feature map is the same as the original image size.
7. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 401, h0 and h2 after two deconvolutions are sent to the FFM feature fusion module, and the output result is recorded as h5. At this time, the number of feature channels is 64, and the size of the feature map is 1 / 4 of the original image.
8. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 402, h4 is deconvolved twice and then sent to the FFM feature fusion module with h5. The output result is recorded as h6. At this time, the number of channels is 32, and the size of the feature map is the same as the original image size.
9. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 501, h6 is input into the BLSTM network. The BLSTM network can bring sequence features to the entire algorithm, make preliminary preparations for the next step of node-to-node relationship reasoning, and thus improve the algorithm accuracy.
10. A DRRG algorithm optimization method according to claim 1, characterized in that: In step 601, the output result of the BLSTM network is processed and sent to the Text-GCN deep relationship reasoning network to perform inter-node relationship reasoning, and the output result is output as the final result.