Improved SwinTextSpotter v2-based robust generalization detection and identification method for plant station wiring image text

Abstract

The invention belongs to the field of smart power grids and computer vision, and particularly relates to an improved SwinTextSpotter v2-based robust generalization detection and identification method for a power station wiring image text. Comprising the following steps of 1, sending an input image into a text detection and recognition network based on multi-modal learning for training and prediction, extracting a backbone network through shared features to obtain a shared feature map, and further sending the shared feature map into a text detection module to obtain a text detection result and a text feature map; 2, sending the text feature map into a visual feature extraction and prediction module to obtain a feature sequence, and then matching the predicted feature sequence with the standard representation obtained by the character structure feature extraction and prediction module to obtain an identification result; and the like. According to the method, the detection and recognition precision of the model on irregular texts and Chinese character texts is improved robustly, and the generalization performance of detection and recognition on various types of wiring texts and texts is improved.

Description

Technical Field

[0001] The invention belongs to the field of smart grid and computer vision, and in particular relates to a robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2. Background Art

[0002] As the scale of power grid drawings expands, traditional manual identification of drawing content is not only inefficient, but also prone to errors. Automation of power grid drawing management and search is very urgent. Due to the complexity of information in power grid wiring diagrams, in order to realize the automation process, it is particularly important to detect the position of text annotations and recognize the text content in power drawings. The text annotation forms in power grid plant wiring diagrams are often diverse, with different shapes, directions and sizes. The characters include not only Arabic numerals and letters, but also many Chinese characters. In addition, the scale of characters inside the text in the plant wiring diagram is often different, and contains a large number of similar and easily confused characters that are not based on natural semantics.

[0003] In recent years, methods based on deep learning have achieved remarkable results in the field of optical character recognition. OCR technology based on deep learning has become a potential way to automate and intelligentize power grid wiring diagrams. Applying deep learning to text recognition in engineering design drawings can effectively realize the intelligent automatic recognition task of power grid wiring diagrams.

[0004] The closest existing technology and its review:

[0005] Li et al (SHANBIN L, HAOYU W, JUNHAO Z. Electrical cabinet wiring detection method based on improved YOLOv5 and PP-OCRv3; proceedings of the2022ChineseAutomation Congress,CAC 2022,November 25,2022-November 27,2022,Xiamen,China,F,2022[C].Institute of Electrical and Electronics Engineers Inc.) used YOLOv5 and PP-OCRv3 to improve the accuracy of text detection and recognition in electrical cabinet wiring diagrams; Wei Wei et al. (Wei Wei, Long Na, Tian Yue, et al. Research on text detection method of power equipment nameplate based on improved DBNet [J]. High Voltage Technology, 2023, 49: 63-67.) used pixel-level interpolation and pooling on the basis of DBNet to more accurately detect the text contour in the power equipment nameplate; Liu Wei et al. (Liu Wei. Research on text recognition and wiring verification system of secondary circuit terminal block [D]. Enshi: Hubei University for Nationalities, 2023.) introduced transfer learning to train the OCR model for the application scenario of terminal block design drawings. The research results show that although the selected deep learning method can detect and recognize text in simple scenarios, there are still challenges for more complex text detection and recognition scenarios, such as text located near the primitives and text with different scales and mixed horizontal and vertical characters involved in the plant wiring diagram dataset.

[0006] SwinTextSpotter v2 is an end-to-end text detection and recognition model designed to handle text recognition tasks in complex scenarios. Unlike traditional convolutional neural networks (CNNs), SwinTextSpotter leverages the powerful feature extraction and modeling capabilities of SwinTransformer

[58] to effectively capture long-range contextual information and global features, thereby improving the detection of text regions and the recognition accuracy of text content. It uses a query-based detector to achieve iterative optimization and gradually improves text detection accuracy. It uses a recognition transformation module to connect the detector and recognizer to eliminate the error accumulation problem in the two-stage text detection and recognition process. However, SwinTextSpotter v2 performs poorly for Chinese text that does not have natural semantics and small characters in irregular text, and does not have the ability to generalize to wiring diagrams of multiple types and specifications. Summary of the invention

[0007] The object of the present invention is to provide a robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2.

[0008] In order to solve the problems of robustness and generalization of existing text detection and recognition methods for power grid plant wiring diagrams, the present invention introduces an end-to-end integrated baseline model to jointly train the two tasks of text position detection and content recognition, which greatly reduces error accumulation. Through a series of deep learning-based strategies, the robustness and generalization performance of the model are effectively improved.

[0009] In the preprocessing stage, a training set screening strategy based on data mining is designed to reduce the cost of end-to-end model joint training while ensuring model accuracy, thereby improving the training efficiency of the end-to-end model.

[0010] In the shared feature extraction stage, a lightweight text region mask generation branch based on the improved MobileNetv3 is introduced to effectively solve the problem of poor model generalization performance caused by inconsistent text region distribution between plant wiring diagrams of different types and specifications.

[0011] In the main part of the model, a text detection and recognition network based on multimodal learning is constructed. A visual feature extraction and prediction module based on multi-level attention and a character structure feature extraction and prediction module based on CCR-CLIP are introduced to fit the information of visual modality and character structure feature modality. A multimodal prediction fusion module based on contrastive learning is designed to fuse visual features with character structure features, so as to robustly improve the detection and recognition accuracy of irregular text and Chinese text.

[0012] In the processing stage, a fine-tuning post-processing strategy based on prior knowledge is designed to solve the problem of semantic segmentation errors of similar texts in plant wiring diagrams in a targeted manner according to the specific grammar of power grid drawings; in the formulation of learning strategies, an incremental learning strategy for text detection and recognition of multiple types of wiring diagrams based on knowledge distillation is designed, which enables the model to effectively retain the memory of old data while gradually adapting to new data, thereby improving the generalization performance of text detection and recognition of multiple types of wiring diagrams.

[0013] Experimental results show that the proposed model can accurately and robustly detect and recognize texts of different scales and languages ​​in plant wiring diagrams, and has strong generalization ability for text detection and recognition in wiring diagrams of different types and specifications.

[0014] Technical solution of the present invention:

[0015] A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2, characterized by comprising the following steps:

[0016] Step 1: Send the input image to the text detection and recognition network based on multimodal learning for training and prediction, extract the shared feature backbone network, obtain the shared feature map, and further send it to the text detection module to obtain the text detection result and text feature map;

[0017] Step 2: Send the text feature map to the visual feature extraction and prediction module to obtain a feature sequence, and then match the predicted feature sequence with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result;

[0018] Step 3: Send the text recognition results to the fine-tuning post-processing module, fine-tune some recognition results based on the prior knowledge of the power grid, and obtain the final text recognition results;

[0019] Step 4: Calculate the detection and recognition losses by combining the text detection results and text recognition results with the corresponding true values;

[0020] Step 5: Jointly optimize the entire network model based on the loss;

[0021] Step 6: Divide the general scenario text dataset and the power plant wiring diagram dataset into a training set and a test set respectively. Use the training set screening strategy based on data mining to screen the general scenario training set for model pre-training. The training set for model fine-tuning training is the power plant wiring diagram training set. Both pre-training and fine-tuning training are cycled through steps 1-5 until the network converges, and the model file is saved.

[0022] Step 7: Use the new plant wiring diagram dataset to build a two-stream network for feature extraction based on text region mask generation, and use a knowledge distillation-based incremental learning strategy for multi-category wiring diagram text detection and recognition to perform incremental learning on the current model;

[0023] Step 8: Input the test drawing into the model designed and trained in steps 1-7 to obtain the detection and recognition results.

[0024] By adopting the above scheme, the beneficial effects of the present invention are:

[0025] 1. The present invention constructs a text detection and recognition network based on multimodal learning, which robustly improves the detection and recognition accuracy of the model for irregular text and Chinese character text.

[0026] 2. The present invention designs a fine-tuning post-processing strategy based on prior knowledge to specifically solve the problem of semantic segmentation errors of similar texts in plant wiring diagrams.

[0027] 3. The present invention designs a training set screening strategy based on data mining, which effectively improves the training and prediction efficiency of the end-to-end text detection and recognition model.

[0028] 4. The present invention has developed an incremental learning strategy for text detection and recognition of multiple types of wiring diagrams based on knowledge distillation. By proposing a dual-stream network for feature extraction based on text region mask generation for model shared feature extraction, it effectively solves the problem of poor model generalization performance due to inconsistent text region distribution between plant wiring diagrams of different types and specifications, and improves the generalization performance of text detection and recognition of multiple types of wiring diagrams. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a basic flow chart of the robust generalized detection and recognition method of plant wiring diagram text based on improved SwinTextSpotter v2 of the present invention;

[0030] Figure 2 It is a text detection and recognition network structure diagram based on multimodal learning in step 2 of the present invention;

[0031] Figure 3 It is the multi-level attention branch structure diagram mentioned in step 2 of the present invention;

[0032] Figure 4 It is the structure diagram of the text recognition codec based on proportional upsampling mentioned in step 2 of the present invention;

[0033] Figure 5 It is the structure diagram of the character structure feature extraction and prediction module based on CCR-CLIP mentioned in step 2 of the present invention;

[0034] Figure 6 It is a structural diagram of the multimodal prediction fusion module based on contrastive learning mentioned in step 2 of the present invention;

[0035] Figure 7 It is the overall framework diagram of the fine-tuning post-processing strategy based on prior knowledge in step 3 of the present invention;

[0036] Figure 8 is a flow chart of the training set screening strategy based on data mining in step 6 of the present invention;

[0037] Fig. 9 It is the overall framework of the incremental learning strategy for text detection and recognition of multiple types of wiring diagrams based on knowledge distillation in step 7 of the present invention;

[0038] Fig.10 It is the feature extraction dual-stream network structure diagram based on text area mask generation mentioned in step 7 of the present invention;

[0039] Fig.11 It is a quantitative comparison of the performance of the embodiment of the present invention and other OCR models;

[0040] Fig.12It is a comparison of the generalization performance of the embodiments of the present invention on the new and old plant wiring diagram datasets. DETAILED DESCRIPTION

[0041] Aiming at the problems of robustness and generalization of the existing power grid plant wiring diagram text detection and recognition methods, the present invention proposes a robust generalization detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2. The basic flow chart of the present invention is as follows: Figure 1 shown.

[0042] In order to improve the robustness of the model for detecting and recognizing irregular text, small characters, and Chinese characters that are not based on natural semantics, a text detection and recognition network based on multimodal learning is constructed. The overall structure is as follows: Figure 2 As shown in Figure 2, the multi-level attention branch structure is shown in Figure 2. Figure 3 As shown in the figure, the structure of the text recognition codec based on equal-proportional upsampling is as follows Figure 4 As shown in the figure, the structure diagram of the character structure feature extraction and prediction module based on CCR-CLIP is as follows Figure 5 As shown in the figure, the structure of the multimodal prediction fusion module based on contrastive learning is shown in Figure 6 shown.

[0043] In order to solve the problem of semantic segmentation errors of similar texts in plant wiring diagrams, a fine-tuning post-processing strategy based on prior knowledge is designed. The overall framework diagram is shown in the figure. Figure 7 shown.

[0044] In order to reduce the size of the general training set and improve the efficiency of model pre-training, a training set screening strategy based on data mining is designed. The flow chart of this strategy is shown in the figure. Figure 8 shown.

[0045] In order to achieve a balanced learning of new and old knowledge, an incremental learning strategy for text detection and recognition of multiple types of wiring diagrams based on knowledge distillation is developed. The overall framework is as follows: Fig. 9 In order to solve the problem of poor model generalization performance due to inconsistent text area distribution between plant wiring diagrams of different types and specifications, a feature extraction dual-stream network based on text area mask generation is constructed in the incremental learning model. The structure diagram is shown in Fig.10 shown.

[0046] In order to enable the model to have text detection and recognition capabilities, and on this basis to achieve accurate detection and recognition of plant wiring diagram text, the model is first pre-trained on a general scene text dataset, iterated 800K times, so that the model has strong text detection and recognition generalization capabilities. Then fine-tuning training is performed on the plant wiring diagram training set, iterated 300K times, so that the model can more accurately adapt to the specific text structure and layout characteristics in the plant wiring diagram.

[0047] During the pre-training process, the learning rate decay strategy is adopted, and the initial learning rate is 2.5×10 -5 When the number of iterations reaches 400K, the learning rate decays to 2.5×10 -6 When the number of iterations reaches 600K, the learning rate decays to 2.5×10 -7 ; During fine-tuning training, the learning rate is set to 2.5×10 -7 The batch size is the number of samples passed into the model in one iteration during the training process. Due to the limitation of video memory resources, the batch size of the text detection stage is set to 4, that is, 4 complete images to be tested are passed in one iteration, and the batch size of the text recognition stage is set to 64, that is, 64 text images cropped according to the bbox are passed in one iteration. The λ in the CCR-CLIP module is the proportional coefficient of the loss function, which is used to reconcile and Set to 0.80; the aspect ratio threshold γ in the fine-tuning post-processing module is set to 6.00.

[0048] The quantitative analysis results of the accuracy of the present invention and other OCR models are as follows Fig.11 As shown in the figure, the generalization performance comparison on the new and old plant wiring diagram dataset is Fig.12 The experimental results show that the present invention can significantly improve the robustness and generalization ability of text detection and recognition in plant wiring diagrams through a series of improvements to SwinTextSpotter v2.

[0049] The following combination Figure 1 The flowchart shown further illustrates the technical solution of the present invention, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0050] Step 1: Feed the input image into the text detection and recognition network based on multimodal learning (such as Figure 2 The shared feature map is obtained by extracting the backbone network through the shared feature, and then sent to the text detection module to obtain the text detection result and text feature map.

[0051] The shared feature extraction backbone network is a feature extraction backbone network based on Swin Transformer and FPN;

[0052] The text detection module uses the detector structure of SwinTextSpotterv2. Specifically, it generates and optimizes a set of learnable candidate boxes to replace the traditional RPN to generate a large number of candidate objects. Each stage uses a dynamic Transformer encoder to encode the feature map. The latter stage can access the information stored in the candidate features in the previous stage. Through multiple stages of iteration and continuous optimization, the final predicted text box is output.

[0053] Step 2: Send the text feature map to the visual feature extraction and prediction module to obtain a feature sequence, and then match the predicted feature sequence with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result.

[0054] In order to better extract text features and robustly solve the problem of missing small characters in horizontal and vertical mixed texts, a visual feature extraction and prediction module based on multi-level attention is proposed. Through local attention and global attention branches, the receptive field features of different scales are extracted. The introduction of dilated convolution can further capture the relationship between long-distance pixels and improve the fitting ability of neighborhood and global features. With the help of the characteristics of joint optimization, the detection branch is optimized through recognition loss to achieve the purpose of correcting the text detection and segmentation results. In the text recognition codec, proportional upsampling is added to reduce the loss of small character features in the serialization process and improve the recall rate of small characters.

[0055] In order to improve the recognition accuracy of Chinese characters and solve the problem of "long-tail distribution" of Chinese characters, a character structure feature extraction and prediction module based on CCR-CLIP was introduced. The image encoder and character encoder were constructed and pre-trained based on contrastive loss. The character structure features were introduced into the visual model, which improved the model's recognition ability of Chinese characters from the perspective of multimodal learning.

[0056] In order to improve the fitting effect of multimodal prediction information, a multimodal prediction fusion module based on contrastive learning is designed. The prediction of the visual branch and the character specification representation of the character structure feature branch are multiplied after passing through two linear layers. Through a series of convolutional structures and linear layers, the text recognition accuracy of complex models in complex scenes is improved.

[0057] The multi-level attention-based visual feature extraction and prediction module includes multi-level attention branches (such as Figure 3 ), recognition conversion and text recognition codec based on proportional upsampling (as shown in Figure 4 The recognition conversion module is the original structure of the SwinTextSpotter v2 network, which is used to generate a tight mask of the text area and realize the joint optimization of text detection and recognition in two stages.

[0058] The multi-level attention branch includes local attention and global attention. The local attention branch uses the residual dilated convolution structure and the window self-attention mechanism, and focuses on the local detail features of the image. The dilated convolution can build a multi-scale receptive field for the model in each convolution layer, which is beneficial for the model to extract texture features of different granularities. The residual structure can learn and save more subtle spatial information during the back propagation process. Based on this, the residual dilated convolution structure enables the features of irregular text areas to be extracted more accurately, and the irregular characters in the text can also be given more reasonable and accurate weights in the subsequent attention mechanism. Specifically, the RoI feature map Input the residual hole convolution structure as follows:

[0059] X 1 =ReLU(DialatedResLayer(F))

[0060] X = DialatedResLayer(X 1 )+Conv 1×1 (F)

[0061] Among them, DilatedResLayer(·) consists of a dilated convolution layer with a convolution kernel size of 5×5 and a dilation rate of 2, and a BatchNorm layer. Next, the fitted features are fed into the window self-attention to capture local dependencies, as follows:

[0062] Q=XW q ,K=XW k, K=XW v

[0063]

[0064] in, They represent the query, key, and value matrices in the self-attention mechanism, d is the dimension of the key / value matrix, and M 2 is the number of sampling points in a window. is the relative position deviation. The relative position encoding of the self-attention weight matrix is ​​realized by introducing B, so as to construct the local dependency relationship within the image window. Finally, the obtained window self-attention fitting result is passed through the feedforward neural network MLP to obtain the nonlinear transformation of the feature, so that the network can capture higher-order image features. The global attention branch uses hollow convolution and multi-head self-attention mechanism, mainly focusing on the global contour features of the image. Hollow convolution is used to improve the convolution receptive field, and a feature map that fits the neighborhood spatial information is obtained. The feature map is used as the query matrix, and the feature map after global average pooling is used as the key and value matrix. The global dependency is captured through multi-head self-attention, as shown in the following formula:

[0065] X'=ReLU(DialatedResLayer(F))

[0066] Q′=XW q ′,K′=AP(X′)W k ′,V′=AP(X′)W v '

[0067] F global =MLP(MultiHeadSelfAttn(Q′,K′,V′))

[0068] Among them, AP(·) is the global average pooling layer, MultiHeadSelfAttn(·) is the multi-head self-attention mechanism, and the number of heads is 8. Through the above structure, each position of the input feature image F processed by the dilated convolution interacts with the global features (i.e., K and V after pooling) to form a self-attention output. This mechanism can be understood as comparing each position with the global information of the entire image to extract long-distance dependencies and global contextual relationships. Through the multi-head setting, this structure can capture rich contextual information from different subspaces, improving the network's feature expression ability and global feature fitting effect in complex scenes.

[0069] The text recognition codec based on proportional upsampling generally follows the architecture of SwinTextSpotter v2. However, during the experiment, it was found that the common problem of such RNN / Transformer-based text recognizers is that they all require text features to be sent to the recognizer in the state of "width>height", that is, when encountering vertical text, they will be rotated and flattened. At the same time, before text prediction, the input two-dimensional feature map will be downsampled into a one-dimensional feature sequence by column, which will lead to the loss of high-frequency features and details. That is, the information in the side-by-side numbers in these "flattened" horizontal and vertical mixed texts is easily lost by downsampling, and the proportion of information loss becomes more serious with the number of side-by-side numbers. This can also explain why three side-by-side numbers are more likely to be misrecognized. In order to solve this problem and take into account the optimized model scale, based on the original SwinTextSpotterv2 text recognition codec architecture, a bilinear interpolation upsampling layer is simply introduced in the sequence encoding to convert the text feature map (where B′ is the batch size and C is the number of feature channels) Make the image more detailed and the pixels smoother, so that more high-frequency information can be retained during downsampling.

[0070] Character structure feature extraction and prediction module based on CCR-CLIP (such as Figure 5As shown in the figure, the recognition result of Chinese characters is obtained by extracting the structural stroke features of Chinese characters, and the recognition result of Chinese characters is obtained by contrastive learning. It is not limited by the frequency of occurrence of Chinese characters in the training set and whether the training is sufficient. It can greatly improve the recognition ability of similar and easily confused Chinese characters and "zero-sample" Chinese characters. It is composed of an image encoder and a character encoder. This module obtains the character standard representation P through separate pre-training, and performs multimodal fusion with the prediction of the visual branch as the prediction of the character feature branch to obtain the final prediction result. The training set of this module is the printed pictures of all characters and the radical stroke sequence of the characters. The image encoder is responsible for extracting the visual features of the input character image, while the text encoder extracts the features of the corresponding radical sequence. Finally, two contrast losses are used to supervise the model. Specifically, the image encoder uses Resnet-50 as the backbone network to obtain the image feature. Then after global average pooling, we get f c Embedded in the visual feature space, as follows:

[0071] I=f c W c

[0072] in, is the projection matrix, and C′ is the alignment dimension. The text encoder consists of 2 layers of Transformer encoders and an embedding layer. Through the encoder, the radical sequence R is encoded into a sequence feature Corresponding to the output features of each time step, l is the length of the radical sequence, is considered to be the full feature of R, and D is f l r Finally, it is embedded into the text feature space to obtain T, as follows:

[0073] T=f l r W r

[0074] in, is the projection matrix. In order to match the image and character structure features, the contrast loss between the image and character structure is designed Make the loss between the corresponding image and character structure as small as possible, and the loss between the non-corresponding image and character structure as large as possible, as shown in the following formula:

[0075]

[0076] Where N is the total number of characters in the dictionary. In order to reduce the prediction error caused by different font styles and similar characters, the contrast loss between the visual features of the input images with the same radical stroke labels is introduced. Make the feature loss between similar characters as large as possible, as shown in the following formula:

[0077]

[0078] in, is a set of visual features I with the same radical stroke sequence; thus, the final loss function of the CCR-CLIP model is obtained as follows:

[0079]

[0080] From this point on, the CCR-CLIP model can be trained using printed Chinese character images and a text encoder can be used to generate normalized representations of all candidate Chinese characters.

[0081] The designed multimodal prediction fusion module based on contrastive learning (such as Figure 6 As shown in the figure, the prediction of the visual branch The character specification representation P of the character structure feature branch is multiplied by two linear layers, and then further predicted and fused through a set of depth-wise separable convolutions. Finally, the final text recognition result y is obtained through Softmax, as shown in the following formula:

[0082] F′=Linear(ReLU(Linear(F O )))

[0083] P′=Linear(ReLU(Linear(P)))

[0084] y = Softmax(DSConv(F′P′ T ))

[0085] Among them, the dimensions of F′ and P′ are the same as F o The same dimension as P.

[0086] Step 3: Send the text recognition results to the fine-tuning post-processing module, fine-tune some recognition results based on the prior knowledge of the power grid, and obtain the final text recognition results.

[0087] The prior knowledge includes electrical component naming rules, identification writing rules, etc.

[0088] Fine-tuning post-processing strategies based on prior knowledge (e.g. Figure 7 The method shown in the figure includes adjacent text segmentation post-processing based on regular expressions and text segmentation inconsistency and missed detection post-processing based on merged reconstruction, which can improve the text detection and recognition performance in a targeted manner for the specific scenario of plant wiring diagrams.

[0089] Specifically, the adjacent text segmentation post-processing based on regular expressions is used to solve the problem that the text positions are too close and thus detected as a whole. The text segmentation inconsistency and missed detection post-processing based on merge reconstruction is used to solve the problem that the text logo as a whole is recognized as two parts due to the long distance and the "I" character is missed. The text detection and recognition results obtained from the end-to-end text detection and recognition network are sent to this module, and the post-processing operations of the two branches can be completed in parallel according to the screening conditions. The post-processing strategies of the two branches are described in detail below.

[0090] The specific steps for post-processing of adjacent text segmentation based on regular expressions are as follows:

[0091] S3.1.1 Length threshold screening:

[0092] After text detection and recognition are completed, based on the "text aspect ratio" screening strategy, all text prediction boxes with aspect ratios greater than the set threshold γ are extracted. This process can effectively screen out long and thin text boxes that may cause segmentation problems.

[0093] S3.1.2 Regular expression matching:

[0094] Use regular expressions to split long texts to ensure that the split text boxes conform to the set format. Specifically, set fixed structures such as "35KV", "110KV", "Backup + Numbers", "Chinese Characters + Numbers", etc., and use regular expressions to perform pattern matching and splitting.

[0095] S3.1.3 Text box cutting:

[0096] According to the results of regular expression matching and segmentation, the text frame is cut with the help of the relative position of the text content to obtain several processed new text frames.

[0097] The specific methods for post-processing inconsistent text segmentation and missed detection based on merge reconstruction are as follows:

[0098] S3.2.1 Keyword screening:

[0099] Filter the text boxes containing the keywords "P", "Q", "Temperature" and "Gear". Subsequent operations will be based on these text boxes.

[0100] S3.2.2 Extended Merger:

[0101] For the filtered text boxes, if there are and only the above keywords, they will be extended to the right by a certain pixel range. If there are text boxes with digital text content within the range, they will be merged with the original text boxes.

[0102] S3.2.3 Traversal reconstruction:

[0103] To solve the problem of missing the "I" character, for all text boxes containing the above keywords, a certain pixel range is traversed downward. If there is a text box with digital text content in a certain area below, the digital text box is extended to the left by a certain distance (the distance is determined by the position of the text box containing the keyword above it) and reconstructed into a new text box.

[0104] Step 4: Calculate the detection and recognition losses by comparing the text detection results and text recognition results with the corresponding true values.

[0105] Step 5: Based on the loss, jointly optimize the entire network model.

[0106] Step 6: Train the model through two stages: pre-training and fine-tuning training. Repeat steps 1-5 until the network converges and save the model file.

[0107] When training the algorithm model of the present invention, a strategy of pre-training first and then fine-tuning training is adopted, that is, the model is first pre-trained in a general scenario using a general scenario text dataset to enable the model to have basic text detection and recognition capabilities; and then the model is fine-tuned using the power grid plant wiring diagram dataset.

[0108] As an embodiment, the general scene text datasets are ICDAR2019-LSVT, ICDAR2017-RCTW, ICDAR2015 and Chinese_Dataset, and the general scene training sets are screened using a training set screening strategy based on data mining for model pre-training.

[0109] As an example, the power grid plant wiring drawing data set includes 113 drawings and 23,377 text images of the 10KV switch station, 35KV substation, 110KV substation and ultra-high voltage power station of State Grid Shanghai Electric Power Company.

[0110] Furthermore, the pre-training process applies a training set screening strategy based on data mining. First, the network model constructed in steps 1-5 is roughly trained using the original general scene data set to obtain a low-precision model. The low-precision model is used to perform high-confidence data filtering on the training set data. Then, the model based on the improved PP-OCRv3 (reference [1]) is used as a high-precision model to perform low-confidence data filtering. After two data filterings, a high-quality training set is obtained. Finally, the high-quality training set is used to perform fine training and iteration on the low-precision model. The details are as follows: (e.g. Figure 8 )

[0111] (1) High-confidence data filtering based on low-precision models

[0112] First, use part of the training data to perform rough training (400K iterations) on the network model constructed in steps 1-5 to obtain a low-precision model. Although the model has limited accuracy, it can quickly predict large-scale data. Using this low-precision model, text detection and text recognition are performed on a general scene text dataset of tens of millions, and text prediction boxes with a confidence level greater than 0.95 are screened out. This part is considered to be redundant text data because they contribute less to the subsequent improvement of model accuracy. For the training images where these redundant texts are located, if there is only this redundant text in the training image, the entire sample is deleted from the training set; if there are other texts in this training image in addition to this redundant text, the training image is retained in the training set.

[0113] (2) Low confidence data filtering based on high-precision models

[0114] After filtering out high-confidence redundant text data through the high-confidence data filtering step, in order to further filter out low-confidence training data, a high-precision model is introduced to predict the remaining training data. For text prediction boxes with a confidence of less than 0.15, they are classified as negative samples with poor quality or difficult to identify, which may interfere with model training. For these negative samples, their true value identifiers are directly deleted in the training set identifier file. If there is only one text in the training image where the negative sample is located, the entire image is deleted from the training set.

[0115] Specifically, the high-precision model uses an algorithm model based on the improved PP-OCRv3 (reference [1]), which does not require training and can be directly used for prediction.

[0116] Through the above steps, high-confidence redundant data and low-confidence negative data can be filtered out from the general scenario training set, and the remaining training data are all high-quality training data that have a positive impact on the next stage of pre-training of the model.

[0117] (3) Detailed training and iteration

[0118] The screened high-quality training data is used to fine-tune the network model constructed in steps 1-5, and it is iterated again 400K times to obtain a higher-precision network model.

[0119] Compared with using the original training set for pre-training, the introduction of high-quality training sets not only improves the efficiency of pre-training, but also effectively improves the accuracy of the model, enabling it to have stronger text detection and text recognition capabilities in general scenarios. The model accuracy obtained by fine training is higher than that of low-precision models and high-precision models, and can be directly used for subsequent fine-tuning training based on plant wiring diagram drawings.

[0120] Step 7: Use the new plant wiring diagram dataset to build a dual-stream network for feature extraction based on text region mask generation, and use a knowledge distillation-based incremental learning strategy for multi-category wiring diagram text detection and recognition to perform incremental learning on the current model.

[0121] Incremental learning strategy for text detection and recognition of multi-category wiring diagrams based on knowledge distillation (e.g. Fig. 9 As shown in the figure, the model obtained through initial training is used as the original model. On this basis, all parameters of the text content recognition stage (step 2) are frozen into the new model, and only the model parameters involved in the feature extraction and text detection stages (step 1) are incrementally learned.

[0122] The specific training strategies are as follows:

[0123] Assume that before time t, the original model has been trained based on the historical wiring diagram Data old At time t, a new set of wiring diagrams to be trained appears. old Randomly retain some images to construct some historical wiring diagram datasets old-part , together with the new wiring diagram training set Data new This constitutes the training set for this knowledge distillation. In addition, all parameters of the original model's text content recognition stage are frozen into the new model.

[0124] In the process of knowledge distillation of the new model, assuming batch size = B, then in Data old-part and Data new Sample B / 2 samples for training respectively, and send the input to the feature extraction two-stream network based on text region mask generation to extract image features and generate rough text region masks respectively, and send the masked feature map to the subsequent module to predict text detection and recognition results. The detection and recognition results are compared with the true value label to calculate the prediction loss The prediction loss includes the original model's original loss And the weighted cross entropy loss of the text region mask generation branch As follows:

[0125]

[0126] Among them, λ is the mask branch ratio coefficient, is the L1 loss of the text candidate box bbox in the text detection stage, is the cross entropy loss of the text segmentation result in the text detection stage, is the loss of the text recognition stage.

[0127] In order to realize the supervision of the original model on the training of the new model, Datold-part The training samples are fed into the original model for prediction and the soft labels of the bbox and text prediction sequence are output. The soft labels are used as prior knowledge and the prediction results of the new model to calculate the distillation loss. As follows:

[0128]

[0129] Among them, P d ,Q d Represent the prediction vectors of the original model and the new model bbox, P r ,Q r Represent the vectors of the original model and the new model text prediction sequence respectively. d (i),q d (i) represents the bbox prediction value of the original model and the new model at the i-th point, p r (i),q r (i) represents the predicted value of the text prediction sequence of the original model and the new model at the i-th point. N and M represent the number of bboxes and text prediction sequences predicted in the current round, respectively. old-part The distillation loss is calculated based on the training samples of , because when there is a significant distribution difference between the new wiring diagram data and the historical wiring diagram, the original model may make mistakes in analyzing the new data, which will mislead the supervision of the distillation process.

[0130] Finally, the prediction loss and the distillation loss are added together to obtain the loss function of the new model knowledge distillation process, as shown below:

[0131]

[0132] Wherein, μ is the distillation loss proportional coefficient.

[0133] In the incremental learning process, the constructed feature extraction two-stream network based on text region mask generation (such as Fig.10The proposed method (as shown in the figure) includes a feature extraction backbone network based on Swin Transformer and FPN and a lightweight text region mask generation branch based on improved MobileNetv3. In the process of shared feature extraction, the feature extraction branch of the entire image remains unchanged, and the feature extraction backbone network based on Swin Transformer and FPN is still used to obtain feature maps of four different scales, features. In order to improve the accuracy and generalization ability of the text detection stage under different types of wiring diagrams, a lightweight text region mask generation branch based on improved MobileNetv3 is introduced on the basis of the original backbone network. The input image is fitted through the network layer to obtain a rough text region mask in the input entire drawing, which is then multiplied by the features to obtain the final feature map for the text detection branch to predict and output the detection result. In order to ensure that the text region mask fully covers the text, a weighted cross entropy loss function is designed to focus on the recall rate while supervising the overall image segmentation accuracy.

[0134] Specifically, in the feature extraction stage, the input is the original drawing image, which is fed into a convolution layer with a convolution kernel of 3×3 and a step size of 2 to extract features. Subsequently, it is sequentially fed into 11 depth-separable convolution blocks to extract depth feature information, and the F 1 ,F 2 ,F 3 ,F 4 Four scale feature maps, among which the depth-separable convolution block is the inverted residual structure of MobileNetv3. In the feature fusion stage, based on the deconvolution and upsampling strategy, the feature map F 1 ,F 2 ,F 3 ,F 4 The initial text region mask Mask is obtained by fusion, and further dimensionality reduction is performed through a linear layer to obtain a text region mask M with a scale of 1 / 4 of the length and width of the original image. 1 Finally, in the text region mask generation stage, M 1 Gradually downsample to obtain three more scale-reduced text region masks M 2 ,M 3 ,M 4 , perform a dot multiplication operation on the text area mask and features to obtain the feature map F based on the text area mask S .

[0135] After the feature extraction and fusion of the above network structure, the four text region masks M of different scales are obtained. 1 ,M 2 ,M 3 ,M 4Indicates the rough area where text exists in the drawing image. Then, the text detection stage can fit the text candidate box based on the features of these areas, which requires the text area mask to completely cover all possible text areas. To achieve this goal, a weighted cross entropy loss function is designed in the text area mask generation branch, as follows:

[0136]

[0137] Among them, f(i,j) represents the text area mask M 1 The predicted value at pixel (i, j), Represents the text area mask M 1 The true value label at the pixel point (i, j), that is, whether the pixel point is contained in a certain true value text box. A is the positive example weight hyperparameter. The hyperparameter greater than 1 indicates that the current supervision pays more attention to the positive example pixels, that is, while ensuring the overall prediction accuracy of the text area, more attention is paid to the recall rate of the text area, so that the predicted rough text area can achieve full coverage of the true value.

[0138] Step 8: Input the test drawing into the model designed and trained in steps 1-7 to obtain the detection and recognition results.

[0139] Specifically, the input drawing is first sent to the feature extraction dual-stream network for feature extraction, and then sent to the text detection stage to obtain the text detection result and text feature map. Then the text feature map is sent to the visual feature extraction and prediction module to obtain the feature sequence, and matched with the canonical representation obtained by pre-training the character structure feature extraction prediction module to obtain the text recognition result. The text detection result and the text recognition result are integrated to obtain the final test prediction result.

[0140] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

[0141] [1] Zhang Dongdong, Zhao Yuqian. An automatic detection and recognition method for plant wiring diagram text based on improved PP-OCRv3[P]. 202311000952.9.2023-12-01.

Claims

1. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2, characterized in that: The steps include: Step 1: Send the input image to the text detection and recognition network based on multimodal learning for training and prediction, extract the shared feature backbone network, obtain the shared feature map, and further send it to the text detection module to obtain the text detection result and text feature map; Step 2: Send the text feature map to the visual feature extraction and prediction module to obtain a feature sequence, and then match the predicted feature sequence with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result; Step 3: Send the text recognition results to the fine-tuning post-processing module, fine-tune some recognition results based on the prior knowledge of the power grid, and obtain the final text recognition results; Step 4: Calculate the detection and recognition losses by combining the text detection results and text recognition results with the corresponding true values; Step 5: Jointly optimize the entire network model based on the loss; Step 6: Divide the general scenario text dataset and the power plant wiring diagram dataset into a training set and a test set respectively. Use the training set screening strategy based on data mining to screen the general scenario training set for model pre-training. The training set for model fine-tuning training is the power plant wiring diagram training set. Both pre-training and fine-tuning training are cycled through steps 1-5 until the network converges, and the model file is saved. Step 7: Use the new plant wiring diagram dataset to build a two-stream network for feature extraction based on text region mask generation, and use a knowledge distillation-based incremental learning strategy for multi-category wiring diagram text detection and recognition to perform incremental learning on the current model; Step 8: Input the test drawing into the model designed and trained in steps 1-7 to obtain the detection and recognition results.

2. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 1, characterized in that: In step 2: The visual feature extraction and prediction module is a visual feature extraction and prediction module based on multi-level attention. It extracts receptive field features of different scales through local attention and global attention branches. The introduction of hole convolution can further capture the relationship between long-distance pixels and improve the fitting ability of neighborhood and global features. With the help of the characteristics of joint optimization, the detection branch is optimized by recognition loss to achieve the purpose of correcting the text detection and segmentation results. In the text recognition codec, proportional upsampling is added to reduce the loss of small character features in the serialization process and improve the recall rate of small characters. The character structure feature extraction and prediction module is a character structure feature extraction and prediction module based on CCR-CLIP, which constructs an image encoder and a character encoder and pre-trains the module based on contrast loss, introduces character structure features into the visual model, and improves the model's recognition ability for Chinese characters from the perspective of multimodal learning; In order to improve the fitting effect of multimodal prediction information, a multimodal prediction fusion module based on contrastive learning is designed. The prediction of the visual branch and the character specification representation of the character structure feature branch are multiplied after passing through two linear layers. Through a series of convolutional structures and linear layers, the text recognition accuracy of complex models in complex scenes is improved.

3. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 2, characterized in that: The multi-level attention-based visual feature extraction and prediction module includes three sub-modules: a multi-level attention branch, a recognition transformation, and a text recognition codec based on proportional upsampling; Among them, the multi-level attention branch includes local attention and global attention. The local attention branch adopts the residual hole convolution structure and the window self-attention mechanism, and mainly focuses on the local detail features of the image; the hole convolution can build a multi-scale receptive field for the model in each convolution layer, which is beneficial for the model to extract texture features of different granularities. The residual structure can learn and save more subtle spatial information in the process of back propagation. Based on this, the residual hole convolution structure enables the features of irregular text areas to be extracted more accurately, and the irregular characters inside the text can also be given more reasonable and accurate weights in the subsequent attention mechanism; specifically, the RoI feature map Input the residual hole convolution structure as follows: X1=ReLU(DialatedResLayer(F)) X=DialatedResLayer(X1)+Conv 1×1 (F) Among them, DilatedResLayer(·) consists of a dilated convolution layer with a convolution kernel size of 5×5 and a dilation rate of 2 and a BatchNorm layer; next, the fitted features are fed into the window self-attention to capture local dependencies, as shown in the following formula: Q=XW q ,K=XW k, K=XW v in, They represent the query, key, and value matrices in the self-attention mechanism, d is the dimension of the key / value matrix, and M 2 is the number of sampling points in a window; is the relative position deviation. The relative position encoding of the self-attention weight matrix is ​​realized by introducing B, so as to construct the local dependency relationship within the image window. Finally, the obtained window self-attention fitting result is passed through the feedforward neural network MLP to obtain the nonlinear transformation of the feature, so that the network can capture higher-order image features. The global attention branch adopts the dilated convolution and multi-head self-attention mechanism, mainly focusing on the global contour features of the image. The dilated convolution is used to improve the convolution receptive field, and the feature map of the fitting neighborhood spatial information is obtained. The feature map is used as the query matrix, and the feature map after global average pooling is used as the key and value matrix. The global dependency is captured through multi-head self-attention, as shown in the following formula: X'=ReLU(DialatedResLayer(F)) Q′=XW′ q ,K′=AP(X′)W′ k ,V′=AP(X′)W′ v F global =MLP(MultiHeadSelfAttn(Q′,K′,V′)) Among them, AP(·) is the global average pooling layer, MultiHeadSelfAttn(·) is the multi-head self-attention mechanism, and the number of heads is 8. Through the above structure, each position of the input feature image F processed by the dilated convolution interacts with the global features (i.e., K and V after pooling) to form a self-attention output. This mechanism can be understood as comparing each position with the global information of the entire image to extract long-distance dependencies and global contextual relationships. Through the multi-head setting, this structure can capture rich contextual information from different subspaces, improving the feature expression ability and global feature fitting effect of the network in complex scenes. The text recognition codec based on proportional upsampling generally follows the architecture of SwinTextSpotter v2. Based on the original SwinTextSpotter v2 text recognition codec architecture, a bilinear interpolation upsampling layer is introduced in the sequence encoding to convert the text feature map Shengweicheng Make the image more detailed and the pixels smoother, so that more high-frequency information can be retained during downsampling.

4. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 2, characterized in that: The character structure feature extraction and prediction module based on CCR-CLIP is composed of an image encoder and a character encoder. It extracts the structural stroke features of Chinese characters to construct their standard representation, and obtains the recognition result of Chinese characters by using the idea of ​​contrastive learning. The module obtains the character standard representation P through separate pre-training, and performs multimodal fusion with the prediction of the visual branch as the prediction of the character feature branch to obtain the final prediction result. The training set of the module is the printed pictures of all characters and the radical stroke sequences of the characters. The image encoder is responsible for extracting the visual features of the input character image, while the text encoder extracts the features of the corresponding radical sequence. Finally, two contrast losses are used to supervise the model. Specifically, the image encoder uses Resnet-50 as the backbone network to obtain image features Then after global average pooling, we get f c Embedded in the visual feature space, as follows: I=f c W c in, is the projection matrix, C′ is the alignment dimension; The text encoder consists of 2 layers of Transformer encoders and an embedding layer; through the encoder, the radical sequence R is encoded into sequence features Corresponding to the output features of each time step, l is the length of the radical sequence, is considered as all the features of R; finally, it is embedded into the text feature space to obtain T, as shown below: in, is the projection matrix; in order to match the image and character structure features, the contrast loss between the image and character structure is designed Make the loss between the corresponding image and character structure as small as possible, and the loss between the non-corresponding image and character structure as large as possible, as shown in the following formula: In order to reduce the prediction error caused by different font styles and similar characters, a contrast loss between the visual features of input images with the same radical stroke labels is introduced. Make the feature loss between similar characters as large as possible, as shown in the following formula: in, is a set of visual features I with the same radical stroke sequence; thus, the final loss function of the CCR-CLIP model is obtained as follows: From this point on, the CCR-CLIP model can be trained using printed Chinese character images and a text encoder can be used to generate normalized representations of all candidate Chinese characters.

5. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 2, characterized in that: The multimodal prediction fusion module based on contrastive learning combines the prediction of the visual branch The character specification representation P of the character structure feature branch is multiplied by two linear layers, and then further predicted and fused through a set of depth-wise separable convolutions. Finally, the final text recognition result y is obtained through Softmax, as shown in the following formula: F′=Linear(ReLU(Linear(F O ))) P′=Linear(ReLU(Linear(P))) y=Softmax(DSConv(F′P′ T )) Among them, the dimensions of F′ and P′ are the same as F o The same dimension as P.

6. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 1, characterized in that: In step 3: The prior knowledge includes the naming rules of electrical components and the writing rules of identification; The fine-tuning post-processing strategies based on prior knowledge include: regular expression-based adjacent text segmentation post-processing and merge-reconstruction-based text segmentation inconsistency and missed detection post-processing; Specifically, the adjacent text segmentation post-processing based on regular expressions is used to solve the problem that the text positions are too close and thus are detected as a whole. The text segmentation inconsistency and missed detection post-processing based on merge reconstruction is used to solve the problem that the text logo as a whole is recognized as two parts due to the long distance and the "I" character is missed. The text detection and recognition results obtained from the end-to-end text detection and recognition network are sent to this module, and the post-processing operations of the two branches can be completed in parallel according to the screening conditions. The specific steps for post-processing of adjacent text segmentation based on regular expressions are as follows: S3.1.1 Length threshold screening: After text detection and recognition are completed, based on the "text aspect ratio" screening strategy, all text prediction boxes with aspect ratios greater than the set threshold γ are extracted; this process can effectively screen out long and thin text boxes that may cause segmentation problems; S3.1.2 Regular expression matching: Use regular expressions to split long texts and ensure that the split text boxes conform to the set format; S3.1.3 Text box cutting: According to the results of regular expression matching and segmentation, the text frame is cut with the help of the relative position of the text content to obtain several processed new text frames; The specific methods for post-processing inconsistent text segmentation and missed detection based on merge reconstruction are as follows: S3.2.1 Keyword screening: Filter the text boxes containing the keywords "P", "Q", "temperature", and "gear". Subsequent operations are based on these text boxes; S3.2.2 Extended Merger: For the filtered text boxes, if there are and only the above keywords, extend them to the right by a certain pixel range. If there are text boxes with digital text content within the range, merge them with the original text boxes; S3.2.3 Traversal reconstruction: To solve the problem of missing the "I" character, for all text boxes containing the above keywords, a certain pixel range is traversed downward. If there is a text box with digital text content in a certain area below, the digital text box is extended to the left by a certain distance (the distance is determined by the position of the text box containing the keyword above it) and reconstructed into a new text box.

7. A robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 1, characterized in that: In step 6: When training the network model constructed in steps 1-5, the strategy of pre-training first and then fine-tuning is adopted, that is, the model is first pre-trained in a general scenario to enable the model to have basic text detection and recognition capabilities, and then the model is fine-tuned using the power grid plant wiring diagram dataset; The training set screening strategy based on data mining in the pre-training stage is to first use the original general scene data set to roughly train the network model constructed in steps 1-5 to obtain a low-precision model, and then use the low-precision model to filter the training set data with high confidence. Then, the model based on the improved PP-OCRv3 is used as a high-precision model to filter the low-confidence data. After two data filterings, a high-quality training set is obtained. Finally, the low-precision model is finely trained and iterated using the high-quality training set; the details are as follows: (1) High-confidence data filtering based on low-precision models First, the proposed model is roughly trained on part of the training data to obtain a low-precision model that can quickly predict large-scale data; Using this low-precision model, we perform text detection and text recognition on a general scene text dataset of tens of millions of words, and screen out text prediction boxes with a confidence level greater than 0.95, which are considered redundant text data. For the training images where these redundant texts are located, if there is only this redundant text in the training image, the entire sample is deleted from the training set; if there is other text in this training image in addition to this redundant text, the training image is retained in the training set. (2) Low confidence data filtering based on high-precision models After filtering out high-confidence redundant text data through the high-confidence data filtering step, in order to further filter out low-confidence training data, the algorithm model based on the improved PP-OCRv3 is used as a high-precision model to predict the remaining training data. For text prediction boxes with a confidence less than 0.15, they are classified as negative samples with poor quality or difficult to identify, which may interfere with model training; for these negative samples, their true value identifiers are directly deleted in the training set identifier file. If the training image where the negative text is located only has this text, the entire image is deleted from the training set; Through the above steps, high-confidence redundant data and low-confidence negative data can be filtered out from the general scenario training set, and the remaining training data are all high-quality training data that have a positive impact on the next stage of pre-training of the model; (3) Detailed training and iteration The screened high-quality training data is used to perform fine training on the network model constructed in steps 1-5 to obtain a higher-precision network model; and used for subsequent fine-tuning training based on the plant wiring diagram drawings.

8. The robust generalized detection and recognition method for plant wiring diagram text based on improved SwinTextSpotter v2 as claimed in claim 1, characterized in that: In step 7: The incremental learning strategy for text detection and recognition of multi-category wiring diagrams based on knowledge distillation uses the model obtained through initial training as the original model. On this basis, all parameters of the text content recognition stage (step 2) are frozen into the new model, and only the model parameters involved in the feature extraction and text detection stages (step 1) are incrementally learned. The specific training strategies are as follows: Assume that before time t, the original model has been trained based on the historical wiring diagram Data old At time t, a new set of wiring diagrams to be trained appears. old Randomly retain some images to construct some historical wiring diagram datasets old-part , together with the new wiring diagram training set Data new The training set for this knowledge distillation is formed. In addition, all parameters of the text content recognition stage of the original model are frozen into the new model. In the process of knowledge distillation of the new model, assuming batch size batchsize = B, then in Data old-part and Data new Sample B / 2 samples for training respectively, and send the input to the feature extraction two-stream network based on text area mask generation to extract image features and generate rough text area masks respectively, and send the masked feature map to the subsequent module to predict text detection and recognition results; the detection and recognition results are compared with the true value label to calculate the prediction loss The prediction loss includes the original model's original loss And the weighted cross entropy loss of the text region mask generation branch As follows: Among them, λ is the mask branch ratio coefficient, is the L1 loss of the text candidate box bbox in the text detection stage, is the cross entropy loss of the text segmentation result in the text detection stage, is the loss of the text recognition stage; In order to realize the supervision of the original model on the training of the new model, Data old-part The training samples are fed into the original model for prediction and the soft labels of the bbox and text prediction sequence are output. The soft labels are used as prior knowledge and the prediction results of the new model to calculate the distillation loss. As follows: Among them, P d ,Q d Represent the prediction vectors of the original model and the new model bbox, P r ,Q r Represent the vectors of the original model and the new model text prediction sequence respectively; p d (i),q d (i) represents the bbox prediction value of the original model and the new model at the i-th point, p r (i),q r (i) represents the predicted value of the text prediction sequence of the original model and the new model at the i-th point; N and M represent the number of bboxes and the number of text prediction sequences predicted in the current round, respectively; Finally, the prediction loss and the distillation loss are added together to obtain the loss function of the new model knowledge distillation process, as shown below: Wherein, μ is the distillation loss proportional coefficient; In the incremental learning process, the constructed feature extraction dual-stream network based on text region mask generation includes a feature extraction backbone network based on Swin Transformer and FPN and a lightweight text region mask generation branch based on improved MobileNetv3. In the shared feature extraction process, the feature extraction branch of the entire image remains unchanged, and the feature extraction backbone network based on Swin Transformer and FPN is still used to obtain feature maps of four different scales. In order to improve the accuracy and generalization ability of the text detection stage under different types of wiring diagrams, a lightweight text region mask generation branch based on improved MobileNetv3 is introduced on the basis of the original backbone network. The input image is fitted through the network layer to obtain a rough text region mask in the input entire drawing, which is then multiplied by the features to obtain the final feature map for the text detection branch to predict and output the detection result. In order to ensure the full coverage of the text region mask for the text, a weighted cross entropy loss function is designed to focus on the recall rate while supervising the overall image segmentation accuracy. Specifically, in the feature extraction stage, the input Input is the original drawing image, which is fed into a convolution layer with a convolution kernel of 3×3 and a step size of 2 to extract features. Subsequently, 11 depth-separable convolution blocks are sequentially sent to extract deep feature information to obtain feature maps of four scales F1, F2, F3, and F4, among which the depth-separable convolution block is the inverted residual structure of MobileNetv3; in the feature fusion stage, based on the deconvolution and upsampling strategy, the feature maps F1, F2, F3, and F4 are fused to obtain a preliminary text area mask Mask, and further reduced in dimension through a linear layer to obtain a text area mask M1 with a scale of 1 / 4 of the length and width of the original image; finally, in the text area mask generation stage, M1 is gradually downsampled to obtain another three text area masks M2, M3, and M4 with decreasing scales, and the text area mask is multiplied with the features to obtain a feature map F based on the text area mask. S ; After feature extraction and fusion of the above network structure, the obtained four text region masks M1, M2, M3, and M4 of different scales represent the rough areas where text exists in the drawing image. Then, in the text detection stage, the text candidate boxes can be fitted based on the features of these areas. This requires that the text region mask should completely cover all possible text regions. To achieve this goal, a weighted cross entropy loss function is designed in the text region mask generation branch, as shown in the following formula: Among them, f(i,j) represents the predicted value of the text area mask M1 at the pixel point (i,j), represents the true value label of the text area mask M1 at the pixel point (i, j), that is, whether the pixel point is contained in a certain true value text box; A is the positive example weight hyperparameter. If the hyperparameter is greater than 1, it means that the current supervision pays more attention to the positive example pixels, that is, while ensuring the overall prediction accuracy of the text area, more attention is paid to the recall rate of the text area, so that the predicted rough text area can achieve full coverage of the true value.

Images