A substation wiring diagram text robust generalization detection and recognition method based on improved SwinTextSpotter v2

Abstract

The application belongs to the field of smart grid and computer vision, and particularly relates to a power station wiring diagram text robust generalization detection and recognition method based on an improved SwinTextSpotter v2. The method comprises the following steps: step 1: inputting an image into a text detection and recognition network based on multi-modal learning for training and prediction, obtaining a shared feature map through a shared feature extraction backbone network, and further inputting the shared feature map into a text detection module to obtain a text detection result and a text feature map; step 2: inputting the text feature map into a visual feature extraction and prediction module to obtain a feature sequence, and then matching the predicted feature sequence with a canonical representation obtained by a character structure feature extraction and prediction module to obtain a recognition result; and the like. The application robustly improves the detection and recognition accuracy of the model for irregular text and Chinese character text, and improves the generalization performance of the text detection and recognition of various types of wiring diagrams.

Description

Technical fields:

[0001] This invention belongs to the fields of smart grid and computer vision, and specifically relates to a robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2. Background technology:

[0002] With the increasing scale of power grid drawings, traditional manual identification of drawing content is not only inefficient but also prone to errors. Automation of power grid drawing management and retrieval is therefore urgently needed. Due to the complexity of information in power grid wiring diagrams, the detection of text annotation locations and the recognition of text content in power grid drawings become particularly important for achieving automation. Text annotations in power grid substation wiring diagrams often take many forms, with different shapes, orientations, and sizes. The characters include not only Arabic numerals and letters but also many Chinese characters. Furthermore, the size of characters within the text in substation wiring diagrams often varies, and they contain a large number of easily confused characters with similar shapes that are not based on natural semantics.

[0003] In recent years, deep learning-based methods have achieved remarkable results in the field of optical character recognition. Deep learning-based OCR technology has become a potential pathway for the automation and intelligentization of power grid wiring diagrams. Applying deep learning to text recognition in engineering design drawings can effectively achieve the intelligent and automatic recognition of power grid wiring diagrams.

[0004] Closest to existing technology and its evaluation:

[0005] Li et al (SHANBIN L, HAOYU W, JUNHAO Z. Electrical cabinet wiring detection method based on improved YOLOv5 and PP-OCRv3; proceedings of the2022ChineseAutomation Congress,CAC 2022,November 25,2022-November 27,2022,Xiamen,China,F,2022[C].Institute of Electrical and Electronics Engineers Inc. improved the accuracy of text detection and recognition in electrical cabinet wiring diagrams using YOLOv5 and PP-OCRv3; Wei Wei et al. (Wei Wei, Long Na, Tian Yue, et al. Research on Text Detection Method for Power Equipment Nameplates Based on Improved DBNet [J]. High Voltage Engineering, 2023, 49: 63-67.) used pixel-level interpolation and pooling on the basis of DBNet to more accurately detect the text contours in power equipment nameplates; Liu Wei et al. (Liu Wei. Research on Text Recognition and Wiring Verification System for Secondary Circuit Terminal Blocks [D]. Enshi: Hubei University for Nationalities, 2023.) introduced transfer learning to train an OCR model for the application scenario of terminal block design diagrams. The research results show that although the selected deep learning methods can detect and recognize text in simple scenarios, they still face challenges for more complex text detection and recognition scenarios, such as text located near primitives and text with varying scales and mixed horizontal and vertical characters involved in the plant wiring diagram dataset.

[0006] SwinTextSpotter v2 is an end-to-end text detection and recognition model designed to handle text recognition tasks in complex scenarios. Unlike traditional convolutional neural networks (CNNs), SwinTextSpotter utilizes the powerful feature extraction and modeling capabilities of SwinTransformer

[58] to effectively capture long-distance contextual information and global features, thereby improving the accuracy of text region detection and text content recognition; it uses a query-based detector to achieve iterative optimization, gradually improving text detection accuracy; and it uses a recognition transformation module to connect the detector and the recognizer, eliminating the problem of error accumulation in the two-stage text detection and recognition process. However, SwinTextSpotter v2 performs poorly for Chinese texts without natural semantics and small characters in irregular texts, and it does not have generalization ability for various types and specifications of wiring diagrams. Summary of the Invention

[0007] The purpose of this invention is to provide a robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2.

[0008] This invention addresses the robustness and generalization issues of existing power grid substation wiring diagram text detection and recognition methods. By introducing an end-to-end integrated baseline model to train both text location detection and content recognition tasks together, it significantly reduces error accumulation. Through a series of deep learning-based strategies, it effectively improves the robustness and generalization performance of the model.

[0009] In the preprocessing stage, a training set selection strategy based on data mining is designed to reduce the cost of joint training of end-to-end models while ensuring model accuracy and improving the training efficiency of end-to-end models.

[0010] In the shared feature extraction stage, a lightweight text region mask generation branch based on improved MobileNetv3 is introduced, which effectively solves the problem of poor model generalization performance caused by inconsistent text region distribution among different types and specifications of plant wiring diagrams.

[0011] In the main body of the model, a text detection and recognition network based on multimodal learning is constructed. A visual feature extraction and prediction module based on multi-level attention and a character structure feature extraction and prediction module based on CCR-CLIP are introduced to fit the information of the visual modality and the character structure feature modality. A multimodal prediction fusion module based on contrastive learning is designed to fuse visual features and character structure features, which robustly improves the detection and recognition accuracy of irregular text and Chinese text.

[0012] During the processing phase, a fine-tuning post-processing strategy based on prior knowledge was designed to address the problem of semantic segmentation errors of similar texts in power grid drawings in a targeted manner, according to the specific syntax of power grid drawings. In terms of learning strategy formulation, an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation was designed, which enables the model to effectively retain the memory of old data while gradually adapting to new data, thereby improving the generalization performance of multi-type wiring diagram text detection and recognition.

[0013] Experimental results show that the proposed model can accurately and robustly detect and recognize text of different scales and languages ​​in plant wiring diagrams, and has strong generalization ability for text detection and recognition in wiring diagrams of different types and specifications.

[0014] Technical solution of the present invention:

[0015] A robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 is characterized by the following steps:

[0016] Step 1: The input image is fed into a text detection and recognition network based on multimodal learning for training and prediction. The shared feature map is obtained through the shared feature extraction backbone network and then fed into the text detection module to obtain the text detection result and text feature map.

[0017] Step 2: The text feature map is fed into the visual feature extraction and prediction module to obtain the feature sequence. Then, the predicted feature sequence is matched with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result.

[0018] Step 3: Send the text recognition results to the fine-tuning post-processing module, and fine-tune some of the recognition results based on prior knowledge of the power grid to obtain the final text recognition results;

[0019] Step 4: Calculate the detection and recognition losses by combining the text detection results and text recognition results with the corresponding ground truth values;

[0020] Step 5: Perform joint optimization of the entire network model based on the loss;

[0021] Step 6: Divide the general scenario text dataset and the power grid substation wiring diagram dataset into training set and test set respectively. Use a data mining-based training set selection strategy to select the general scenario training set for model pre-training. The training set for model fine-tuning training is the substation wiring diagram training set. Both pre-training and fine-tuning training are performed in a loop from steps 1 to 5 until the network converges. Save the model file.

[0022] Step 7: Using the new plant wiring diagram dataset, construct a two-stream network for feature extraction based on text region mask generation, and adopt an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation to incrementally learn the current model;

[0023] Step 8: Input the test drawing into the model designed and trained in steps 1-7 to obtain the detection and recognition results.

[0024] By adopting the above solution, the beneficial effects of the present invention are:

[0025] 1. This invention constructs a text detection and recognition network based on multimodal learning, which robustly improves the detection and recognition accuracy of the model for irregular text and Chinese text.

[0026] 2. This invention designs a fine-tuning post-processing strategy based on prior knowledge to specifically solve the problem of semantic segmentation errors of similar texts in plant wiring diagrams.

[0027] 3. This invention designs a training set selection strategy based on data mining, which effectively improves the training and prediction efficiency of the end-to-end text detection and recognition model.

[0028] 4. This invention proposes an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation. By proposing a dual-stream network for feature extraction based on text region mask generation for model sharing feature extraction, it effectively solves the problem of poor model generalization performance caused by inconsistent text region distribution among different types and specifications of plant wiring diagrams, and improves the generalization performance for multi-type wiring diagram text detection and recognition. Attached Figure Description

[0029] Figure 1 This is a basic flowchart of the robust generalization detection and recognition method for plant wiring diagram text based on the improved SwinTextSpotter v2 of this invention;

[0030] Figure 2 This is a diagram of the text detection and recognition network structure based on multimodal learning in step 2 of this invention;

[0031] Figure 3 This is the multi-level attention branch structure diagram mentioned in step 2 of this invention;

[0032] Figure 4 This is a structural diagram of the text recognition codec based on proportional upsampling mentioned in step 2 of this invention;

[0033] Figure 5 This is a structural diagram of the character structure feature extraction and prediction module based on CCR-CLIP mentioned in step 2 of this invention;

[0034] Figure 6 This is a structural diagram of the multimodal prediction fusion module based on contrastive learning mentioned in step 2 of this invention;

[0035] Figure 7 This is the overall framework diagram of the fine-tuning post-processing strategy based on prior knowledge in step 3 of the present invention;

[0036] Figure 8 This is a flowchart of the training set selection strategy based on data mining in step 6 of the present invention;

[0037] Figure 9 This is the overall framework of the incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation in step 7 of the present invention;

[0038] Figure 10 This is the structure diagram of the two-stream network for feature extraction based on text region mask generation mentioned in step 7 of this invention;

[0039] Figure 11 This is a quantitative comparison of the performance of the embodiments of the present invention with other OCR models;

[0040] Figure 12This is a comparison of the generalization performance of the embodiments of the present invention on the old and new plant wiring diagram datasets. Detailed Implementation

[0041] This invention addresses the robustness and generalization issues of existing power grid substation wiring diagram text detection and recognition methods. It proposes a robust and generalized detection and recognition method for substation wiring diagram text based on the improved SwinTextSpotter v2. The basic flowchart of this invention is as follows: Figure 1 As shown.

[0042] To improve the robustness of the model in detecting and recognizing irregularly shaped text, small characters, and Chinese characters not based on natural language semantics, a text detection and recognition network based on multimodal learning was constructed. The overall structure is as follows: Figure 2 As shown, the multi-level attention branch structure diagram is as follows: Figure 3 As shown in the figure, the structure diagram of the text recognition codec based on proportional upsampling is as follows: Figure 4 As shown in the figure, the structure diagram of the character structure feature extraction and prediction module based on CCR-CLIP is as follows: Figure 5 As shown in the diagram, the structure of the multimodal prediction fusion module based on contrastive learning is as follows: Figure 6 As shown.

[0043] To address the issue of semantic segmentation errors in similar text in plant wiring diagrams, a fine-tuning post-processing strategy based on prior knowledge was designed. The overall framework diagram is shown below. Figure 7 As shown.

[0044] To streamline the size of the general training set and improve the efficiency of model pre-training, a training set selection strategy based on data mining was designed. The flowchart of this strategy is as follows: Figure 8 As shown.

[0045] To achieve a balanced learning of new and old knowledge, an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation was developed. The overall framework is as follows: Figure 9 As shown. To address the issue of poor model generalization performance caused by inconsistent text region distribution among different types and specifications of plant wiring diagrams, a two-stream network for feature extraction based on text region mask generation was constructed in the incremental learning model. The structure diagram is shown below. Figure 10 As shown.

[0046] To enable the model to perform text detection and recognition, and to achieve accurate detection and recognition of text in plant wiring diagrams, the model was first pre-trained on a general-purpose text dataset for 800,000 iterations to develop strong generalization capabilities for text detection and recognition. Then, it was fine-tuned on a plant wiring diagram training set for 300,000 iterations to allow the model to more accurately adapt to the specific text structure and layout features in plant wiring diagrams.

[0047] During pre-training, a learning rate decay strategy is adopted, with an initial learning rate of 2.5 × 10⁻⁶. -5 When the number of iterations reaches 400K, the learning rate decays to 2.5×10. -6 When the number of iterations reaches 600K, the learning rate decays to 2.5×10. -7 During fine-tuning training, the learning rate is set to 2.5 × 10⁻⁶. -7 The batch size is the number of samples fed into the model in one iteration during training. Limited by GPU memory resources, the batch size is set to 4 for text detection (meaning 4 complete test images are fed into each iteration), and to 64 for text recognition (meaning 64 text images cropped according to bounding boxes are fed into each iteration). λ in the CCR-CLIP module is the scaling factor of the loss function, used to harmonic... and Set it to 0.80; set the aspect ratio threshold γ in the fine-tuning post-processing module to 6.00.

[0048] The quantitative analysis results of the accuracy of this invention and other OCR models are as follows: Figure 11 As shown, the generalization performance on the old and new plant wiring diagram datasets is compared to, for example... Figure 12 As shown. Experimental results demonstrate that, through a series of improvements to SwinTextSpotter v2, this invention can significantly enhance the robustness and generalization ability of text detection and recognition in plant wiring diagrams.

[0049] The following combination Figure 1 The flowchart shown further illustrates the technical solution of the present invention. The described embodiments are only some embodiments of the present invention, and not all embodiments.

[0050] Step 1: Feed the input image into a text detection and recognition network based on multimodal learning (e.g., ...). Figure 2 (As shown) Training and prediction are performed, and a shared feature map is obtained by extracting the backbone network through shared features. The map is then fed into the text detection module to obtain the text detection result and the text feature map.

[0051] The shared feature extraction backbone network is a feature extraction backbone network based on Swing Transformer and FPN;

[0052] The text detection module adopts the detector structure of SwinTextSpotterv2. Specifically, it generates and optimizes a set of learnable candidate boxes instead of generating a large number of candidate objects in the traditional RPN. Each stage uses a dynamic Transformer encoder to encode the feature map. Later stages can access the information stored in the candidate features in the previous stages. Through iterative optimization across multiple stages, the final predicted text box is output.

[0053] Step 2: The text feature map is fed into the visual feature extraction and prediction module to obtain the feature sequence. Then, the predicted feature sequence is matched with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result.

[0054] To better extract text features and robustly address the issue of missing small characters in mixed horizontal and vertical text, a multi-level attention-based visual feature extraction and prediction module is proposed. Through local and global attention branches, receptive field features at different scales are extracted. The introduction of dilated convolution further captures distant pixel relationships, improving the fitting ability to neighborhood and global features. Leveraging the joint optimization property, the detection branch is optimized through recognition loss to correct the text detection and segmentation results. Proportional upsampling is added to the text recognition codec to reduce the loss of small character features during serialization and improve the recall rate of small characters.

[0055] To improve the recognition accuracy of Chinese characters and address the "long-tail distribution" problem of Chinese characters, a character structure feature extraction and prediction module based on CCR-CLIP is introduced. An image encoder and a character encoder are constructed, and the module is pre-trained based on contrastive loss. Character structure features are introduced into the visual model, improving the model's ability to recognize Chinese characters from the perspective of multimodal learning.

[0056] To improve the fitting effect of multimodal prediction information, a multimodal prediction fusion module based on contrastive learning was designed. The prediction of the visual branch and the character canonical representation of the character structure feature branch are multiplied by two linear layers. A series of convolutional structures and linear layers are used to improve the text recognition accuracy of complex models in complex scenarios.

[0057] The visual feature extraction and prediction module based on multi-level attention includes multi-level attention branches (such as...). Figure 3 (as shown), recognition conversion and text recognition codec based on proportional upsampling (e.g.) Figure 4 (As shown) Three sub-modules. Among them, the recognition and conversion module uses the original structure of the SwinTextSpotter v2 network to generate a tight mask for text regions, realizing joint optimization of text detection and recognition in two stages.

[0058] The multi-level attention branch includes local attention and global attention. The local attention branch employs a residual dilated convolution structure and a window self-attention mechanism, primarily focusing on local image details. Dilated convolution allows the model to construct multi-scale receptive fields in each convolutional layer, facilitating the extraction of texture features at different granularities. The residual structure learns and preserves more subtle spatial information during backpropagation. Based on this, the residual dilated convolution structure enables more accurate extraction of features from irregular text regions, and irregular characters within the text can be assigned more reasonable and accurate weights in subsequent attention mechanisms. Specifically, the RoI feature map... The input residual dilated convolution structure is fed in as follows:

[0059] X1=ReLU(DialatedResLayer(F))

[0060] X = DialatedResLayer(X1) + Conv 1×1 (F)

[0061] The DilatedResLayer(·) consists of a dilated convolutional layer with a kernel size of 5×5 and a dilation rate of 2, and a BatchNorm layer. Next, the fitted features are fed into the window self-attention to capture local dependencies, as shown in the following equation:

[0062] Q = XW q K = XW k, K = XW v

[0063]

[0064] in, Let M represent the query, key, and value matrices in the self-attention mechanism, respectively, where d is the dimension of the key / value matrix, and M is the value of M. 2 It is the number of sampling points in a window. The relative positional deviation is achieved by introducing B to encode the relative position of the self-attention weight matrix, thereby constructing local dependencies within the image window. Finally, the obtained window self-attention fitting result is passed through a feedforward neural network (MLP) to obtain a nonlinear transformation of the features, enabling the network to capture higher-order image features. The global attention branch employs dilated convolution and multi-head self-attention mechanisms, primarily focusing on the global contour features of the image. Dilated convolution is used to increase the receptive field of the convolution, obtaining a feature map that fits the neighborhood spatial information. This feature map is used as the query matrix, and the feature map after global average pooling is used as the key-value matrix. Global dependencies are captured through multi-head self-attention, as shown in the following equation:

[0065] X'=ReLU(DialatedResLayer(F))

[0066] Q′=XW q K′ = AP(X′)W k V′ = AP(X′)W v ′

[0067] F global =MLP(MultiHeadSelfAttn(Q′,K′,V′))

[0068] In this structure, AP(·) is a global average pooling layer, and MultiHeadSelfAttn(·) is a multi-head self-attention mechanism with 8 heads. Through this structure, each location of the input feature image F, processed by dilated convolution, interacts with global features (i.e., the pooled K and V) to form a self-attention output. This mechanism can be understood as comparing each location with global information of the entire image to extract long-distance dependencies and global contextual relationships. By using a multi-head setup, this structure can capture rich contextual information from different subspaces, improving the network's feature representation ability and global feature fitting effect in complex scenes.

[0069] The text recognition codec based on proportional upsampling generally follows the architecture of SwinTextSpotter v2. However, during experiments, a common problem found in such RNN / Transformer-based text recognizers is that they all require text features to be fed into the recognizer in a "width > height" state, meaning that vertical text is rotated and flattened. Furthermore, before text prediction, the input two-dimensional feature map is downsampled column-wise into a one-dimensional feature sequence. This leads to the loss of high-frequency features and details; that is, information in side-by-side numbers in these "flattened" mixed horizontal and vertical texts is easily lost during downsampling, and the proportion of information loss increases with the number of side-by-side numbers. This also explains why three side-by-side numbers are more likely to be misidentified. To address this issue while maintaining the optimized model size, a simple bilinear interpolation upsampling layer is introduced into the sequence encoding of the original SwinTextSpotterv2 text recognition codec architecture to upsample the text feature map... (Where B′ is the batch size and C is the number of feature channels) Dimensional Upgrading This makes the image more detailed in terms of height and the pixels smoother, so that more high-frequency information can be preserved during downsampling.

[0070] Character structure feature extraction and prediction module based on CCR-CLIP (e.g.) Figure 5The module (shown) primarily constructs standardized representations of Chinese characters by extracting structural stroke features. It uses contrastive learning to achieve Chinese character recognition results, and is not limited by the frequency of characters in the training set or the sufficiency of training. This significantly improves the recognition ability for easily confused characters with similar shapes and "zero-sample" characters. It consists of an image encoder and a character encoder. This module obtains a standardized character representation P through separate pre-training. This representation is then used for multimodal fusion with the prediction from the visual branch to obtain the final prediction result. The training set for this module consists of printed images of all characters and the radical stroke sequences of the characters. The image encoder extracts the visual features of the input character images, while the text encoder extracts the features of the corresponding radical sequences. Finally, two contrastive losses are used to supervise the model. Specifically, the image encoder uses ResNet-50 as the backbone network to obtain image features. Then, after global average pooling, we get... f c Embedded in the visual feature space, as shown in the following equation:

[0071] I = f c W c

[0072] in, Let C' be the projection matrix and C′ be the alignment dimension. The text encoder consists of two Transformer encoder layers and one embedding layer. Through the encoder, the radical sequence R is encoded into sequence features. For each time step, the output feature is represented by l, which is the length of the radical sequence. Considered to be all the features of R, D is f l r The number of feature channels. Finally, embedding it into the text feature space yields T, as shown in the following equation:

[0073] T = f l r W r

[0074] in, It is a projection matrix. To match the image and character structural features, a contrast loss is designed between the image and the character structure. The goal is to minimize the loss between corresponding images and character structures, and maximize the loss between non-corresponding images and character structures, as shown in the following formula:

[0075]

[0076] Where N is the total number of characters in the dictionary. To reduce prediction errors caused by different font styles and similar characters, a contrast loss is introduced between visual features of input images with the same radical stroke label. To maximize the feature loss between similar-looking characters, as shown in the following formula:

[0077]

[0078] in, Let I be the set of visual features that share the same radical stroke sequence; from this, we obtain the final loss function of the CCR-CLIP model, as shown in the following equation:

[0079]

[0080] From this point on, the CCR-CLIP model can be trained using printed images of Chinese characters, and a text encoder can be used to generate standardized representations of all candidate Chinese characters.

[0081] The designed multimodal prediction fusion module based on contrastive learning (such as...) Figure 6 (As shown) Prediction of the visual branch The character canonical representation P of the character structure feature branch is multiplied by two linear layers, then further predicted and fused through a set of depthwise separable convolutions, and finally the final text recognition result y is obtained through Softmax, as shown in the following formula:

[0082] F′=Linear(ReLU(Linear(F O )))

[0083] P′=Linear(ReLU(Linear(P)))

[0084] y = Softmax(DSConv(F′P′) T ))

[0085] Wherein, the dimensions of F′ and P′ are the same as those of F. o It has the same dimensions as P.

[0086] Step 3: Send the text recognition results to the fine-tuning post-processing module, and fine-tune some of the recognition results based on prior knowledge of the power grid to obtain the final text recognition results.

[0087] The prior knowledge includes naming rules for electrical components, rules for writing labels, etc.

[0088] Fine-tuning post-processing strategies based on prior knowledge (such as...) Figure 7 (As shown) It includes post-processing of adjacent text segmentation based on regular expressions and post-processing of inconsistent and missed detections in text segmentation based on merging and reconstruction, which can specifically improve the text detection and recognition performance for the specific scenario of plant wiring diagrams.

[0089] Specifically, the regular expression-based adjacent text segmentation post-processing addresses the issue of texts being detected as a single unit due to their proximity. The merged reconstruction-based text segmentation inconsistency and missed detection post-processing addresses the problem of text identifiers being identified as two parts due to their large distance, as well as the missed detection of the "I" character. The text detection and recognition results obtained from the end-to-end text detection and recognition network are fed into this module, which can perform the post-processing operations of the two branches in parallel according to the filtering conditions. The post-processing strategies for the two branches are described in detail below.

[0090] The specific steps for post-processing adjacent text segmentation based on regular expressions are as follows:

[0091] S3.1.1 Length threshold filtering:

[0092] After text detection and recognition are completed, based on the "text aspect ratio" filtering strategy, all text prediction boxes with an aspect ratio greater than a set threshold γ are extracted. This process can effectively filter out long and thin text boxes that may cause segmentation problems.

[0093] S3.1.2 Regular Expression Matching:

[0094] Long text is segmented using regular expressions to ensure that the segmented text boxes conform to a set format. Specifically, fixed structures such as "35KV", "110KV", "backup + number", and "Chinese character + number" are defined, and regular expressions are used for pattern matching and splitting.

[0095] S3.1.3 Text box segmentation:

[0096] Based on the results of regular expression matching and segmentation, and by using the relative positions of the text content, the text boxes are cut into several new text boxes after processing.

[0097] The specific steps for handling inconsistencies and missed detections in text segmentation based on merging and reconstruction are as follows:

[0098] S3.2.1 Keyword Filtering:

[0099] Filter the text boxes containing the keywords "P", "Q", "temperature", and "gear". All subsequent operations will be based on these text boxes.

[0100] S3.2.2 Extended Merging:

[0101] For the selected text boxes, if there are only the above keywords, extend them to the right by a certain number of pixels. If there are text boxes with numeric text within the range, merge them with the original text boxes.

[0102] S3.2.3 Traversal Restructuring:

[0103] To address the issue of missing the "I" character, for all text boxes containing the aforementioned keywords, a certain pixel range is traversed downwards. If a text box with numeric content is found within a certain area below, that numeric text box is extended to the left by a certain distance (the distance is determined by the position of the text box containing the keyword above it) and reconstructed into a new text box.

[0104] Step 4: Calculate the detection and recognition losses by combining the text detection results and text recognition results with the corresponding ground truth values.

[0105] Step 5: Perform joint optimization of the entire network model based on the loss.

[0106] Step 6: Train the model through two stages: pre-training and fine-tuning training. Repeat steps 1-5 until the network converges, and then save the model file.

[0107] When training the algorithm model of this invention, a strategy of pre-training and then fine-tuning is adopted. That is, the model is first pre-trained in a general scenario using a general scenario text dataset to enable the model to have basic text detection and recognition capabilities; and then the model is fine-tuned using a power grid substation wiring diagram dataset.

[0108] As an example, the general scene text datasets are ICDAR2019-LSVT, ICDAR2017-RCTW, ICDAR2015, and Chinese_Dataset. The general scene training sets are selected using a data mining-based training set selection strategy for model pre-training.

[0109] As an example, the power grid substation wiring diagram dataset consists of 113 drawings and 23,377 text images from State Grid Shanghai Electric Power Company's 10KV switching stations, 35KV substations, 110KV substations, and ultra-high voltage substations.

[0110] Furthermore, the pre-training process employs a data mining-based training set selection strategy. First, the original general scenario dataset is used to coarsely train the network model constructed in steps 1-5 to obtain a low-precision model. This low-precision model is then used to perform high-confidence data filtering on the training set data. Next, the model based on the improved PP-OCRv3 (refer to [1]) is used as the high-precision model and subjected to low-confidence data filtering. After two data filtering processes, a high-quality training set is obtained. Finally, this high-quality training set is used to perform fine training and iteration on the low-precision model. Specifically, as follows: (e.g.) Figure 8 )

[0111] (1) High-confidence data filtering based on low-precision models

[0112] First, using a portion of the training data, the network model constructed in steps 1-5 is coarsely trained (400K iterations) to obtain a low-precision model. Although this model has limited accuracy, it can quickly predict large-scale data. Using this low-precision model, text detection and recognition are performed on a general-purpose text dataset with tens of millions of records. Text prediction boxes with a confidence score greater than 0.95 are selected; these are considered redundant text data because they contribute little to improving the model's accuracy. For training images containing these redundant texts, if the training image contains only this one redundant text, the entire sample is removed from the training set; if the training image contains other text besides this redundant text, the training image is retained in the training set.

[0113] (2) Low-confidence data filtering based on high-precision model

[0114] After filtering out redundant text data with high confidence through a high-confidence data filtering step, a high-precision model is introduced to predict the remaining training data in order to further filter out low-confidence training data. For text prediction boxes with a confidence level less than 0.15, they are classified as negative samples that are of poor quality or difficult to identify and may interfere with model training. For these negative samples, their ground truth labels are directly deleted from the training set label file. If the training image containing a negative sample contains only this one text, the entire image is deleted from the training set.

[0115] Specifically, the high-precision model uses an algorithm model based on the improved PP-OCRv3 (see [1]), which does not require training and can be used directly for prediction.

[0116] By following the steps above, redundant data with high confidence and negative data with low confidence can be filtered out from the general scenario training set. The remaining training data are high-quality training data that have a positive impact on the pre-training of the model in the next stage.

[0117] (3) Fine-tuning and iteration

[0118] The selected high-quality training data is used to refine the network model constructed in steps 1-5, and the process is repeated 400K times to obtain a more accurate network model.

[0119] Compared to using the original training set for pre-training, the introduction of a high-quality training set not only improves the efficiency of pre-training but also effectively enhances the accuracy of the model, giving it stronger text detection and text recognition capabilities in general scenarios. The model obtained through fine-tuning is more accurate than both low-precision and high-precision models and can be directly used for subsequent fine-tuning training based on plant wiring diagrams.

[0120] Step 7: Using the new plant wiring diagram dataset, construct a two-stream network for feature extraction based on text region mask generation, and adopt an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation to incrementally learn the current model.

[0121] Incremental learning strategies for multi-type wiring diagram text detection and recognition based on knowledge distillation (e.g.) Figure 9 As shown, the model obtained from the initial training is used as the original model. Based on this, all parameters of the text content recognition stage (step 2) are frozen into the new model, and only the model parameters involved in the feature extraction and text detection stages (step 1) are learned incrementally.

[0122] The specific training strategy is as follows:

[0123] Assume that before time t, the original model has been trained on the historical wiring diagram data set. old Early training is completed. At time t, a new set of wiring diagrams to be trained appears, and at this time, data is generated from the Data... old A partial historical wiring diagram dataset was constructed by randomly retaining a portion of the images. old-part Along with the new wiring diagram training set Data new This constitutes the training set for this knowledge distillation. Furthermore, all parameters from the original model's text content recognition stage are frozen into the new model.

[0124] In the process of knowledge distillation for the new model, assuming the batch size = B, then in the Data... old-part and Data new B / 2 samples are sampled for training. The input is fed into a dual-stream feature extraction network based on text region masking to extract image features and generate a coarse text region mask. The masked feature map is then fed into subsequent modules to predict text detection and recognition results. The detection and recognition results are compared with the ground truth labels to calculate the prediction loss. Predictive loss includes the original loss of the original model. and the weighted cross-entropy loss of the text region mask generation branch. As shown in the following formula:

[0125]

[0126] Where λ is the mask branching ratio coefficient. For the L1 loss of the text candidate bounding box (bbox) in the text detection stage, The cross-entropy loss is the result of text segmentation during the text detection stage. This represents the loss during the text recognition stage.

[0127] To achieve supervision of the training of the new model by the original model, Data will be used.old-part The training samples are fed into the original model for prediction, and the model outputs soft labels for bounding boxes and text prediction sequences. The soft labels are then used as prior knowledge to calculate the distillation loss along with the prediction results of the new model. As shown in the following formula:

[0128]

[0129] Among them, P d Q d P represents the predicted vectors of the bounding boxes of the original model and the new model, respectively. r Q r ... d (i),q d (i) represents the bounding box predictions of the original model and the new model at the i-th point, p r (i),q r (i) represents the predicted values ​​of the text prediction sequences of the original model and the new model at the i-th point. N and M represent the number of bounding boxes predicted and the number of text prediction sequences in the current round, respectively. In the process of supervising knowledge distillation using the original model, the reason for only processing the Data... old-part The reason for calculating the distillation loss using training samples is that when there is a significant distribution difference between the newly added wiring diagram data and the historical wiring diagram, the original model may make errors in its analysis of the new data, which in turn may mislead the supervision of the distillation process.

[0130] Finally, the prediction loss and distillation loss are added together to obtain the loss function of the new model's knowledge distillation process, as shown in the following equation:

[0131]

[0132] Where μ is the distillation loss ratio coefficient.

[0133] In the incremental learning process, the constructed feature extraction two-stream network based on text region mask generation (such as...) Figure 10The diagram (shown) includes a feature extraction backbone network based on Swing Transformer and FPN, and a lightweight text region mask generation branch based on improved MobileNetv3. During shared feature extraction, the feature extraction branch for the entire image remains unchanged, still using the Swing Transformer and FPN-based feature extraction backbone network to obtain four feature maps at different scales. To improve the accuracy and generalization ability of the text detection stage under different types of wiring diagrams, a lightweight text region mask generation branch based on improved MobileNetv3 is introduced on top of the original backbone network. This branch fits a rough text region mask of the entire input drawing through network layers, and then multiplies it with the features to obtain the final feature map for the text detection branch to predict and output the detection results. To ensure that the text region mask fully covers the text, a weighted cross-entropy loss function is designed, focusing on recall while supervising the overall image segmentation accuracy.

[0134] Specifically, in the feature extraction stage, the input is the original drawing image, which is fed into a 3×3 convolutional layer with a stride of 2 to extract features. Subsequently, it is sequentially fed into 11 depthwise separable convolutional blocks to extract depth features, resulting in feature maps at four scales: F1, F2, F3, and F4. The depthwise separable convolutional blocks are the inverted residual structure of MobileNetv3. In the feature fusion stage, based on deconvolution and upsampling strategies, the feature maps F1, F2, F3, and F4 are fused to obtain a preliminary text region mask (Mask). This is further reduced in dimensionality by a linear layer to obtain a text region mask M1 with a scale equal to 1 / 4 the length and width of the original image. Finally, in the text region mask generation stage, M1 is progressively downsampled to obtain three more text region masks with decreasing scales: M2, M3, and M4. The text region masks are then multiplied by the features to obtain the feature map F based on the text region masks. S .

[0135] After feature extraction and fusion using the aforementioned network structure, the resulting four text region masks M1, M2, M3, and M4 at different scales represent approximate regions containing text in the drawing image. Subsequently, the text detection stage can fit text candidate boxes based on the features of these regions. This requires the text region masks to completely cover all possible text regions. To achieve this goal, a weighted cross-entropy loss function is designed in the text region mask generation branch, as shown in the following equation:

[0136]

[0137] Where f(i,j) represents the predicted value of the text region mask M1 at pixel (i,j). The text region mask M1 represents the truth label at pixel (i,j), indicating whether the pixel is contained within a certain truth text box. A is the positive example weight hyperparameter. A hyperparameter greater than 1 indicates that the current supervision pays more attention to positive example pixels, which means that while ensuring the overall accuracy of the text region prediction, more attention is paid to the recall rate of the text region, so that the predicted coarse text region achieves full truth coverage.

[0138] Step 8: Input the test drawing into the model designed and trained in steps 1-7 to obtain the detection and recognition results.

[0139] Specifically, the input drawing is first fed into a dual-stream feature extraction network for feature extraction, then fed into a text detection stage to obtain text detection results and text feature maps. Next, the text feature maps are fed into a visual feature extraction and prediction module to obtain a feature sequence, which is then matched with the pre-trained canonical representation obtained by the character structure feature extraction and prediction module to obtain the text recognition result. The text detection result and the text recognition result are then integrated to obtain the final test prediction result.

[0140] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

[0141] [1] Zhang Dongdong, Zhao Yuqian. An automatic detection and recognition method for power plant wiring diagram text based on improved PP-OCRv3 [P]. 202311000952.9.2023-12-01.

Claims

1. A robust generalization detection and recognition method for power plant wiring diagram text based on an improved SwinTextSpotter v2, characterized in that, Includes the following steps: Step 1: The input image is fed into a text detection and recognition network based on multimodal learning for training and prediction. The shared feature map is obtained through the shared feature extraction backbone network and then fed into the text detection module to obtain the text detection result and text feature map. Step 2: The text feature map is fed into the visual feature extraction and prediction module to obtain the feature sequence. Then, the predicted feature sequence is matched with the canonical representation obtained by the character structure feature extraction and prediction module to obtain the recognition result. Step 3: Send the text recognition results to the fine-tuning post-processing module, and fine-tune some of the recognition results based on prior knowledge of the power grid to obtain the final text recognition results; Step 4: Calculate the detection and recognition losses by combining the text detection results and text recognition results with the corresponding ground truth values; Step 5: Perform joint optimization of the entire network model based on the loss; Step 6: Divide the general scenario text dataset and the power grid substation wiring diagram dataset into training set and test set respectively. Use a data mining-based training set selection strategy to select the general scenario training set for model pre-training. The training set for model fine-tuning training is the substation wiring diagram training set. Both pre-training and fine-tuning training are performed in a loop from steps 1 to 5 until the network converges. Save the model file. Step 7: Using the new plant wiring diagram dataset, construct a two-stream network for feature extraction based on text region mask generation, and adopt an incremental learning strategy for multi-type wiring diagram text detection and recognition based on knowledge distillation to incrementally learn the current model; Step 8: Input the test drawing into the model designed and trained through steps 1-7 to obtain the detection and recognition results; In step 2: The visual feature extraction and prediction module is a multi-level attention-based visual feature extraction and prediction module. Through local attention and global attention branches, it extracts receptive field features at different scales. The introduction of dilated convolution can further capture the relationship between distant pixels and improve the fitting ability of neighborhood and global features. With the help of joint optimization, the detection branch is optimized by recognition loss to correct the text detection and segmentation results. In the text recognition codec, proportional upsampling is added to reduce the loss of small character features in the serialization process and improve the recall rate of small characters. The character structure feature extraction and prediction module is a CCR-CLIP-based character structure feature extraction and prediction module. It constructs an image encoder and a character encoder and pre-trains the module based on contrast loss, introducing character structure features into the visual model and improving the model's ability to recognize Chinese characters from the perspective of multimodal learning. To improve the fitting effect of multimodal prediction information, a multimodal prediction fusion module based on contrastive learning was designed. The prediction of the visual branch and the character canonical representation of the character structure feature branch are multiplied by two linear layers. A series of convolutional structures and linear layers are used to improve the text recognition accuracy of complex models in complex scenarios. The visual feature extraction and prediction module based on multi-level attention includes three sub-modules: a multi-level attention branch, a recognition and conversion module, and a text recognition codec based on proportional upsampling; among which, The multi-level attention branch includes local attention and global attention. The local attention branch employs a residual dilated convolution structure and a window self-attention mechanism to focus on local image details. Dilated convolution constructs multi-scale receptive fields in each convolutional layer, which is beneficial for the model to extract texture features of different granularities. The residual structure learns and preserves more subtle spatial information during backpropagation. Based on this, the residual dilated convolution structure enables the features of irregular text regions to be extracted more accurately, and irregular characters within the text can be assigned more reasonable and accurate weights in subsequent attention mechanisms. Specifically: RoI feature map The input residual dilated convolution structure is fed in as follows: in, It consists of a dilated convolutional layer with a kernel size of 5×5 and an expansion rate of 2, and a The layers are structured as follows; next, the fitted features are fed into the window self-attention to capture local dependencies, as shown in the following equation: in, These represent the query, key, and value matrices in the self-attention mechanism, respectively. It is the dimension of the key / value matrix. It is the number of sampling points in a window; It is a relative positional deviation, through The introduction of [a method] enables relative position encoding of the self-attention weight matrix, thereby constructing local dependencies within the image window; finally, the obtained window self-attention fitting result is passed through a feedforward neural network. The nonlinear transformation of the obtained features enables the network to capture higher-order image features; the global attention branch employs dilated convolution and multi-head self-attention mechanisms to focus on the global contour features of the image; dilated convolution is used to increase the receptive field of the convolution, resulting in a feature map that fits the neighborhood spatial information. This feature map is used as the query matrix, and the feature map after global average pooling is used as the key-value matrix. Global dependency capture is achieved through multi-head self-attention, as shown in the following equation: in, It is a global average pooling layer. This is a multi-head self-attention mechanism. Take 8; the input feature image processed by dilated convolution using the above structure. Each location interacts with global features to form a self-attention output, which is compared with the global information of the overall image at each location to extract long-distance dependencies and global contextual relationships. Through multi-head setup, this structure captures rich contextual information from different subspaces, improving the network's feature representation ability and global feature fitting effect in complex scenes. The recognition and conversion module uses the original structure of the SwinTextSpotter v2 network to generate a tight mask for text regions, thereby achieving joint optimization of text detection and recognition in two stages. The text recognition codec based on proportional upsampling generally follows the architecture of SwinTextSpotter v2. Building upon the original SwinTextSpotter v2 text recognition codec architecture, a bilinear interpolation upsampling layer is introduced into the sequence encoding to enhance the text feature map. Dimensional Ascension This makes the image height more detailed and the pixels smoother, thus preserving more high-frequency information during downsampling.

2. The robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 as described in claim 1, characterized in that, The CCR-CLIP-based character structure feature extraction and prediction module consists of an image encoder and a character encoder. It constructs a standardized representation of Chinese characters by extracting their structural stroke features, and obtains the recognition results of Chinese characters through contrastive learning. This module obtains the standardized character representation through separate pre-training. The prediction of the character feature branch and the prediction of the visual branch are fused in a multimodal manner to obtain the final prediction result. The training set of this module consists of printed images of all characters and the radical stroke sequence of the characters. The image encoder is responsible for extracting the visual features of the input character image, while the text encoder extracts the features of the corresponding radical sequence. Finally, two contrastive losses are used to supervise the model. Specifically, the image encoder uses ResNet-50 as the backbone network to obtain image features. Then, after global average pooling, we get ,Will Embedded in the visual feature space, as shown in the following equation: in, Let be the projection matrix. Alignment dimension; The text encoder consists of two Transformer encoder layers and one embedding layer; through the encoder, the radical sequence... Encoded as sequence features The corresponding output features for each time step The length of the radical sequence. It is considered All features; finally, embedding them into the text feature space to obtain As shown in the following formula: in, It is a projection matrix; to match the image and character structural features, a contrast loss between the image and character structure is designed. The goal is to minimize the loss between corresponding images and character structures, and maximize the loss between non-corresponding images and character structures, as shown in the following formula: To reduce prediction errors caused by different font styles and similar characters, a contrast loss is introduced between visual features of input images with the same radical stroke label. To maximize the feature loss between similar-looking characters, as shown in the following formula: in, Visual features of radicals with the same stroke sequence The set of values; thus, the final loss function of the CCR-CLIP model is obtained, as shown in the following equation: From this point on, the CCR-CLIP model can be trained using printed images of Chinese characters, and a text encoder can be used to generate standardized representations of all candidate Chinese characters.

3. The robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 as described in claim 1, characterized in that, The contrastive learning-based multimodal prediction fusion module will predict the visual branch. Character canonical representation of character structure feature branches Through two linear layers After multiplication, predictions are further fused using a set of depthwise separable convolutions, and finally... Obtain the final text recognition result. As shown in the following formula: in, and Dimensions and and The dimensions are consistent.

4. The robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 as described in claim 1, characterized in that, In step 3: The prior knowledge includes naming rules for electrical components and rules for writing identification marks; Fine-tuning post-processing strategies based on prior knowledge include: post-processing of adjacent text segmentation based on regular expressions and post-processing of inconsistent and missed detections in text segmentation based on merging and reconstruction. Specifically, the post-processing of adjacent text segmentation based on regular expressions is used to solve the problem that texts that are too close together are detected as a whole. The post-processing of inconsistent and missed detections of text segmentation based on merging and reconstruction is used to solve the problem that the text identifier is identified as two parts due to the distance between them and the problem of missing the "I" character. The text detection and recognition results obtained from the end-to-end text detection and recognition network are sent to this module, and the post-processing operations of the two branches can be completed in parallel according to the filtering conditions. The specific steps for post-processing adjacent text segmentation based on regular expressions are as follows: S3.1.1 Length threshold filtering: After text detection and recognition are completed, all texts with an aspect ratio greater than a set threshold are extracted based on the "text aspect ratio" filtering strategy. The text prediction bounding box; this process can effectively filter out long and thin text boxes that may cause segmentation problems; S3.1.2 Regular Expression Matching: Use regular expressions to split long text, ensuring that the resulting text boxes conform to the set format; S3.1.3 Text box segmentation: Based on the results of regular expression matching and segmentation, and by using the relative positions of the text content, the text boxes are cut to obtain several new text boxes after processing. The specific steps for handling inconsistencies and missed detections in text segmentation based on merging and reconstruction are as follows: S3.2.1 Keyword Filtering: Filter the text boxes containing the keywords "P", "Q", "temperature", and "gear". All subsequent operations will be based on these text boxes. S3.2.2 Extended Merging: For the selected text boxes, if there are only the above keywords, extend them to the right by a certain number of pixels. If there are text boxes with numeric text within the range, merge them with the original text boxes. S3.2.3 Traversal Restructuring: To address the issue of missing the "I" character, for all text boxes containing the aforementioned keyword, a certain pixel range is traversed downwards. If a text box with numeric content is found within a certain area below, that numeric text box is extended to the left by a certain distance to reconstruct a new text box. The distance is determined by the position of the text box containing the keyword.

5. The robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 as described in claim 1, characterized in that, In step 6: When training the network model constructed in steps 1-5, a strategy of pre-training and then fine-tuning is adopted. That is, the model is first pre-trained in a general scenario to enable the model to have basic text detection and recognition capabilities, and then the power grid plant wiring diagram dataset is used to fine-tune the model. The pre-training phase employs a data mining-based training set selection strategy. First, the network model constructed in steps 1-5 is coarsely trained using the original general scenario dataset to obtain a low-precision model. This low-precision model is then used to perform high-confidence data filtering on the training set data. Next, the model based on the improved PP-OCRv3 is used as the high-precision model and subjected to low-confidence data filtering. After two rounds of data filtering, a high-quality training set is obtained. Finally, this high-quality training set is used to perform fine-tuning and iteration on the low-precision model. The details are as follows: (1) High-confidence data filtering based on low-precision model First, the proposed model is coarsely trained on a subset of training data to obtain a low-precision model, which can then be used to quickly predict large-scale data. Using this low-precision model, text detection and recognition are performed on a general scene text dataset of tens of millions of words. Text prediction boxes with a confidence score greater than 0.95 are selected, and this part is considered redundant text data. For the training image containing these redundant texts, if there is only one redundant text in the training image, the entire sample is deleted from the training set; if there are other texts in the training image besides the redundant text, the training image is retained in the training set. (2) Low-confidence data filtering based on high-precision model After filtering out redundant text data with high confidence through high-confidence data filtering, in order to further filter out low-confidence training data, an algorithm model based on the improved PP-OCRv3 is applied as a high-precision model to predict the remaining training data. For text prediction boxes with a confidence of less than 0.15, they are classified as negative samples with poor quality or difficult to identify, which may interfere with model training. For these negative samples, their ground truth labels are directly deleted from the training set label file. If the negative text is the only text in the training image, the entire image is deleted from the training set. By following the steps above, redundant data with high confidence and negative data with low confidence can be filtered out from the general scenario training set. The remaining training data are high-quality training data that have a positive impact on the pre-training of the model in the next stage. (3) Fine-tuning and iteration The selected high-quality training data is used to refine the network model constructed in steps 1-5, resulting in a more accurate network model; this data is then used for subsequent fine-tuning training based on the plant wiring diagram.

6. The robust generalization detection and recognition method for power plant wiring diagram text based on the improved SwinTextSpotter v2 as described in claim 1, characterized in that, In step 7: The knowledge distillation-based incremental learning strategy for multi-type wiring diagram text detection and recognition uses the model obtained from the initial training as the original model. Based on this, all parameters of the text content recognition stage in step 2 are frozen into the new model, and only the model parameters involved in feature extraction and the text detection stage in step 1 are incrementally learned. The specific training strategies are as follows: Assumption Before that time, the original model had already been trained on the historical wiring diagram set. Complete early training; At time 1, a new set of wiring diagrams to be trained appears, and at this time from A partial historical wiring diagram dataset was constructed by randomly retaining some images. Along with the new wiring diagram training set The training set constitutes this knowledge distillation; in addition, all parameters of the original model's text content recognition stage are frozen into the new model; In the process of knowledge distillation for the new model, the batch size is assumed to be... Then in and Sample separately The system trains on a sample network, feeding the input into a dual-stream feature extraction network based on text region masking to extract image features and generate a coarse text region mask. The masked feature map is then fed into subsequent modules to predict text detection and recognition results. The detection and recognition results are compared with the ground truth labels to calculate the prediction loss. The predicted loss includes the original loss of the original model. and the weighted cross-entropy loss of the text region mask generation branch. As shown in the following formula: in, This is the mask branch scaling factor. For text candidate boxes in the text detection stage L1 loss, The cross-entropy loss is the result of text segmentation during the text detection stage. This represents the loss during the text recognition stage; To achieve supervised training of the new model by the original model, The training samples are fed into the original model for prediction and output. The soft labels of the predicted text sequence are used as prior knowledge to calculate the distillation loss along with the prediction results of the new model. As shown in the following formula: in, Representing the original model and the new model respectively. The predicted vector, These are vectors representing the text prediction sequences of the original model and the new model, respectively. Indicates the first At each point, the original model and the new model Predicted value Then it means in the first At each point, the predicted values ​​of the text prediction sequences of the original model and the new model; and These represent the predictions made in the current round. The number of [number] and the number of text prediction sequences; Finally, the prediction loss and distillation loss are added together to obtain the loss function of the new model's knowledge distillation process, as shown in the following equation: in, This is the distillation loss ratio coefficient; During incremental learning, the constructed dual-stream feature extraction network based on text region mask generation includes a feature extraction backbone network based on Swing Transformer and FPN, and a lightweight text region mask generation branch based on improved MobileNetv3. During shared feature extraction, the feature extraction branch for the entire image remains unchanged, still employing the feature extraction backbone network based on Swing Transformer and FPN, resulting in feature maps at four different scales. To improve the accuracy and generalization ability of text detection in different types of wiring diagrams, a lightweight text region mask generation branch based on an improved MobileNetv3 was introduced on the original backbone network. This branch fits the input image through network layers to generate a rough text region mask for the entire input diagram, which is then combined with... Multiplying them together yields the final feature map, which is used by the text detection branch to predict and output the detection results. To ensure that the text region mask fully covers the text, a weighted cross-entropy loss function is designed to focus on recall while supervising the overall image segmentation accuracy. Specifically, in the feature extraction stage, the input The original drawing image is fed into a convolution kernel. A convolutional layer with a stride of 2 is used to extract features. Then, 11 depthwise separable convolutional blocks are sequentially fed in to extract depth features, resulting in... The feature maps are divided into four scales, among which the depthwise separable convolutional blocks are the inverted residual structure of MobileNetv3. During the feature fusion stage, the feature maps are fused using deconvolution and upsampling strategies. The fusion yields a preliminary text region mask. Furthermore, a text region mask with a scale of 1 / 4 of the original image's length and width is obtained through a linear layer for dimensionality reduction. Finally, in the text region mask generation stage, Stepwise downsampling yields three additional text region masks with decreasing scales. Mix the text region mask with Performing a dot product operation yields the feature map based on the text region mask. ; After feature extraction and fusion using the above network structure, four text region masks at different scales are obtained. This indicates the presence of approximate text regions in the drawing image. The subsequent text detection stage can then fit text candidate boxes based on the features of these regions. This requires the text region mask to completely cover all possible text regions. To achieve this goal, a weighted cross-entropy loss function is designed in the text region mask generation branch, as shown below: in, Represents a text region mask At pixel The predicted value at that location, Represents a text region mask At pixel The truth label at the location indicates whether the pixel is contained within a certain truth text box; The positive example weight hyperparameter is greater than 1, which means that the current supervision pays more attention to the positive example pixels. That is, while ensuring the overall prediction accuracy of the text region, more attention is paid to the recall rate of the text region, so that the predicted coarse text region can achieve full coverage of the true value.

Images