Text detail map-based method for supervising end-to-end text detection and recognition

Abstract

A text detail map-based method for supervising end-to-end text detection and recognition, pertaining to the field of text processing. The method comprises the following steps: given an input image containing text in any shape, processing the input image by means of two separate processing branches; designing a text attention head (TAH), and designing a feature pyramid enhancement fusion module (FPEFM); the FPEFM performing feature self-enhancement at different sizes, fusing text image local features and global text position information extracted by a TAH module, and fusing features extracted by the TAH from feature maps of different sizes; stacking a plurality of FPEFMs to continuously enhance the feature representation capability of a model and the depth of the model; and sampling the feature maps to a unified size to obtain a final enhanced feature map.

Description

A supervised approach for end-to-end text detection and recognition based on text detail graph Technical Field

[0001] The present invention belongs to the field of text processing and relates to a supervised method for end-to-end text detection and recognition based on a text detail graph. Background Art

[0002] Understanding text content in natural scenes is a fundamental task in computer vision applications in the deep learning era, such as image retrieval, visual question answering, and image understanding. Due to the powerful representation capabilities of deep neural networks, scene text detection based on deep learning algorithms is demonstrated in Figure 1, comparing text detection alone and end-to-end text detection and recognition. The first row shows the results of text detection alone. The second row shows the results of end-to-end text detection and recognition. While recognition models have rapidly developed in the past few years, many challenges remain. First, many works treat text detection and recognition as separate tasks, focusing on either text detection alone or text recognition alone.

[0003] For the currently common text detectors, a convolutional neural network is first used to extract the feature map of the input image, and then a designed decoder is used to generate a rectangular or polygonal bounding box of the scene text. On the other hand, most text recognition methods are based on a sequence prediction network based on the text in the text image. Some previous works divide text detection and recognition into two steps, lacking the complementarity between text detection and recognition. As shown in Figure 1, there are great defects in performing text detection alone, and end-to-end text recognition (the second row of Figure 1 makes up for the shortcomings of detection). Although some research progress in recent years has improved the accuracy of end-to-end arbitrary shape text recognition, due to the excessive number of model parameters or the complexity of the model pipeline, its inference speed is low and it is difficult to apply to real scenarios. Therefore, how to design an efficient and accurate end-to-end arbitrary shape text detection and recognition method remains an urgent problem to be solved.

[0004] In recent years, many researchers have proposed integrating text detection and recognition tasks into a single network, using various strategies. Mask TextSpotter (MTS) is perhaps the first arbitrarily shaped text detection and recognition algorithm to leverage character-level supervised annotations for simultaneous detection and recognition of text lines and characters. It uses Fourier transforms on text instance contours to model highly curved text. FOTS is an end-to-end learnable text detection and recognition model. This model introduces RoIRotate, which generates oriented text regions from convolutional feature maps to support the recognition of tilted text. ABCNet adapts to arbitrarily shaped text using parameterized Bezier curves and designs a new BezierAlign layer to accurately extract convolutional features from arbitrarily shaped text instances, achieving improved accuracy. PAN++ represents arbitrarily shaped text based on the core (i.e., the center region) of the text, enabling better differentiation between adjacent text and achieving real-time detection and recognition accuracy. Mask TextSpotterv3 introduces an anchor-free segmentation proposal network and provides an accurate polygonal representation of the proposals, addressing the shortcomings of RPN-based techniques. Transformers are increasingly being used in image processing. Transformer is considered to be better at capturing global feature information than traditional convolutional structures. However, it is difficult to obtain global feature information. Transformer requires more computing resources than traditional convolutional structures to capture long-term dependencies, which also weakens local feature information. Therefore, designers combine convolution and Transformer to model the local and global dependencies of sequences. Many studies have proposed using Transformer instead of RCNN backbone network for text recognition tasks. For example, SwinTextSpotter used a Transformer-based method and an ensemble prediction scheme for the first time to perform end-to-end scene text detection and recognition. These techniques have demonstrated the effectiveness of Transformer for text detection and recognition. However, in addition to calculating the text target of interest, Transformer also calculates additional unnecessary background areas, which makes the model pipeline of the Transformer structure complex and the computing resources huge, making it difficult to apply to real industrial scenarios.

[0005] When only performing text detection algorithms, text line detection cannot completely surround text lines, and text detection cannot recognize text content. In the STDC-Seg semantic segmentation algorithm, image detail edges are used as supervisory signals, making the model highly accurate and fast. Inspired by the STDC-Seg model, a text detail map (text texture and boundaries) is added to the model as a new supervisory signal. Inspired by the PAN++ algorithm, DiZNet, an end-to-end text detection and recognition algorithm based on text detail maps, is designed. Based on the representation advantages of text detail maps, the model proposed in this paper can effectively detect and recognize text of arbitrary shapes and has a high real-time speed. The following designs are adopted in the model:

[0006] 1) First, the model uses ResNet18 as the backbone network for feature extraction. The model has few parameters and a simple pipeline, while also having good feature representation capabilities.

[0007] 2) To enhance the representation ability of the model, extract and fuse local features of text images and global text position information, a text attention head TAH is proposed. The function of the text attention head TAH is to extract key features from the four feature maps of different pixel sizes (1 / 32, 1 / 16, 1 / 8 and 1 / 4) extracted by the backbone network, that is, to extract the required text features.

[0008] 3) To fuse the local features of text images extracted by the text attention head (TAH) with the global text location information, a low-computation feature enhancement fusion network consisting of a stackable feature pyramid enhancement fusion module (FPEFM) is proposed. It self-enhances shallow backbone features and fuses the text features extracted by the text attention head (TAH). The FPEFM module is a stackable structure. This allows the FPEFM to be superimposed on a lightweight backbone. Combined with the residual structure fusion design of the text attention head (TAH), it effectively enhances the representational power of the backbone network feature extraction and the depth of the model, enabling the model to adaptively extract and enhance feature representation.

[0009] 4) Taking advantage of the text detail graph representation, the model's predicted text detail graph is integrated into the text detection and recognition heads, effectively improving the model's detection and recognition performance. Through this design, the model effectively improves the accuracy of text detection and recognition and achieves excellent inference speed.

[0010] This paper proposes a novel and efficient end-to-end natural scene text detection and recognition framework, called DiZNet (i.e., Arbitrary Shape Text Detector and Recognizer based on Text Details). Unlike previous end-to-end text detectors and recognizers that use complex, hand-crafted network architectures resulting in low inference speed, DiZNet's innovative design uses text detail maps as representations. This model is a lightweight text detection and recognition model based on ResNet18 as the backbone. The model uses text attention heads (TAH) to extract local features of text images and global text location information. The extracted features are integrated into the stackable feature enhancement fusion module (FPEFM). Under the supervision of the text detail ground truth map label, the model predicts the text detail map and fuses the text detail map into the text detection head and recognition head. Through testing on public scene text detection benchmark datasets, the method has strong generalization ability and real-time detection speed.

[0011] Previous text detection and recognition models had too many parameters or complex model pipelines, resulting in slow inference speeds and difficulty in real-world applications. Therefore, designing an efficient and accurate end-to-end method for detecting and recognizing text of arbitrary shapes remains an urgent problem.

[0012] Summary of the Invention

[0013] In view of this, the object of the present invention is to provide a supervised method for end-to-end text detection and recognition based on text detail graph.

[0014] In order to achieve the above object, the present invention provides the following technical solutions:

[0015] The method based on the text detail graph as a supervision method for end-to-end text detection and recognition includes the following steps:

[0016] Given an input image containing text of arbitrary shape, the input image is processed by two independent processing branches;

[0017] The first processing branch extracts text texture and boundary information from the input image and generates a text detail map;

[0018] The second processing branch uses deep learning neural networks to learn and infer the location and content of text in different contexts;

[0019] Under the supervision of text detail maps as new labels, an end-to-end natural scene text detection and recognition model is proposed;

[0020] In the second processing branch, during the deep learning branch processing, a lightweight feature extraction network ResNet18 is used as the backbone network of DiZNet; the backbone network extracts shallow feature maps with different resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 from the input image, corresponding to the conv2, conv3, conv4, and conv5 layers respectively; the shallow feature maps are processed using 1x1 convolution, and the number of channels is unified to 128;

[0021] Design a text attention head (TAH) and a feature pyramid enhancement fusion module (FPEFM). FPEFM performs feature self-enhancement at different scales, integrating local text image features extracted by the TAH module with global text position information, and integrating features extracted by the TAH from feature maps of different scales. By stacking multiple FPEFMs, the model's feature representation capability and depth are continuously enhanced. The feature map is upsampled to a uniform size to obtain the final enhanced feature map.

[0022] The enhanced feature map predicts three components:

[0023] The detection head is responsible for predicting the text location; the detection head predicts the text core, text region and text vector, and predicts the text line by pixel aggregation;

[0024] The detail head uses the text detail map as a supervisory signal to predict text texture and boundaries;

[0025] The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result;

[0026] During the model inference stage, there is no need to use the ground-truth text detail map as a supervision label; instead, the model predicts text texture and boundaries, and then fuses the predicted text detail map into the detection head to infer text location; the predicted text detail map is fused into the recognition head to identify text content.

[0027] Furthermore, in the second processing branch, during the deep learning branch processing, a lightweight feature extraction network ResNet18 is used as the backbone network of DiZNet; the backbone network extracts shallow feature maps with different resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 from the input image, corresponding to the conv2, conv3, conv4, and conv5 layers, respectively; 1x1 convolution is used to process the shallow feature maps, and the number of channels is unified to 128; a text attention head TAH is designed, and a feature pyramid enhancement fusion module FPEFM is designed; FPEFM performs feature self-enhancement at different scales, fuses the local features of the text image extracted by the text attention head TAH module and the global text position information, and fuses the features extracted by the text attention head TAH from feature maps of different sizes; by stacking multiple FPEFMs, the feature representation ability of the model and the depth of the model are continuously enhanced; the feature map is upsampled to a uniform size to obtain the final enhanced feature map, specifically:

[0028] Step 1: Input the original image into the ResNet18 feature extraction backbone network. The ResNet18 feature extraction backbone network will extract features from the original image and obtain feature maps of 128 channels with four resolution sizes. The obtained shallow feature maps have resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively.

[0029] Step 2: The 128-channel feature map of 4 resolutions obtained in the first step is used as input and sent to the text attention head TAH. Each 128-channel feature map will be sent to the text attention head TAH for calculation, extracting the information of the shallow feature map, and obtaining the local feature information of the text image and the global text position information. The mathematical calculation process of the shallow feature map is as follows. Assume that the number of input channels is C, C is set to 128 in this experiment, W is the width of the feature map, H is the height of the feature map, and the input shallow feature map is expressed as C×W×H; use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the horizontal X-axis and vertical Y-axis coordinates; the vertical direction expression is written as:

[0030] The horizontal expression is written as:

[0031] The shallow feature map is encoded along the horizontal and vertical directions and then aggregated to obtain feature maps that are sensitive to text areas in the horizontal and vertical directions; these feature maps are represented as and Subsequently, the text region sensitive feature map is fused with the feature maps of the two directions through a 1×1 convolution kernel, and the channel dimension is reduced by the convolution operation. The Sigmund activation function is then used to obtain a feature map f, which represents the fusion of the text region sensitive features in the two directions.

[0032] f=δ(F1([z h ,z w ]))

[0033] In order to enforce text region sensitive features, TA threshold activation operation is used to enhance important text region features; TA 0.1 (f represents pixel activation, and the pixel values ​​greater than the threshold of 0.1 are retained in the reduced C / r×1×(W+H) feature map. r represents the scaling factor for reducing the number of channels. The threshold activation operation enhances and retains the important information of the text area. Finally, the activated feature map t and the feature map before activation are added pixel by pixel, and the text area feature map is re-weighted to retain important shallow feature information and strengthen the representation of the text area. The default value of the TA threshold activation operation is set to 0.1 in the experiment.

[0034] t=TA 0.1 (f)+f

[0035] For the feature map t, we get the intermediate feature map that encodes the spatial information along the X-axis and Y-axis, and get the attention weight g of the X-axis respectively. h and the attention weight g on the Y axis w ;

[0036] g h =σ(F h (t h ))

[0037] g w =σ(F w (t w ))

[0038] The original input C×W×H shallow feature map is recorded as x c (i, j), obtain the reweighted feature map; the expression of the reweighted output feature map is denoted as r c (i,j), written as:

[0039] To ensure the robustness of the re-weighted feature map, the features are fed into spatial attention and channel attention, and their outputs are summed. Channel attention effectively enhances contextual information in the channel dimension, improving text detection and recognition performance. The sum of the outputs of the dual attention mechanisms is fused to obtain the final features. This then completes the processing of the text attention head TAH module, and the output of the text attention head TAH module becomes part of the input of the feature pyramid enhancement fusion module FPEFM. The text attention head TAH module is connected to the model backbone through a residual structure, which is conducive to adaptive feature extraction and prevents the degradation of feature extraction capabilities.

[0040] Step 3: The 128-channel feature map of 4 resolutions obtained in the first step and the local feature information of the text image and the global text position information extracted in the second step are fed into the feature enhancement and fusion module FPEFM.

[0041] The 128 feature maps of the four shallow layers are subjected to the self-enhancement stage of the feature enhancement fusion module (FPEFM). In the first stage of self-enhancement, the feature map input to the feature enhancement fusion module is sized 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image pixels, respectively, to form a feature pyramid structure. Iterative enhancement is performed from a 1 / 4 pixel step to a 1 / 32 pixel step, and then from a 1 / 32 pixel step to a 1 / 4 pixel step. The enhanced feature pyramid point elements are sequentially added together to obtain the output feature pyramid.

[0042] In the second stage, the fusion stage, the output of the shallow feature map self-enhancement pyramid is fused with the local features of the text image and the global text position information extracted by the text attention head TAH; the corresponding pixels are concat together, and feature fusion is achieved through 1x1 convolution. The feature map after fusion maintains the same number of input feature channels; the output feature map of the enhanced feature pyramid enhancement fusion module becomes part of the input of the next round of feature pyramid enhancement fusion module; the other part of the input comes from the local features of the text and the global text position information that the text attention head TAH module continues to extract; the text attention head TAH module extracts features from the backbone network and uses a residual design structure for fusion, allowing the feature pyramid enhancement fusion module to adaptively fuse the required features, thereby enhancing the representation ability and depth of the model;

[0043] Step 4: The shallow feature maps of the four resolutions are fused through two layers of stacked, residual-structured text attention heads and feature pyramid enhancement to obtain the final 512 enhanced feature maps. The 512 enhanced feature maps are uniformly upsampled to a quarter of the resolution and enter the post-processing of end-to-end text detection and recognition.

[0044] In the post-processing stage, the enhanced feature map is used to predict three components: detection head, detail head, and recognition head;

[0045] The detection head is responsible for predicting the text location; the detection head predicts the text core, text area and text vector, and finally predicts the text line through pixel aggregation; the detection head predicts the text area through three loss functions, and the loss of text detection is expressed as Among them, Lregion represents the loss of text region segmentation, Lkernel represents the loss of text kernel; hyperparameters α and β are used to balance and The weights of are set to 0.6 and 0.25 in all experiments respectively; Dice loss is used to supervise the text region segmentation result Pregion, as well as the point-wise element addition of text kernel and text activation map Pkernel;

[0046] and The mathematical expression is as follows:

[0047] P region and G region (i) represents the predicted value and true value of the text area of ​​the i-th pixel in the segmentation result; similarly, P kernel and G kernel (i) represents the predicted value of the text kernel of the i-th pixel and the true value of the text kernel; aggregation loss Used to minimize the distance between pixels in the same text line and between text kernels;

[0048] N represents the number of text lines; T i represents the text area of ​​the i-th text line, K i Represents the text kernel of the i-th text line; D1(p,Ki) represents the text pixel p and the text kernel K i To distinguish different text kernels and non-text areas of text pixels, the following equation is used:

[0049] D b (Ki) represents the text kernel K i The distance between the kernel and the background; D2(Ki,Kj) represents the distance between different text kernels;

[0050] The detail head utilizes the text detail map as a supervisory signal to predict text texture and boundaries.

[0051] Furthermore, the detail head uses the text detail map as a supervisory signal to predict text texture and boundaries, including the following steps:

[0052] Text detail truth map processing;

[0053] In the DiZNet model, the original image input in the first step is processed in advance to produce a ground-truth text detail map. The DiZNet model uses this ground-truth text detail map as the model's supervisory signal. First, the original image is cropped based on the labeled text location labels, cropping only the text area and filling the pixel values ​​of other non-text areas with zeros. This is the cropped text area. OpenCV's Canny edge extraction algorithm is then used to extract the text texture and boundaries from the cropped image. The extracted text texture and boundary maps are used as ground-truth text detail maps. The processing of two sets of ground-truth text detail maps is demonstrated: the ICDAR2015 dataset for multi-directional straight text and the TotalText dataset for arbitrarily shaped curved text.

[0054] Prediction of text detail graph:

[0055] In the DiZNet model, the detail map prediction process mainly involves predicting the texture and boundaries of the text in the input image based on the text detail map label. First, the original text image is cropped and edge extracted to obtain the ground truth map of the text texture and boundary as the label. After processing according to the steps of the text detail map processing process, white pixels are text information; black pixels are non-text areas.

[0056] In the first step, the original image is sent to the ResNet18 backbone network for feature extraction. In the second step, the text attention head extracts the local features of the text image and the global text position information. In the third step, feature enhancement and fusion are performed to obtain an enhanced feature map with 512 channels.

[0057] Through the CBR operation, which is a collection of 3x3 convolution, normalization, and ReLU activation function operations, the enhanced feature map is further divided into two small steps;

[0058] 1) Under the optimization of text fire kernel loss, the minimum description area of ​​the text is predicted through the text header;

[0059] 2) A 6-channel feature map is obtained through 1x1 convolution fusion feature, and a 1-channel text detail map is predicted through the detail head. The detail head is composed of three CBR operations and finally a Sigmund activation function. The resulting detail map pays more attention to the text position information and texture information. The obtained text kernel is subjected to 1x1 convolution, normalization, ReLU activation function, and finally threshold activation. The 1-channel feature map of the pixel area with a pixel value greater than 0.1 is retained. The two 1-channel feature maps are added point by point to fuse the text kernel and text details. Under the supervision of the detail ground truth map label, the detail loss function is used to simultaneously optimize the text kernel and text detail map. The algorithm prediction process of the text detail map. With the fusion of text detail maps, the text detection capability continues to improve.

[0060] Text detail map loss L detail_loss It optimizes the text detail map and text kernel simultaneously based on the text detail truth map supervision label. The calculation expression is as follows:

[0061] L detail_loss ={l1,,,,,l N},l n

[0062] =-[y n ·log(σ(x n ))+1-y n ·log(1-σx n )]

[0063] The predicted text detail map and the predicted text kernel are flattened into a one-dimensional vector. Then, BCEWithLogitsLoss is used to calculate the loss of each pixel under the supervision of the text detail ground truth map. σ represents the Sigmoid activation function. Finally, the flattened one-dimensional vector is restored to the size of the input image. Under the supervision of the text detail ground truth map, the predicted text detail map is fused with the detection head and recognition head to improve the performance of end-to-end text detection and recognition.

[0064] The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result;

[0065] Loss function for text recognition as follows:

[0066] Where w is the truth record containing the EOS symbol; |w| represents the number of characters; |w|i represents the i-th character;

[0067] The total loss of the model is:

[0068] Text Detection Loss Text recognition loss Text detail map loss sum.

[0069] The beneficial effects of the present invention are as follows: the present invention designs a text detail ground truth map as a supervisory label and designs a text detail ground truth map processing process. Thanks to the advantages of the text detail map representation, DiZNet achieves a good trade-off between accuracy and efficiency on challenging datasets. For example, DiZNet achieves 91.2% accuracy and 85.9% F-measure on Total-Text at a speed of 38.4FPS, and 83.8% F-measure on ICDAR2015 at a speed of 30.0FPS. The code and model will be made public at: https: / / github.com / DiZ-gogogo / DiZNet. The DiZNet model has real-time inference speed, efficient text detection and recognition accuracy, recall rate, F-measure and FPS.

[0070] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0072] Figure 1 shows the comparison between text detection only and end-to-end text detection and recognition;

[0073] Figure 2 shows DiZNet, an end-to-end text detection and recognition method based on text detail graph;

[0074] Figure 3 shows the text attention head TAH;

[0075] Figure 4 shows the feature pyramid enhancement fusion module FPEFM;

[0076] Figure 5 shows the text detail truth map processing process;

[0077] Figure 6 shows the text detail prediction process;

[0078] Figure 7 shows the visualization results of the text attention head TAH;

[0079] Figure 8 shows the visualization results of the Total-Text dataset detection;

[0080] Figure 9 shows the visualization results of the ICDAR15 dataset detection;

[0081] Figure 10 shows the comparison experiment of Total-Text and IC15 dataset detection;

[0082] Figure 11 shows the visualization results of end-to-end detection and recognition on the Total-Text dataset;

[0083] Figure 12 shows the visualization results of end-to-end detection and recognition on the ICDAR15 dataset;

[0084] Figure 13 shows the end-to-end recognition results of the Total-Text dataset;

[0085] Figure 14 shows the end-to-end recognition results of the ICDAR15 dataset. DETAILED DESCRIPTION

[0086] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0087] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0088] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0089] Please refer to Figures 1 to 14. The end-to-end text detection and recognition method DiZNet based on text detail graph is as follows: given an input image containing text of arbitrary shape, the image is processed by two independent branches. First, the first processing branch extracts text texture and boundary information from the image to generate a text detail graph (as shown in A→K→L in Figure 2). The second processing branch uses a deep learning neural network to learn and infer the location and content of text in different contexts. Under the supervision of the text detail graph as a new label, a novel end-to-end natural scene text detection and recognition model is proposed. In the text detail truth graph processing branch, the specific steps are shown in Figure 5.

[0090] During the deep learning branch processing, a lightweight feature extraction network, ResNet18, is used as the backbone network of DiZNet. The backbone network extracts shallow feature maps with different resolutions (1 / 4, 1 / 8, 1 / 16, and 1 / 32) from the input image, corresponding to the conv2, conv3, conv4, and conv5 layers, respectively. Subsequently, these feature maps are processed using 1x1 convolutions, unifying the number of channels to 128. ResNet18 has good feature representation capabilities, few parameters, and low model complexity. In order to extract important local features and global position information of text images and further enhance the representational capabilities of the model, a text attention head (TAH) is designed, as shown in Figure 2 (D). The text attention head (TAH) can effectively extract local features of text images and global text position information. How to effectively fuse the extracted text features to enhance the robustness of the model? To this end, a feature pyramid enhancement fusion module (FPEFM) is designed, as shown in Figure 2 (E). FPEFM further integrates the local features of the text image and global text location information extracted by the text attention head (TAH) module by performing feature self-enhancement at different scales. This allows the model to effectively fuse the features extracted by the text attention head (TAH) from feature maps of different scales. FPEFM is stackable, and by stacking multiple FPEFMs, the model's feature representation capabilities and depth are continuously enhanced. Finally, the feature map is upsampled to a uniform size to obtain the final enhanced feature map, as shown in G in Figure 2. In the post-processing stage of the model, the enhanced feature map predicts three components:

[0091] The detection head is primarily responsible for predicting text locations. It predicts text kernels, text regions, and text vectors (shown as H, I, and J in Figure 2), and ultimately predicts text lines through pixel aggregation.

[0092] The detail head uses the text detail map as a supervisory signal to predict text texture and boundaries, as shown by M in Figure 2).

[0093] The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result (as shown by O in Figure 2).

[0094] During the model inference phase, the ground-truth text detail map is not used as a supervised label. Instead, the model predicts a text detail map (text texture and boundaries), which is then fused into the detection head to infer text locations. The predicted text detail map is then fused into the recognition head to identify text content.

[0095] Step 1: Input the original image into the ResNet18 feature extraction backbone network. The ResNet18 feature extraction backbone network will input the original image for feature extraction and obtain feature maps of 128 channels with 4 resolution sizes, and obtain shallow feature maps with resolution sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively.

[0096] Step 2: The 128-channel feature map of 4 resolutions obtained in the first step is used as input and sent to the text attention head TAH. Each 128-channel feature map will be sent to the text attention head TAH for calculation, extracting the information of the shallow feature map, and obtaining the local feature information of the text image and the global text position information. The mathematical calculation process of the shallow feature map is as follows. Assume that the number of input channels is C. C is set to 128 in this experiment. W is the width of the feature map and H is the height of the feature map. Therefore, the input shallow feature map is represented as C×W×H. Use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the X-axis (horizontal) and Y-axis (vertical) coordinates. The vertical expression can be written as:

[0097] The horizontal expression can be written as:

[0098] The shallow feature map is encoded along the horizontal and vertical directions and then aggregated to obtain feature maps that are sensitive to text areas in the horizontal and vertical directions. These feature maps are represented as and Subsequently, the text area sensitive feature map is fused with the feature maps in two directions through a 1×1 convolution kernel, and the channel dimension is reduced by the convolution operation. The Sigmund activation function is used to obtain a feature map f, which represents the fusion of the text area sensitive features in two directions.

[0099] f=δ(F1([z h ,z w ]))

[0100] To further enforce text region-sensitive features, a TA threshold activation operation is used to enhance important text region features. 0.1(f) represents pixel activation. The reduced C / r × 1 × (W + H) feature map retains pixel values ​​greater than a threshold of 0.1. r represents the scaling factor for reducing the number of channels. The threshold activation operation enhances and preserves important information in the text region. Finally, the activated feature map t is added pixel by pixel to the pre-activation feature map, reweighting the text region feature map. This preserves important shallow feature information and strengthens the representation of the text region. The default value of the TA threshold activation operation is set to 0.1 in the experiment.

[0101] t=TA 0.1 (f)+f

[0102] Next, for the feature map t, we get the intermediate feature map that encodes the spatial information along the X-axis and Y-axis, and get the attention weight g of the X-axis respectively. h and the attention weight g on the Y axis w .

[0103] g h =σ(F h (t h ))

[0104] g w =σ(F w (t w ))

[0105] In addition, the original input C×W×H shallow feature map is recorded as x c (i, j), and obtain the reweighted feature map. The expression representing the reweighted output feature map is denoted as r c (i,j), can be written as:

[0106] To ensure the robustness of the reweighted feature maps, the features are fed into spatial attention and channel attention, and their outputs are summed. Channel attention effectively enhances contextual information in the channel dimension, improving text detection and recognition performance. Finally, the sum of the outputs of the dual attention mechanisms is fused to obtain the final features. This then completes the processing of the text attention head TAH module, whose output becomes part of the input of the Feature Pyramid Enhanced Fusion Module (FPEFM). The text attention head TAH module is connected to the model backbone via a residual structure, which facilitates adaptive feature extraction and prevents degradation of feature extraction capabilities.

[0107] Function of the Text Attention Head (TAH): The main function of the TAH is to adaptively extract local features and global text location information from text images. As shown in Figure 1, assuming the input feature size of the TAH module is CxHxW, the feature map is average pooled along the X and Y axes. The pooled features are then concatenated and convolved to reduce the channel dimension. The final channel size is C / r×1×(W+H), where r is a channel reduction factor. Subsequently, threshold activation (TA) is applied, which effectively captures sensitive information in the feature map, namely, local text features or global text location information extracted along the X and Y axes of the input feature map. The activated features are element-wise added to the feature map before activation, followed by normalization and nonlinear operations along the X and Y axes, convolution to increase the channel dimension, and finally a sigmoid activation function. The extracted features are then fused with the original input feature map to produce a reweighted fused feature map.

[0108] The reweighted feature maps are further enhanced with dual attention to ensure robustness. Position attention continuously captures the positional information of text of any shape, while channel attention effectively enhances contextual information in the channel dimension, improving text detection and recognition performance. Finally, the outputs of the dual attention mechanisms are combined to obtain the final features. The text attention head (TAH) module is connected to the model backbone via a residual structure, which facilitates adaptive feature extraction and prevents degradation of feature extraction capabilities.

[0109] Step 3: The 128-channel feature map of 4 resolution sizes obtained in the first step and the local feature information of the text image and the global text position information extracted in the second step are taken as input and sent to the feature enhancement fusion module (FPEFM).

[0110] The 128 feature maps from the four shallow layers undergo the self-enhancement phase of the Feature Enhancement and Fusion Module (FPEFM). In the first phase, the input feature maps are scaled to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image pixels, respectively, to form a feature pyramid structure. Iterative enhancement is then performed from a 1 / 4 pixel step size to a 1 / 32 pixel step size. Iterative enhancement continues from a 1 / 32 pixel step size to a 1 / 4 pixel step size. Finally, the enhanced feature pyramid points are sequentially added together to form the output feature pyramid.

[0111] In the second stage, the fusion stage, the output of the shallow feature map self-enhancement pyramid is fused with the local features of the text image and the global text position information extracted by the text attention head TAH. The corresponding pixels are concat together and feature fusion is achieved through 1x1 convolution. The fused feature map maintains the same number of input feature channels. The output feature map of the enhanced feature pyramid enhancement fusion module becomes part of the input of the next round of feature pyramid enhancement fusion module. The other part of the input comes from the local features of the text and the global text position information that the text attention head TAH module continues to extract. The text attention head TAH module extracts features from the backbone network and fuses them using a residual design structure, allowing the feature pyramid enhancement fusion module to adaptively fuse the required features, thereby enhancing the representation ability and depth of the model.

[0112] The role of Feature Pyramid Enhanced Fusion (FPEFM): The Feature Pyramid Enhanced Fusion module is stackable. Stacking experiments were conducted on the ICDAR2015 and Total-Text datasets, with the short side set to 640 pixels for TotalText and 896 pixels for ICDAR2015. On Total-Text, a two-layer stack achieved a maximum precision of 88.8%, a recall of 81.2%, and an F-measure of 84.8, with an inference speed of 60.9 FPS. On ICDAR2015, a two-layer stack achieved a maximum recall of 78.4% and an F-measure of 83.2, with a precision only 0.1% lower than that of a single-layer stack of the Feature Pyramid Enhanced Fusion module, and an inference speed of 30.0 FPS. Comprehensive analysis of the experimental results shows that setting the Feature Pyramid Enhanced Fusion module to two layers achieves a good balance between text detection and recognition accuracy and real-time speed.

[0113] Step 4: The shallow feature maps of the four resolutions are fused through two layers of stacked, residual-structured text attention heads and feature pyramid enhancement to obtain the final 512 enhanced feature maps (shown in Figure 2F). The 512 enhanced feature maps are uniformly upsampled to a quarter of the resolution (shown in Figure 2G). The 512 enhanced feature maps enter the post-processing of end-to-end text detection and recognition.

[0114] In the post-processing stage, the enhanced feature map predicts three components: detection head, detail head, and recognition head.

[0115] The detection head is mainly responsible for predicting the text location. The detection head predicts the text core, text region, and text vector (shown as H, I, and J in Figure 2), and finally predicts the text line through pixel aggregation. The detection head predicts the text region through three loss functions, so the loss of text detection is expressed as Among them, Lregion represents the loss of text region segmentation, and Lkernel represents the loss of text kernel. Use hyperparameters α and β to balance and The weights of , are set to 0.6 and 0.25 in all experiments. Dice loss is used to supervise the text region segmentation result Pregion, as well as the point-wise element addition of the text kernel and text activation map Pkernel.

[0116] therefore, and The mathematical expression is as follows:

[0117] In these equations, P region and G region (i) represents the predicted value and true value of the text area of ​​the i-th pixel in the segmentation result. Similarly, P kernel and G kernel (i) represents the predicted value of the text kernel of the i-th pixel and the true value of the text kernel. Aggregation loss Used to minimize the distance between pixels in the same text line and between text kernels.

[0118] N represents the number of text lines. i represents the text area of ​​the i-th text line, K i Represents the text kernel of the i-th text line. D1(p,Ki) represents the relationship between text pixel p and text kernel K i In order to distinguish different text kernels of text pixels and non-text areas, the following equation can be used:

[0119] D b (Ki) represents the text kernel K i The distance between the kernel and the background. D2(Ki,Kj) represents the distance between different text kernels.

[0120] The detail head uses the text detail map as a supervisory signal to predict text texture and boundaries, as shown by M in Figure 2).

[0121] The detail head predicts the text detail map, which is also the core idea of ​​this paper.

[0122] The first step of the detail head is: text detail truth map processing.

[0123] In the DiZNet model, the original image input in the first step is pre-processed to produce a ground-truth text detail map. The DiZNet model uses this ground-truth text detail map (text texture and boundaries) as the model's supervisory signal. Figure 5 shows the method for extracting the text detail map label. First, the original image is cropped based on the labeled text location labels. The image contains only the text area, and the pixel values ​​of all other non-text areas are filled with zeros. This is the cropped text area. OpenCV's Canny edge extraction algorithm is then used to extract the text texture and boundaries from the cropped image. The extracted text texture and boundary map serves as the ground-truth text detail map.

[0124] The processing process of two sets of text detail truth maps is demonstrated: the ICDAR2015 dataset of multi-oriented straight text and the TotalText dataset of arbitrarily shaped curved text.

[0125] The second step of the detail head is: prediction of text detail map.

[0126] In the DiZNet model, detail map prediction primarily involves predicting the text texture and boundaries of the input image based on the text detail map labels. As shown in Figure 6, the original text image is first cropped and edge extracted to obtain a ground-truth map of the text texture and boundaries, which serves as the label. After following the steps in the text detail map processing, white pixels represent text information, while black pixels represent non-text areas.

[0127] In the first step, the original image is sent to the ResNet18 backbone network for feature extraction, in the second step, the text attention head extracts the local features of the text image and the global text position information, and in the third step, feature enhancement and fusion are performed to obtain an enhanced feature map of 512 channels.

[0128] The specific steps for predicting text detail maps are as follows: through CBR operation, which is a collection of 3x3 convolution, normalization, and ReLU activation function operations, the enhanced feature map is further divided into two small steps.

[0129] 1) The first step is to predict the minimum description area of ​​the text through the text header under the optimization of the text fire kernel loss.

[0130] 2) Another step is to obtain a 6-channel feature map through 1x1 convolution fusion feature, and predict a 1-channel text detail map through the detail head. The detail head is composed of three CBR operations and finally through the Sigmod activation function. The obtained detail map pays more attention to the text position information and texture information. The obtained text kernel is subjected to 1x1 convolution, normalization, ReLU activation function, and finally threshold activation, retaining the 1-channel feature map of the pixel area with a pixel value greater than 0.1. The two 1-channel feature maps are added point by point to fuse the text kernel and text details. Under the supervision of the detail truth map label, the detail loss function is used to simultaneously optimize the text kernel and text detail map. With the fusion of text detail maps, the text detection capability continues to improve.

[0131] Text detail map loss L detail_loss It optimizes the text detail map and text kernel simultaneously based on the text detail truth map supervision label. The calculation expression is as follows:

[0132] L detail_loss ={l1,,,,,l N},l n

[0133] =-[y n ·log(σ(x n ))+1-y n ·log(1- ( σx n )]

[0134] The predicted text detail map and predicted text kernel are flattened into a one-dimensional vector. Then, using the ground-truth text detail map as a supervision label, per-pixel loss is calculated using BCEWithLogitsLoss. σ represents the Sigmoid activation function. Finally, the flattened one-dimensional vector is restored to the size of the input image. Under the supervision of the ground-truth text detail map, the predicted text detail map is fused with the detection and recognition heads to improve end-to-end text detection and recognition performance.

[0135] The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result, as shown in Figure 2.

[0136] Loss function for text recognition as follows:

[0137] Where w is the truth record (text content) containing the EOS (End of Sentence) symbol. |w| represents the number of characters. |w|i represents the i-th character.

[0138] The total loss of the model is:

[0139] Text detection loss Text recognition loss Text detail map loss sum.

[0140] Describe the effects of this solution in comparison with the background technology.

[0141] Effectiveness of the Text Attention Head: Four shallow feature maps of different resolutions (1 / 4, 1 / 8, 1 / 16, and 1 / 32) are generated from the conv2, conv3, conv4, and conv5 layers of the backbone network. The shallow feature maps are fed into the text attention head (TAH) module to extract local text features and global position information. The extracted features are then fed into the feature pyramid enhancement fusion module to fuse the shallow feature maps with the text information. The feature mapping of the text attention head (TAH) module for feature maps of different resolutions is visualized. It can be seen that the 1 / 4 feature map focuses on local information and continuously extracts image information, while the 1 / 8 feature map focuses more on features required by the model and begins to focus on text areas. The 1 / 16 feature map pays more attention to text position information, while the 1 / 32 feature map pays more attention to the most important global information and only focuses on global text position information, as shown in Figure 7.

[0142] This effect corresponds to the problem in the background art.

[0143] Curved Text Detection: Experiments were conducted on the challenging Total-Text dataset, which features arbitrarily curved text. Our method achieved 91.2% precision, 85.9% F-measure, and 38.4 FPS on the Total-Text dataset when the short-side pixel size was 736. Compared to the PSND method, our method improved precision by 3.9% and F-measure by 1.6%. When the short-side pixel size was 640, it achieved 60.6 FPS. Our method demonstrates highly effective detection performance on curved text, as shown in Figure 8.

[0144] Multi-Oriented Straight Text Detection: We further evaluated our model on the ICDAR2015 dataset. When the short side pixel size is 896, our model improves precision by 1.6% and F-measure by 0.6% compared to PSND, while achieving a respectable detection speed of 30.0 FPS. While our method is slightly slower than DBNet in text detection on the ICDAR2015 dataset, it still achieves comparable performance, as shown in Figure 9.

[0145] Figure 10 shows the comparison experiment of Total-Text and IC15 dataset detection.

[0146] Figure 11 shows the visualization results of end-to-end detection and recognition on the Total-Text dataset;

[0147] Figure 12 shows the visualization results of end-to-end detection and recognition on the ICDAR15 dataset;

[0148] Figure 13 shows the end-to-end recognition results of the Total-Text dataset;

[0149] Figure 14 shows the end-to-end recognition results of the ICDAR15 dataset.

[0150] Curved text recognition: In the end-to-end text localization results on the Total-Text dataset, when the short side is 736 pixels, the end-to-end text recognition F-measure is 77.7% and 18.9 FPS when using a dictionary. This is 0.2% higher than PAN++ using joint training with larger data (77.7% vs. 77.5%). It is worth noting that the method achieved the best end-to-end text recognition Fmeasure of 67.1% without using a dictionary, which is 0.4% higher than PAN++. The highest recognition speed is 26.1 FPS, which is higher than all methods. The results show that this method achieves good performance in text recognition tasks, especially in inference speed.

[0151] Multi-directional straight text recognition: In the end-to-end text detection results on the IC15 dataset. The F-measure with common vocabulary (labeled as 'G') is the most important metric. When the short side is 896 pixels, the method achieves a 'G' F-measure of 66.9% at a recognition speed of 30.0FPS, surpassing other algorithms, including PAN++ (66.9% vs. 66.4%), an improvement of 0.5%. It is 3.4% higher than the jointly trained MaskTextSpotterV2 (66.9% vs. 63.5%). When the short side is 736 pixels, the model's FPS can reach a higher 19.4, while the 'G' F-measure still performs well. These results show that in addition to having excellent detection performance on arbitrary shape, multi-directional text, the DiZNet model also has good recognition capabilities and achieves a good balance between accuracy and real-time performance.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A supervised approach for end-to-end text detection and recognition based on a text detail graph, characterized by: The method comprises the following steps: Given an input image containing text of arbitrary shape, the input image is processed by two independent processing branches; The first processing branch extracts text texture and boundary information from the input image and generates a text detail map; The second processing branch uses deep learning neural networks to learn and infer the location and content of text in different contexts; Under the supervision of text detail maps as new labels, we propose an end-to-end natural scene text detection and recognition model DiZNet; In the second processing branch, during the deep learning branch processing, a lightweight feature extraction network ResNet18 is used as the backbone network of DiZNet; the backbone network extracts shallow feature maps with different resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 from the input image, corresponding to the conv2, conv3, conv4, and conv5 layers respectively; the shallow feature maps are processed using 1x1 convolution, and the number of channels is unified to 128; Design a text attention head (TAH) and a feature pyramid enhancement fusion module (FPEFM). FPEFM performs feature self-enhancement at different scales, integrating local text image features extracted by the TAH module with global text position information, and integrating features extracted by the TAH from feature maps of different scales. By stacking multiple FPEFMs, the model's feature representation capability and depth are continuously enhanced. The feature map is upsampled to a uniform size to obtain the final enhanced feature map. The enhanced feature map predicts three components: The detection head is responsible for predicting the text location; the detection head predicts the text kernel, text region and text vector, and predicts the text line by pixel aggregation; The detail head uses the text detail map as a supervisory signal to predict text texture and boundaries; The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result; During the model inference stage, there is no need to use the ground-truth text detail map as a supervision label; instead, the model predicts text texture and boundaries, and then fuses the predicted text detail map into the detection head to infer text location; the predicted text detail map is fused into the recognition head to identify text content.

2. The supervised method for end-to-end text detection and recognition based on a text detail graph according to claim 1, characterized in that: In the second processing branch, during the deep learning branch processing, a lightweight feature extraction network ResNet18 is used as the backbone network of DiZNet; the backbone network extracts shallow feature maps with different resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 from the input image, corresponding to the conv2, conv3, conv4, and conv5 layers, respectively; 1x1 convolution is used to process the shallow feature maps, and the number of channels is unified to 128; a text attention head TAH and a feature pyramid enhancement fusion module FPEFM are designed; FPEFM performs feature self-enhancement at different scales, fuses the local features of the text image extracted by the text attention head TAH module and the global text position information, and fuses the features extracted by the text attention head TAH from feature maps of different sizes; by stacking multiple FPEFMs, the feature representation capability and depth of the model are continuously enhanced; the feature map is upsampled to a uniform size to obtain the final enhanced feature map, specifically: Step 1: Input the original image into the ResNet18 feature extraction backbone network. The ResNet18 feature extraction backbone network will extract features from the original image and obtain feature maps of 128 channels with four resolution sizes. The obtained shallow feature maps have resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively. Step 2: The 128-channel feature map of 4 resolutions obtained in the first step is used as input and sent to the text attention head TAH. Each 128-channel feature map will be sent to the text attention head TAH for calculation, extracting the information of the shallow feature map, and obtaining the local feature information of the text image and the global text position information. The mathematical calculation process of the shallow feature map is as follows. Assume that the number of input channels is C, C is set to 128 in this experiment, W is the width of the feature map, H is the height of the feature map, and the input shallow feature map is expressed as C×W×H; use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the horizontal X-axis and vertical Y-axis coordinates; the vertical direction expression is written as: The horizontal expression is written as: The shallow feature map is encoded along the horizontal and vertical directions and then aggregated to obtain feature maps that are sensitive to text areas in the horizontal and vertical directions; these feature maps are represented as and Subsequently, the text region sensitive feature map is fused with the feature maps of the two directions through a 1×1 convolution kernel, and the channel dimension is reduced by the convolution operation. The Sigmund activation function is then used to obtain a feature map f, which represents the fusion of the text region sensitive features in the two directions. f=δ(F1([z h ,z w ])) To enforce text region-sensitive features, TA threshold activation operation is used to enhance important text region features; TA 0.1 (f represents pixel activation, and the pixel values ​​greater than the threshold of 0.1 are retained in the reduced C / r×1×(W+H) feature map. r represents the scaling factor for reducing the number of channels. The threshold activation operation enhances and retains the important information of the text area. Finally, the activated feature map t is added to the feature map before activation pixel by pixel, and the feature map of the text area is re-weighted to retain important shallow feature information and strengthen the representation of the text area. The default value of the TA threshold activation operation was set to 0.1 in the experiment; t=TA 0.1 (f)+f For the feature map t, we get the intermediate feature map that encodes the spatial information along the X-axis and Y-axis, and get the attention weight g of the X-axis respectively. h and the attention weight g on the Y axis w ; g h =σ(F h (t h )) g w =σ(F w (t w )) The original input C×W×H shallow feature map is recorded as x c (i, j), obtain the reweighted feature map; the expression of the reweighted output feature map is denoted as r c (i,j), written as: To ensure the robustness of the re-weighted feature map, the features are fed into spatial attention and channel attention, and their outputs are summed. Channel attention effectively enhances contextual information in the channel dimension, improving text detection and recognition performance. The sum of the outputs of the dual attention mechanisms is fused to obtain the final features. This then completes the processing of the text attention head TAH module, and the output of the text attention head TAH module becomes part of the input of the feature pyramid enhancement fusion module FPEFM. The text attention head TAH module is connected to the model backbone through a residual structure, which is conducive to adaptive feature extraction and prevents the degradation of feature extraction capabilities. Step 3: The 128-channel feature map of 4 resolutions obtained in the first step and the local feature information of the text image and the global text position information extracted in the second step are fed into the feature enhancement and fusion module FPEFM. The 128 feature maps of the four shallow layers are subjected to the self-enhancement stage of the feature enhancement and fusion module (FPEFM). In the first stage of the self-enhancement stage, the input feature map sizes of the feature enhancement and fusion module are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image pixels, respectively, forming a feature pyramid structure. Iterative enhancement from 1 / 4 pixel step to 1 / 32 pixel step; Continue iterative enhancement from 1 / 32 pixel step size to 1 / 4 pixel step size; add the enhanced feature pyramid point elements to the enhanced feature pyramid point elements in sequence to obtain the output feature pyramid; In the second stage, the fusion stage, the output of the shallow feature map self-enhancement pyramid is fused with the local features of the text image and the global text position information extracted by the text attention head TAH; the corresponding pixels are concat together, and feature fusion is achieved through 1x1 convolution. The feature map after fusion maintains the same number of input feature channels; the output feature map of the enhanced feature pyramid enhancement fusion module becomes part of the input of the next round of feature pyramid enhancement fusion module; the other part of the input comes from the local features of the text and the global text position information that the text attention head TAH module continues to extract; the text attention head TAH module extracts features from the backbone network and uses a residual design structure for fusion, allowing the feature pyramid enhancement fusion module to adaptively fuse the required features, thereby enhancing the representation ability and depth of the model; Step 4: The shallow feature maps of the four resolutions are fused through two layers of stacked, residual-structured text attention heads and feature pyramid enhancement to obtain the final 512 enhanced feature maps. The 512 enhanced feature maps are uniformly upsampled to a quarter of the resolution and enter the post-processing of end-to-end text detection and recognition. In the post-processing stage, the enhanced feature map is used to predict three components: detection head, detail head, and recognition head; The detection head is responsible for predicting the text location; the detection head predicts the text core, text area and text vector, and finally predicts the text line through pixel aggregation; the detection head predicts the text area through three loss functions, and the loss of text detection is expressed as in, represents the loss of text region segmentation, Represents the loss of the text kernel; hyperparameters α and β are used to balance and The weights of are set to 0.6 and 0.25 in all experiments respectively; Dice loss is used to supervise the text region segmentation result Pregion, as well as the point-wise element addition of text kernel and text activation map Pkernel; and The mathematical expression is as follows: P region and G region (i) represents the predicted value and true value of the text area of ​​the i-th pixel in the segmentation result; similarly, P kernel and G kernel (i) represents the predicted value of the text kernel of the i-th pixel and the true value of the text kernel; aggregation loss Used to minimize the distance between pixels in the same text line and between text kernels; N represents the number of text lines; T i represents the text area of ​​the i-th text line, K i Represents the text kernel of the i-th text line; D1(p,Ki) represents the text pixel p and the text kernel K i To distinguish different text kernels and non-text areas of text pixels, the following equation is used: D b (Ki) represents the text kernel K i The distance between the kernel and the background; D2(Ki,Kj) represents the distance between different text kernels; The detail head utilizes the text detail map as a supervisory signal to predict text texture and boundaries.

3. The supervised method for end-to-end text detection and recognition based on a text detail graph according to claim 1, characterized in that: The detail head uses the text detail map as a supervisory signal to predict text texture and boundaries, including the following steps: Text detail truth map processing; In the DiZNet model, the original image input in the first step is processed in advance to produce a ground-truth text detail map. The DiZNet model uses this ground-truth text detail map as the model's supervisory signal. First, the original image is cropped based on the labeled text location labels, cropping only the text area and filling the pixel values ​​of other non-text areas with zeros. This is the cropped text area. OpenCV's Canny edge extraction algorithm is then used to extract the text texture and boundaries from the cropped image. The extracted text texture and boundary maps are used as the ground-truth text detail maps. The processing of two sets of ground-truth text detail maps is demonstrated: the ICDAR2015 dataset for multi-directional straight text and the Total-Text dataset for arbitrarily shaped curved text. Prediction of text detail graph: In the DiZNet model, the detail map prediction process mainly involves predicting the texture and boundaries of the text in the input image based on the text detail map label. First, the original text image is cropped and edge extracted to obtain the ground truth map of the text texture and boundary as the label. After processing according to the steps of the text detail map processing process, white pixels are text information; black pixels are non-text areas. In the first step, the original image is sent to the ResNet18 backbone network for feature extraction. In the second step, the text attention head extracts the local features of the text image and the global text position information. In the third step, feature enhancement and fusion are performed to obtain an enhanced feature map with 512 channels. Through the CBR operation, which is a collection of 3x3 convolution, normalization, and ReLU activation function operations, the enhanced feature map is further divided into two small steps; 1) Under the optimization of text fire kernel loss, the minimum description area of ​​the text is predicted through the text header; 2) A 6-channel feature map is obtained through 1x1 convolution fusion feature, and a 1-channel text detail map is predicted through the detail head. The detail head is composed of three CBR operations and finally a Sigmund activation function. The resulting detail map pays more attention to the text position information and texture information. The obtained text kernel is subjected to 1x1 convolution, normalization, ReLU activation function, and finally threshold activation. The 1-channel feature map of the pixel area with a pixel value greater than 0.1 is retained. The two 1-channel feature maps are added point by point to fuse the text kernel and text details. Under the supervision of the detail ground truth map label, the detail loss function is used to simultaneously optimize the text kernel and text detail map. The algorithm prediction process of the text detail map. With the fusion of text detail maps, the text detection capability continues to improve. Text detail map loss L detail_loss It optimizes the text detail map and text kernel simultaneously based on the text detail truth map supervision label. The calculation expression is as follows: The predicted text detail map and the predicted text kernel are flattened into a one-dimensional vector. Then, BCEWithLogitsLoss is used to calculate the loss of each pixel under the supervision of the text detail ground truth map. σ represents the Sigmoid activation function. Finally, the flattened one-dimensional vector is restored to the size of the input image. Under the supervision of the text detail ground truth map, the predicted text detail map is fused with the detection head and recognition head to improve the performance of end-to-end text detection and recognition. The recognition head recognizes the text content by integrating the text detail graph and obtains the recognition result; Loss function for text recognition as follows: Where w is the truth record containing the end-of-sentence symbol (EOS); |w| represents the number of characters; |w|i represents the i-th character; The total loss of the model is: Text Detection Loss Text recognition loss Text detail map loss sum.

Images