adaptive feature extraction module afem based on deep learning

Abstract

Natural scene text recognition (STR) is a challenging task due to variations in text style, shape, and background. Although image super-resolution solutions exist, these methods often require high computational resources, which is not suitable for low-power devices. We propose an adaptive feature extraction module (AFEM) that dynamically selects representative image features to improve text recognition. Our "convolutional channel integrator" (CCI) processes global feature information. In the decoder, we use a masking and replacement context learning method that effectively integrates word length information, achieving state-of-the-art performance. Our AFEM outperforms existing methods on mainstream benchmarks and a self-built Uyghur dataset, demonstrating its effectiveness and efficiency.

Description

Technical Field

[0001] This invention, based on deep learning technology, investigates an adaptive feature extraction module (AFEM) for natural scene text recognition (STR), a challenging task due to variations in text style, shape, and background. While image super-resolution solutions exist, these methods typically require high computational resources, making them unsuitable for low-power devices. We propose an adaptive feature extraction module (AFEM) that dynamically selects representative image features to improve text recognition performance. Our Convolutional Channel Integrator (CCI) processes global feature information. In the decoder, masking and substitution context learning methods are employed to effectively integrate word length information, achieving state-of-the-art performance. Our AFEM outperforms existing methods on mainstream benchmarks and a self-constructed Uyghur dataset, demonstrating its effectiveness and efficiency. Background Technology

[0002] Scene text recognition typically falls into two main categories: segmentation-based methods and sequence-based methods. In segmentation-based methods, text characters are segmented or located before recognition, which usually requires training with character-level labeled data. In contrast, sequence-based methods include the CTC method and attention-based encoder-decoder frameworks, both of which are mainstream techniques in scene text recognition. Compared to the CTC method, attention-based encoder-decoder methods are better able to capture the dependencies between generated characters, and can be considered as using an implicit language model. However, attention-based methods usually sacrifice some decoding speed because their recursive decoding process is relatively time-consuming.

[0003] To improve decoding speed while maintaining high recognition accuracy, several methods have been proposed, such as guiding the training of CTC models through attention branches or designing two-stage decoding systems that include relational attention modules. Another approach employs parallel visual attention modules followed by explicit learning of semantic information through a self-attention network. Unlike these methods, this paper proposes an innovative scene text recognition framework that, unlike ViTSTR as an encoder, aims to better model long-range feature relationships in natural scene text, thereby improving decoding speed while maintaining accuracy.

[0004] The Transformer architecture has made significant progress in natural language processing by modeling pairwise interactions between sequential elements. In computer vision, the Transformer has also demonstrated its great potential in various visual tasks in recent years. Some systems directly apply Transformers to image patches for image classification, while others combine convolutional layers and Transformer encoder layers as image feature encoders. DETR and its follow-up research introduced Transformer-based object detectors. Recently, the Swin Transformer, as a Transformer-based network backbone architecture, has performed excellently in visual tasks. However, reducing the high computational cost of Transformer layers remains a challenge. To address this issue, some methods directly replace Transformer units by stacking convolutional layers. Inspired by these methods, this paper proposes the AFEM method, which effectively enhances the entire feature map by dynamically selecting representative feature representations from the input image. Summary of the Invention

[0005] This invention, based on deep learning technology, studies an adaptive feature extraction module (AFEM) based on deep learning. This network consists of a feature extraction encoder and an MP decoder. The encoder includes a feature mixing module (FMM) stack and an upsampling layer, responsible for extracting visual features from the input image and predicting word lengths. The decoder employs an MP decoder, combined with masked language modeling (MLM) and partial language modeling (PLM), to decode the visual representation into the final predicted text, while simultaneously embedding contextual information. This organic combination of MLM and PLM allows the model to effectively learn and utilize contextual information during the decoding process.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Feature Extractor: First, the input low-resolution (LR) scene text image is convolved using a 3 × 3 pixel convolutional kernel, transforming it into a feature space and generating shallow features F0. Subsequently, more detailed deep features are extracted from F0, facilitating high-resolution reconstruction of the scene text image through multiple stacked Feature Modulation Modules (FMMs). Each FMM layer contains an Adaptive Feature Modulation (AFM) sublayer and a Convolutional Channel Integrator (CCI). The restoration of the high-resolution target image of the scene text introduces global residual stitching to capture high-frequency details, while a lightweight upsampling layer is employed for fast reconstruction, containing only 3 × 3 convolutions and sub-pixel convolutions.

[0007] This network can be represented as: F0 = Cₜ(ILR) (1) (2) Here, ISR represents the predicted high-resolution (HR) image, ILR represents the input low-resolution (LR) image, Cₜ(·) represents the first convolution operation, whose parameters are controlled by t, Mₜ(·) represents the stacked FMM modules, whose parameters are controlled by θ, and Uᵧ represents the upsampling function, whose parameters are controlled by γ. Similar to previous studies, these parameters are optimized by combining mean squared absolute error (MAE) loss and FFT-based frequency loss functions, which are defined while preserving the content.

[0008] L = ∥ISR − IHR ∥1 + λ ∥ F(ISR) − F(IHR) ∥1 (3) Compared to self-attention mechanisms or large-kernel convolutions, a lightweight alternative is proposed to capture long-range dependencies from multi-scale feature representations, thereby enhancing the exploration of more valuable features in high-resolution (HR) image reconstruction. As shown in Figure 3, a feature pyramid is used to generate an attention map with spatially adaptive feature modulation. To simplify the model and obtain pyramid feature representations, channel segmentation is first applied to the normalized input features, resulting in four parts. A 3×3 depthwise convolution processes the first part, while the remaining parts undergo multiple convolutions to generate mid-scale features. Given input features X, the process can be represented as: [X0, X1, X2, X3] = Split(X) (4) X̂0 = DW − Conv3ₓ3(X0) (5) X̂ᵢ = ↑ₚ(DW − Conv3ₓ3(↓ p²ᵢ(Xᵢ))), 1 ≤ i ≤ 3 (6) Here, Split(·) is the channel splitting operation, and DW − Conv3ₓ3(·) represents a depthwise convolution with a 3×3 pixel kernel. The operation ↑ₚ(·) involves sampling the original features at a specific level and resizing them to p using nearest-neighbor interpolation for efficient implementation. Similarly, ↓ p²ᵢ represents downsampling the input features to a size of 2ᵖ. 1To select discriminative features to capture nonlocal interactions, adaptive max pooling is used to process the input features, resulting in multi-scale features. Language Decoder: The decoder adopts a similar architecture to the Transformer decoder, particularly the PreLayerNorm configuration, the only difference being the use of twice the number of attention heads, where nhead is set to dmodel / 32. The decoder accepts three inputs: location information, contextual information, and image labels, and can optionally include an attention mask. For brevity, LayerNorm and Dropout operations are omitted in the following equations. The first multi-head self-attention module handles attention to contextual locations.

[0009] hc = p + MHA(p, c, c, m) ∈ R^(T+1)×dmodel (9) Where T represents the length of the context, p ∈ R^(T+1)×dmodel represents the position marker, c ∈ R^(T+1)×dmodel represents the context embedding containing positional information, and m ∈ R^(T+1)×(T+1) represents the optional attention mask. It should be noted that using special delimiters ([B] or [E]) will increase the total sequence length to T+1.

[0010] Location markers are used to encode the target location, with each marker corresponding to a specific location in the output. This parameterization approach is similar to the query stream in two-stream attention. It is used to separate the context from the target location so that the model can draw information from a pre-trained language model (PLM). Without location markers, i.e., if the context markers themselves are used as queries, the model will not be able to effectively learn content from the PLM and will simply operate like a standard autoregressive (AR) model. The provided masks are used differently in different models. During training, the mask is randomly generated, but during the inference phase, different methods can be used, such as the standard left-to-right prior mask (AR decoding), perfectly filled masks (iterative refinement), or even no mask at all (NAR decoding). These masks are used in multi-head self-attention (MHA) for location attention processing in images.

[0011] hi = hc + MHA(hc, z, z) ∈ R^(T+1)×dmodel (10) Without an attention mask, the final hidden state of the decoder, denoted as hdec, is the output of the multilayer perceptron (MLP). Mathematically, hdec = hi + MLP(h) ∈ R^(T+1)×dmodel. The final output logits is represented as y = Linear(hdec) ∈ R^(T+1)×(S+1), where S represents the character set used for training, and S+1 is the size of the character set.

[0012] hdec = hi + MLP(h), y = Linear(hdec) (11) Furthermore, the character set includes the appended character [E] to indicate the end of the sequence. In summary, considering the attention mask m, the functional form of the decoder is: y = Dec(z, p, c, m) ∈ R^(T+1)×(S+1) (12) For a given image x, the goal is to maximize the probability of the text label y = [y1, y2, ..., yᵀ] under the model parameter set θ. In standard autoregressive (AR) modeling, the log-likelihood can be calculated in the prescribed order [1, 2, ..., T]: log p(y|x) = Σ (t=1 to T) log pθ(yt | y <t, x) (13) However, the Transformer model has the ability to process all labels in parallel, thus allowing conditional dependencies between the output label and all input labels. To ensure effective autoregressive modeling, it is necessary to prevent past labels from accessing future labels. This autoregressive property can be enforced using attention masks, similar to the approach in "Transformers". The key idea behind probabilistic language modeling (PLM) is to factorize all T: log p(y|x) = Eₓ Zt Σ (t=1 to T) log pθ(yzt | yz <t, x) (14) In this context, ZT represents the set of all possible permutations [1, 2, ..., T], where zt and z<t denote the t-th element and the first t - 1 elements in the permutation z ∈ ZT, respectively. Each permutation z corresponds to a different factorization order. In practical applications, training all possible T permutations is infeasible because factorization leads to an exponential growth in computational requirements. Therefore, as a compromise, usually only K possible orderings are considered. It is worth emphasizing that our selection adopts a non-uniform sampling strategy, rather than a uniform sampling strategy. Specifically, K / 2 permutations are used, where the first half are the left-to-right permutations [1, 2, ..., T], and the other K / 2 - 1 permutations are randomly sampled. The remaining half consists of the flipped versions of the first half. This sampling strategy has been shown to help promote a more stable training process. For these K permutations and the true label ŷ, the overall training loss is calculated as the average of the cross-entropy losses of all individual attention masks mk: L = (1 / K) Σ (k=1 to K) Lce(yk, ŷ) (15) Here, yk = Dec(z, p, c, mk). Description of the Drawings Figure 1 is a feature modulation mechanism based on multi-scale representations to dynamically select the most representative feature maps (sequence decoding method that directly decodes character sequences from visual features and extracts semantic information); Figure 2 is the overall model diagram of AFEM (the overall architecture of AFEM, mainly composed of an encoder and a decoder); Figure 3 is a display diagram of the sample natural scene English recognition dataset (samples of the natural scene English text recognition dataset); Figure 4 is a display diagram of the sample natural scene Uyghur recognition dataset and its annotation (samples of the natural scene Uyghur text recognition dataset and their annotations); Figure 5 is a display diagram of the sample synthetic Uyghur text dataset (samples of the synthetic Uyghur text dataset); Figure 6 is a visualization of the attention score maps generated by the FAEM method for input images on English and Uyghur datasets (examples of attention scores generated from (a) English dataset and (b) Uyghur dataset); Figure 7 is a diagram showing some test results of the AFM model and the method in this paper (challenging examples where AFM fails, but the performance is improved or successful by integrating the AFEM model).

Claims

1. This paper proposes an innovative deep learning-based adaptive feature extraction module, AFEM, which dynamically selects representative image features, significantly improving the performance of scene text recognition. Our Convolutional Channel Integrator (CCI) handles global feature information, further enhancing the model's recognition capabilities. In the decoder, we employ a masking and substitution context learning strategy to effectively integrate word length information, achieving leading recognition accuracy. AFEM outperforms existing state-of-the-art methods on multiple mainstream benchmarks and our self-built Uyghur dataset, fully demonstrating its superiority in improving recognition performance and computational efficiency.

2. The deep learning-based adaptive feature extraction module AFEM according to claim 1 extracts global feature information through a "convolutional channel integrator" (CCI), thereby effectively integrating the global context of the image. Through this mechanism, the model can recognize text in complex natural scenes, significantly improving recognition capability and accuracy. Simultaneously, the CCI module can capture long-distance dependencies in the image, enhancing the model's robustness to various text deformations and providing strong support for improving scene text recognition performance.

3. The deep learning-based adaptive feature extraction module AFEM according to claim 1, the model consists of a feature extraction encoder and an MP decoder. The encoder includes a feature mixing module (FMM) stack and an upsampling layer, responsible for extracting visual features from the input image and predicting word length. The decoder employs an MP decoder, combining masked language modeling (MLM) and partial language modeling (PLM) methods to decode the visual representation into the final predicted text, and incorporates contextual embeddings. This tightly integrated approach effectively combines MLM and PLM, enabling the model to fully learn and utilize contextual information during the decoding process, thereby improving recognition performance.

4. The deep learning-based adaptive feature extraction module AFEM according to claim 1 first applies a 3×3 pixel kernel-sized convolutional layer to transform the input low-resolution (LR) scene text image into a feature space, generating shallow features F0. Subsequently, a multi-layer stacked feature modulation module (FMM) extracts finer depth features from F0 to reconstruct the scene text image at high resolution. Each FMM layer contains an adaptive feature modulation (AFM) sublayer and a convolutional channel integrator (CCI). To recover the high-resolution target image of the scene text, global residual stitching is introduced to capture high-frequency details, while a lightweight upsampling layer is used for fast reconstruction, containing only 3×3 convolutions and subpixel convolutions.

5. The deep learning-based adaptive feature extraction module AFEM according to claim 1 introduces a lightweight alternative to the computational overhead of self-attention mechanisms or large convolutional kernel methods. This aims to capture long-range dependencies from multi-scale feature representations, thereby enhancing the exploration of more valuable features during high-resolution (HR) image reconstruction. As shown in Figure 3, a feature pyramid is used to generate an attention map for spatially adaptive feature modulation. To simplify the model and obtain the pyramid feature representation, channel segmentation is first applied to the normalized input features, dividing them into four parts. A 3×3 depthwise convolution processes the first part, while the remaining parts undergo multiple convolutions to generate mid-scale features.

6. The deep learning-based adaptive feature extraction module AFEM according to claim 1, the decoder adopts an architecture similar to the Transformer decoder, especially using the preLayerNorm (1) configuration, the difference being that it uses twice the number of attention heads, where n head Set to d model / 32. The decoder accepts three inputs: positional encoding, contextual information, and image labels, and optionally includes an attention mask. For brevity, LayerNorm and Dropout are omitted from the following equations. The first multi-head self-attention module handles attention to the contextual position.

7. In the deep learning-based adaptive feature extraction module AFEM according to claim 1, this study used three different datasets. First, two synthetic datasets were used for model training. Subsequently, two augmented datasets were used to evaluate the model's performance. In addition, six widely used public datasets were incorporated. Finally, a self-constructed Uyghur natural scene dataset was introduced for ablation experiments and compared with existing methods.

8. Based on the deep learning-based adaptive feature extraction module AFEM as described in claim 1, due to the lack of publicly available natural scene Uyghur text recognition datasets, this paper establishes a dataset based on 5095 natural scene images. These images were taken under different weather, lighting, and viewing angle conditions, and include street signs, advertising banners, etc. After labeling and cropping the images, 30990 text images were finally obtained, and divided into training, validation, and test sets in an 8:1:1 ratio.

9. In the deep learning-based adaptive feature extraction module AFEM according to claim 1, given that most text images in natural scenes have irregular shapes, this paper employs a Spatial Transformation Network (STN) as a text correction network to improve the effectiveness of character feature extraction and enhance the model's robustness in recognizing irregular text features. This correction network utilizes predicted TPS (Thin Plate Spline) transformation parameters to correct the input irregular text image into a regular text image. By using TPS as the transformation method, non-rigid deformation processing of the text image can be performed, effectively correcting the shape of the irregular text image. Furthermore, the Spatial Transformation Network can be trained end-to-end with the model to achieve overall optimization, thereby improving the overall performance of the model.

10. The Deep Learning-Based Adaptive Feature Extraction Module (AFEM) as described in claim 1 addresses the critical challenge of enhancing the model's recognition capabilities in natural scene text recognition tasks, particularly in the face of complex environmental changes and irregular text shapes. This paper proposes a Deep Learning-Based Adaptive Feature Extraction Module (AFEM) method, which effectively improves the extraction accuracy of text features by combining global and channel information. The introduced Spatial Transformation Network (STN) and attention mechanism optimize text alignment and contextual understanding, enhancing the model's robustness to deformation and noise. Experimental results demonstrate that this method achieves accuracies of 97.9% and 89.51% on the SVT and IC15 datasets, respectively, significantly improving the performance of natural scene text recognition.

Images