A scene text recognition method based on a Mamba framework fusing multiplexing Transformers
By integrating and reusing the Transformer-based Mamba framework, this method solves the balance problem between local detail modeling and global context utilization in existing scene text recognition methods, achieving efficient scene text recognition, improving recognition accuracy and robustness, and reducing computational complexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-03-25
- Publication Date
- 2026-06-09
AI Technical Summary
Existing scene text recognition methods based on convolutional neural networks, visual Transformers, and state-space models struggle to achieve an ideal balance between local detail modeling capabilities, utilization of bidirectional contextual information, and computational efficiency, resulting in limited recognition accuracy and efficiency in complex scenes.
We adopt a method based on the Mamba framework that integrates and reuses Transformer, constructs diverse training samples through data augmentation strategies, and performs feature extraction by combining a hierarchical feature encoder and an NA-Mamba feature fusion module. We enhance the local structure modeling capability by utilizing local bidirectional feature fusion and neighborhood attention branch, while introducing a reused Transformer module for global context modeling and combining it with a two-stream attention decoder to generate character sequences.
It improves the accuracy and robustness of scene text recognition, enhances the ability to model fine-grained character structure and global context information, reduces computational complexity, and improves parameter efficiency and inference efficiency.
Smart Images
Figure CN121921763B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and pattern recognition technology, and in particular to a scene text recognition method based on the Mamba framework that integrates and reuses Transformer. Background Technology
[0002] Scene text recognition (STR) aims to transcribe text content in natural scene images into corresponding character sequences. It is an important research direction in computer vision and pattern recognition, and is widely used in scenarios such as image information extraction, intelligent transportation, and human-computer interaction. However, text in real-world scenes often features diverse fonts, large scale variations, irregular arrangement, and complex backgrounds, which often leads to blurred local details of characters or missing contextual information, posing a significant challenge to scene text recognition.
[0003] Early scene text recognition methods primarily relied on convolutional neural networks for feature extraction, combined with sequence modeling modules for character prediction. While these methods achieved some progress in regular text recognition, their ability to explicitly model long-range dependencies between characters was limited due to the local receptive field bias inherent in convolutional operations. This resulted in performance bottlenecks in irregular text or scenarios with strong contextual relevance. To enhance global contextual modeling capabilities, researchers began introducing the Transformer architecture into scene text recognition tasks. However, the computational complexity of the self-attention mechanism increases quadratically with sequence length, often requiring the processing of long visual feature sequences in scene text recognition tasks. This put significant pressure on the model's memory usage and inference efficiency, limiting its application in efficiency-sensitive scenarios.
[0004] In recent years, State Space Models (SSMs) have attracted attention due to their ability to model long sequences in linear time complexity. Among them, the Mamba method, by introducing an input-dependent selective state space mechanism, achieves efficient global sequence modeling capabilities, providing a new approach to replace the self-attention mechanism of the Transformer. However, Mamba's original design was primarily geared towards one-dimensional sequence tasks, employing a causal sequential scanning method in its core computation, resulting in unidirectional information flow and an unexplained modeling of two-dimensional spatial structures. When directly applied to scene-based text recognition tasks, it struggles to adequately characterize fine-grained local visual structures such as character strokes and boundaries, and it also fails to effectively utilize bidirectional information during contextual modeling.
[0005] To address the aforementioned issues, researchers have proposed several Mamba variants for visual tasks. These variants are primarily designed for general visual tasks, focusing on directional information propagation or high-level semantic aggregation. However, they have not been specifically optimized for the characteristics of scene text recognition tasks, which emphasize both character-level local structure modeling and bidirectional contextual reasoning. Some methods have also introduced additional structural complexity, weakening the computational efficiency advantage of the state-space model.
[0006] In summary, existing scene text recognition methods based on convolutional neural networks, visual Transformers, and state-space models still struggle to achieve an ideal balance between local detail modeling capabilities, bidirectional contextual information utilization, and computational efficiency. Therefore, how to maintain the advantages of efficient sequence modeling while introducing structural designs more suited to the characteristics of scene text recognition tasks to enhance the joint modeling capabilities of fine-grained character structures and global contextual information remains a pressing technical challenge in this field. Summary of the Invention
[0007] Purpose of the invention: To address the above problems, this invention proposes a scene text recognition method based on the Mamba framework that integrates and reuses Transformer, which improves the recognition accuracy of scene text while also increasing parameter efficiency and inference efficiency.
[0008] Technical solution: A scene text recognition method based on the Mamba framework that integrates and reuses Transformer, including the following steps:
[0009] Step 1: Construct diverse training samples using data augmentation strategies, and input them along with the original samples into a deep neural network for training to generate augmented sample images;
[0010] Step 2: Perform feature encoding on the augmented sample image data, and send the encoding results to the hierarchical feature encoder for feature extraction;
[0011] The hierarchical feature encoder includes multiple cascaded feature encoding stages. Each stage adaptively adjusts the spatial resolution and channel dimension of the features through a downsampling module, and constructs multi-scale feature representations step by step.
[0012] In the hierarchical feature encoding process, the Mamba framework based on the state space model is improved to form the NA-Mamba feature hybrid module, which models the text features of two-dimensional scenes. By introducing the parameter-sharing reused Transformer module, bidirectional global context modeling and iterative enhancement are performed on multi-stage features to extract scene text features.
[0013] Step 3: The scene text features obtained in Step 2 are used as the content stream input and fed into the sequence decoder based on the dual-stream attention mechanism. The character sequence is generated through the cross-attention of the query stream and the content stream, and the final text recognition result is output.
[0014] As a preferred embodiment, step 1 specifically comprises:
[0015] Step 101: The training dataset used in this invention is the Union14M-L scene text recognition dataset, which is stored in LMDB format and divided into several subsets according to the recognition difficulty. Each subset has differences in text clarity, character scale and orientation, degree of deformation, occlusion and background noise complexity, so as to cover a variety of natural scene text imaging conditions from easy to difficult, and improve the generalization ability and robustness of the model in complex scenes.
[0016] Step 102: Using the LMDB-based data reading method, read training sample images from the Union14M scene text dataset and load the corresponding text annotation information; the image data is decoded in RGB format, and empty annotation samples and excessively long character samples are filtered during the loading process to ensure the effectiveness of the training data;
[0017] Step 103: Perform random enhancement operations on the scene text image read in step 102. The random enhancement operations are implemented based on the RandAugment strategy, which includes randomly selecting several enhancement operations from a preset set of enhancement operators and applying them in a random order. The enhancement operators include at least one or more of rotation, translation, blurring and noise perturbation. The rotation operation uses a canvas expansion method to avoid cropping the character content, thereby enhancing the model's adaptability to scenes with rotation, deformation and noise interference.
[0018] Step 104: Adjust the image processed in step 103 to a preset input size and scale it using bicubic interpolation. Then, convert the image into a tensor representation and normalize it to obtain standardized input features for subsequent model training. At the same time, package the processed image and its related metadata together and use it as training samples to input into the deep neural network to achieve effective augmentation of the target samples.
[0019] As a preferred embodiment, step 2 specifically comprises:
[0020] Step 201: Input the augmented sample image generated in Step 1 into the feature encoder for feature extraction; The feature encoder first processes the input image through the image block embedding module, divides the input scene text image into multiple image blocks with fixed spatial size, and maps the image blocks to initial feature representations that maintain spatial correspondence, so as to reduce the computational overhead caused by directly modeling the two-dimensional image in the subsequent feature encoding process.
[0021] Step 202: Input the initial feature representation obtained in step 201 into two feature encoding stages for multi-stage feature encoding. Each feature encoding stage adopts a unified stage processing flow, consisting of a downsampling module and several NA-Mamba feature mixing modules in sequence. The feature encoding stages are connected in sequence from shallow to deep to realize the gradual evolution of features in spatial scale, channel dimension and semantic abstraction level.
[0022] Specifically, in the first In each feature encoding stage, let the input features of that stage be represented as... First, the input features are processed by a downsampling module to obtain intermediate feature representations for each stage. :
[0023] ;
[0024] In the formula, Indicates downsampling; Input the height, width, and number of channels of the feature. This determines the height, width, and number of channels of the output feature.
[0025] Subsequently, the intermediate features Feature modeling is performed using several consecutively stacked NA-Mamba feature fusion modules; let the first... The output of each NA-Mamba feature fusion module is And satisfy the initial conditions Then, each NA-Mamba feature mixing module executes the feature mixing process sequentially according to the following recursive relationship:
[0026] ;
[0027] In the formula, This represents feature mixing. Specifically, the output of the last NA-Mamba feature mixing module... The output features of the i-th feature encoding stage are used as input to the next feature encoding stage or for subsequent global context modeling. After two feature encoding stages, the output features of the first stage are obtained. Second-stage output features .
[0028] Furthermore, the specific downsampling module and NA-Mamba feature mixing module are designed as follows:
[0029] Downsampling module: At the beginning of each feature encoding stage, the input features are first processed by the downsampling module to jointly adjust the spatial resolution and channel capacity of the features so that the feature scale matches the representation requirements of the current feature encoding stage.
[0030] The downsampling module employs an MBConv structure. This structure is built upon depthwise separable convolution, sequentially using extended convolution to increase channel dimensions, depthwise convolution to extract spatial features, and a squeeze-and-excitation (SE) block to recalibrate channels. Finally, pointwise convolution is used for dimensionality projection and information fusion. This achieves spatial scale transformation and channel adjustment while effectively preserving key local structural information such as character strokes and edges. By reducing feature redundancy, it provides a more suitable feature representation for subsequent feature mixing.
[0031] NA-Mamba Feature Hybridization Module: Based on the Mamba architecture of the state-space model, its feature hybridization structure is adaptively improved for scene-based text recognition tasks, specifically including the following processing steps:
[0032] 1) Local bidirectional feature mixing: In the NA-Mamba feature mixing module, local bidirectional feature mixing is performed on the input feature sequence by replacing causal convolution with conventional one-dimensional convolution; let the input sequence be... ,in, The length of the input sequence. Given the number of channels in the input sequence, the local feature mixing process is expressed as:
[0033] ;
[0034] In the formula, Given the input sequence, The length of the one-dimensional convolution kernel. Let be the local radius covered by the convolution kernel on both sides of the current position. For the convolution kernel in relative position The corresponding learnable weight parameters, The index of the current position in the input feature sequence. Position in the input sequence The feature vector at that location, This is the output feature sequence after local bidirectional feature mixing.
[0035] This operation removes the unidirectional information flow restriction imposed by causal convolution, enabling bidirectional information interaction of features within a local range, thereby enhancing the ability to model character strokes, connections, and boundary structures.
[0036] 2) Neighborhood attention branch modeling: In addition to the state space modeling path, a neighborhood attention branch is introduced in parallel to adaptively aggregate features within a local neighborhood; whereby, for the position in the feature sequence... Features of the place X Its neighborhood attention The calculation process is as follows:
[0037] ;
[0038] In the formula, This represents a local neighborhood window centered at position i, used to limit the scope of attention calculation, thereby reducing computational complexity and strengthening the feature associations between local character structures; For position Its location within its neighborhood Normalized attention weights For position The value vector corresponding to the feature at that location. For position The query vector corresponding to the feature. For position The key vector corresponding to the feature at that location. The feature dimensions for query vectors and key vectors.
[0039] 3) Feature fusion and residual connection: The output features of the state space modeling path and the neighborhood attention path are dimensionally aligned using a mapping function and then fused. The fusion result is expressed as follows:
[0040] ;
[0041] In the formula, The linear mapping function that acts on the output of the path modeling the state space. Let be a linear mapping function acting on the output of the neighborhood attention path. For neighborhood attention feature aggregation operation, This is an aggregation operation for neighborhood attention features.
[0042] Meanwhile, the output is added element-wise to the module input features through residual connections, and then normalized to obtain the final output of the NA-Mamba feature mixing module.
[0043] Step 203: Convert the output features from the second feature encoding stage in step 202 into... Input is fed into the reused Transformer module for global context modeling.
[0044] Specifically, the features are first transformed by channel conversion to meet the input requirements of the multiplexed Transformer encoder; then, the aligned features are input into the multiplexed Transformer encoder to perform global context modeling to obtain features and channel conversion back to the original dimension.
[0045] In the Transformer encoding process, the reused Transformer follows the Transformer encoder structure, including a self-attention sub-layer and a feedforward network sub-layer, but modifies the residual connection part of the self-attention sub-layer and incorporates the SE mechanism to enhance feature representation capabilities.
[0046] Let the initial input be The intermediate features output by the self-attention sublayer are represented as Then its intermediate output is:
[0047] ;
[0048] in, This represents a global average pooling operation along the sequence dimension. This represents the Sigmoid activation function. Represents the linear rectified activation function. This indicates a channel-wise multiplication operation. , These are the weight parameters of the first fully connected layer and the weight parameters of the second fully connected layer, respectively, during the channel attention generation process.
[0049] Step 204: The features after global context modeling in step 203 are input into the subsequent feature encoding stage. Further feature modeling is performed through the downsampling module and the NA-Mamba feature mixing module to obtain a deep feature representation with a higher level of semantic abstraction. The deep features are then input again into the reused Transformer encoder with the same parameter set as in step 203 for global context modeling to further enhance the long-range dependencies of the features.
[0050] As a preferred embodiment, step 3 specifically comprises:
[0051] Step 301: Input the scene text features output in Step 2 as the content stream features on the encoder side into the dual-stream decoder; at the same time, construct dual-stream input features for character sequence modeling at the decoding end, including: content stream features (content) for carrying semantic information of generated or target characters, and query stream features (query) for providing location query and sequence structure modeling.
[0052] The content stream features are obtained by superimposing character token embeddings and positional encodings, while the query stream features are generated by learnable positional query vectors or positional encodings, thus forming a decoded input representation based on a dual-stream mechanism.
[0053] Step 302: In each dual-stream decoding layer, pre-normalization is performed on the query stream features and content stream features respectively to improve training stability. Subsequently, self-attention modeling is performed on the dual-stream features in sequence to capture long-range dependencies within the character sequence. Furthermore, cross-attention modeling is performed under the guidance of the content stream features to fuse the visual global context information with the character sequence representation.
[0054] Step 303: Apply a nonlinear transformation of the feedforward network (FFN) to the decoded features after attention modeling, and fuse the attention output and feedforward output through residual connections to enhance representation ability and stabilize gradient propagation; after completing all decoding layer processing, normalize the decoding output to obtain the final sequence feature representation for character classification prediction, and output the character category distribution based on linear mapping and probability normalization to generate the final character sequence recognition result.
[0055] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:
[0056] 1. Regarding target sample augmentation, this invention expands the distribution of training data through diverse data augmentation strategies, thereby improving the model's generalization ability and robustness under different imaging conditions and complex scenarios.
[0057] 2. In terms of feature modeling, this invention significantly enhances the modeling ability of fine-grained features such as character strokes, edges, and local neighborhood structures by integrating the NA-Mamba feature fusion module that incorporates neighborhood attention, while maintaining the advantages of state space models in efficiently modeling long sequences. It also achieves global context fusion between multi-scale features through parameter-sharing reused Transformers, strengthening long-range dependencies and semantic consistency at the character sequence level, while effectively controlling the model parameter scale.
[0058] 3. In terms of feature decoding, this invention adopts a dual-stream attention decoding mechanism to perform sequence modeling of encoded features. By establishing a stable attention interaction relationship between the query stream and the content stream, the cumulative error in the character prediction process is effectively reduced, and the stability and consistency of character sequence generation are enhanced. Attached Figure Description
[0059] Figure 1 This is a flowchart of the scene text recognition method proposed in this invention;
[0060] Figure 2This is a flowchart of the MBConv module processing method using the following sampling method in this invention;
[0061] Figure 3 This is a flowchart of the NA-Mamba mixer processing proposed in this invention;
[0062] Figure 4 This is a flowchart of the reused Transformer module processing proposed in this invention. Detailed Implementation
[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0064] In one embodiment of the present invention, a scene text recognition method based on the Mamba framework that integrates and reuses Transformer is provided, the overall process of which is as follows: Figure 1 As shown, it includes the following steps:
[0065] Step 1: Construct diverse training samples using data augmentation strategies and input them into the deep neural network along with the original samples for training.
[0066] Step 101: The training dataset used in this embodiment is the Union14M-L scene text recognition dataset, which is stored in LMDB format and divided into five subsets according to recognition difficulty: easy, medium, normal, hard, and challenging.
[0067] Each subset differs in terms of text clarity, character scale and orientation, degree of deformation, occlusion, and background noise complexity to cover a variety of natural scene text imaging conditions from easy to difficult, thereby improving the model's generalization ability and robustness in complex scenes.
[0068] Step 102: Using the LMDB-based data reading method, training sample images are read from the Union14M scene text dataset, and the corresponding text annotation information is loaded. The image data is decoded in RGB format, and empty annotation samples and excessively long character samples are filtered during the loading process to ensure the validity of the training data.
[0069] Step 103: Perform random enhancement operation on the scene text image read in step 102. The random enhancement operation is implemented based on the RandAugment strategy, which includes randomly selecting several enhancement operations from a preset set of enhancement operators and applying them in a random order.
[0070] The enhancement operators include at least one or more of rotation, translation, blurring, and noise perturbation. The rotation operation uses a canvas expansion method to avoid cropping of character content, thereby enhancing the model's adaptability to rotation, deformation, and noise interference scenarios.
[0071] Step 104: The image processed in step 103 is uniformly adjusted to the preset input size of 32×128 and scaled using bicubic interpolation. Then, the image is converted into a tensor representation and normalized to obtain standardized input features for subsequent model training. At the same time, the processed image and its related metadata are packaged together and used as training samples to input into the deep neural network to achieve effective augmentation of the target samples.
[0072] Step 2: Perform feature encoding on the sample data processed by the data augmentation strategy in Step 1, and send the encoding result to the hierarchical feature encoder for feature extraction.
[0073] The hierarchical feature encoder consists of multiple cascaded feature encoding stages. Each stage adaptively adjusts the spatial resolution and channel dimension of the features through a downsampling module, thereby constructing multi-scale feature representations step by step.
[0074] In the hierarchical feature encoding process, the Mamba framework based on the state-space model is used as the core feature modeling structure, and a targeted improvement is made to form the NA-Mamba feature hybrid module to achieve effective modeling of text features in two-dimensional scenes. After the partial feature encoding stage, a parameter-sharing reusable Transformer module is introduced to perform bidirectional global context modeling and iterative enhancement of multi-stage features.
[0075] Step 201: Input the augmented sample image generated in step 1 into the feature encoder for feature extraction; The feature encoder first processes the input image through the image block embedding module, divides the input scene text image into multiple image blocks with fixed spatial size, and maps the image blocks to initial feature representations that maintain spatial correspondence, so as to reduce the computational overhead caused by directly modeling the two-dimensional image in the subsequent feature encoding process.
[0076] Step 202: The initial feature representation obtained in step 201 is sequentially input into two feature encoding stages for multi-stage feature encoding. Each feature encoding stage adopts a unified stage processing flow, consisting of a downsampling module and several NA-Mamba feature mixing modules in sequence. The feature encoding stages are connected sequentially in a way that is shallow to deep, so as to realize the gradual evolution of features in terms of spatial scale, channel dimension and semantic abstraction level.
[0077] Specifically, in the first In each feature encoding stage, let the input features of that stage be represented as... First, the input features are processed by the downsampling module to obtain the intermediate feature representation of the stage:
[0078] ;
[0079] In the formula, Indicates downsampling; Input the height, width, and number of channels of the feature. This determines the height, width, and number of channels of the output feature.
[0080] Subsequently, the intermediate features Feature modeling is performed using several consecutively stacked NA-Mamba feature fusion modules; let the first... The output of each NA-Mamba feature fusion module is And satisfy the initial conditions Then, each NA-Mamba feature mixing module executes the feature mixing process sequentially according to the following recursive relationship:
[0081] ;
[0082] in, This represents feature blending, specifically the output of the last NA-Mamba feature blending module. As the output feature of the i-th feature encoding stage, it is used as input to the next feature encoding stage or for subsequent global context modeling processing.
[0083] After two feature encoding stages, the first-stage output features and the second-stage output features are obtained, which can be represented as:
[0084] ;
[0085] in, , This is the output of the first and second stage NA-Mamba feature mixing modules.
[0086] In this embodiment, the downsampling module and the NA-Mamba feature mixing module are designed as follows:
[0087] Downsampling module: At the beginning of each feature encoding stage, the input features are first processed by the downsampling module to jointly adjust the spatial resolution and channel capacity of the features so that the feature scale matches the representation requirements of the current feature encoding stage.
[0088] Among them, the downsampling module adopts Figure 2 The MBConv structure shown is based on depthwise separable convolution. It sequentially increases the channel dimension through extended convolution, extracts spatial features through depthwise convolution, performs channel recalibration through SE blocks, and then performs dimensionality projection and information fusion through pointwise convolution. While realizing spatial scale transformation and channel adjustment, it effectively preserves key local structural information such as character strokes and edges. It reduces feature redundancy and provides a more suitable feature representation for subsequent feature mixing.
[0089] NA-Mamba Feature Mixing Module: Based on the Mamba architecture of the State Space Model (SSM), an NA-Mamba mixer is constructed. Its feature mixing structure is adaptively improved for scene-based text recognition tasks. The processing flow is as follows: Figure 3 As shown, it specifically includes:
[0090] 1) Local bidirectional feature mixing: In the NA-Mamba feature mixing module, local bidirectional feature mixing is performed on the input feature sequence by replacing causal convolution with conventional one-dimensional convolution; let the input sequence be... The local feature mixing process is then expressed as:
[0091] ;
[0092] In the formula, Given the input sequence, The length of the one-dimensional convolution kernel. Let be the local radius covered by the convolution kernel on both sides of the current position. For the convolution kernel in relative position The corresponding learnable weight parameters, The index of the current position in the input feature sequence. Position in the input sequence The feature vector at that location, This is the output feature sequence after local bidirectional feature mixing.
[0093] This operation removes the unidirectional information flow restriction imposed by causal convolution, enabling bidirectional information interaction of features within a local range, thereby enhancing the ability to model character strokes, connections, and boundary structures.
[0094] 2) Neighborhood attention branch modeling: In addition to the state space modeling path, a neighborhood attention branch is introduced in parallel to adaptively aggregate features within the local neighborhood.
[0095] Before the dual-branch processing, the input features of the dual branches are activated by activation functions respectively; at the same time, the features are reshaped before the neighborhood attention branch to adapt to subsequent calculations.
[0096] For the feature at position i in the feature sequence, the neighborhood attention calculation process is as follows:
[0097] ;
[0098] in, This represents a local neighborhood window centered at position i, used to limit the scope of attention calculation, thereby reducing computational complexity and strengthening the feature associations between local character structures; For position Its location within its neighborhood Normalized attention weights For position The value vector corresponding to the feature at that location. For position The query vector corresponding to the feature. For position The key vector corresponding to the feature at that location. The feature dimensions for query vectors and key vectors.
[0099] 3) Feature Fusion and Residual Connection: The output features of the state-space modeling path and the shape-reshaped neighborhood attention path are dimensionally aligned using a mapping function and then fused. The fusion result is expressed as follows:
[0100] ;
[0101] In the formula, The linear mapping function that acts on the output of the path modeling the state space. Let be a linear mapping function acting on the output of the neighborhood attention path. For neighborhood attention feature aggregation operation, Modeling operations for the features of the state-space model.
[0102] Meanwhile, the reshaped and normalized output is added element-wise to the module input features through residual connections, and then normalized again to obtain the final output of the NA-Mamba feature mixing module.
[0103] Step 203: The output features from the second feature encoding stage in step 202 are... Input is fed into the reused Transformer module for global context modeling.
[0104] Specifically, the features are first transformed by channel conversion to meet the input requirements of the multiplexed Transformer encoder; then, the aligned features are input into the multiplexed Transformer encoder to perform global context modeling to obtain features and channel conversion back to the original dimension.
[0105] In the repeated Transformer encoding process, the repeated Transformer follows the Transformer encoder structure, including self-attention sub-layers (linear layers and multi-head attention layers) and feedforward network sub-layers, but modifies the residual connection part of the self-attention sub-layers, incorporating the SE mechanism to enhance feature representation capabilities. The processing flow is as follows: Figure 4 As shown.
[0106] Let the initial input be The intermediate features output by the self-attention sublayer are represented as Then its intermediate output is:
[0107] ;
[0108] in, This represents a global average pooling operation along the sequence dimension. This represents the Sigmoid activation function. Represents the linear rectified activation function. This indicates a channel-wise multiplication operation. , These are the weight parameters of the first fully connected layer and the weight parameters of the second fully connected layer, respectively, during the channel attention generation process.
[0109] Step 204: The features after global context modeling in step 203 are input into the subsequent feature encoding stage. Further feature modeling is performed through the downsampling module and the NA-Mamba feature mixing module to obtain a deep feature representation with a higher level of semantic abstraction. The deep features are then input again into the reused Transformer encoder with the same parameter set as in step 203 for global context modeling to further enhance the long-range dependencies of the features.
[0110] Step 203 and the operation of reusing Transformer in this step can be represented as the same:
[0111] ;
[0112] in, Indicates a set of identical parameters Multiplexed Transformer encoder, and These represent the channel transformations of the input and output, respectively. and express The stage applies to the input and output of the multiplexed Transformer encoder.
[0113] Step 3: The scene text features obtained in Step 2 are used as the content stream input and fed into the sequence decoder based on the dual-stream attention mechanism. The character sequence is generated through the cross-attention of the query stream and the content stream, and the final text recognition result is output.
[0114] Step 301: The scene text features output in Step 2 are input as content stream features on the encoder side to the dual-stream decoder. Simultaneously, dual-stream input features for character sequence modeling are constructed at the decoding end, including: content stream features for carrying semantic information of generated or target characters, and query stream features for providing position query and sequence structure modeling. The content stream features are obtained by superimposing character token embedding and position encoding, and the query stream features are generated by learnable position query vectors or position encodings, thereby forming a decoding input representation based on the dual-stream mechanism.
[0115] Step 302: In each dual-stream decoding layer, pre-normalization processing is performed on the query stream features and content stream features respectively to improve training stability; then, self-attention modeling is performed on the dual-stream features in sequence to capture long-range dependencies within the character sequence, and further cross-attention modeling is performed under the guidance of the content stream features to fuse the visual global context information with the character sequence representation.
[0116] Optionally, only the query stream features are updated in the last decoding layer, without updating the content stream features, to reduce redundant computation and obtain the final sequence representation for character prediction.
[0117] Step 303: Apply a nonlinear transformation of the feedforward network to the decoded features after attention modeling, and fuse the attention output and feedforward output through residual connections to enhance representation ability and stabilize gradient propagation; after completing all decoding layer processing, normalize the decoding output to obtain the final sequence feature representation for character classification prediction, and output the character category distribution based on linear mapping and probability normalization, thereby completing dual-stream decoding to generate the final character sequence recognition result.
[0118] The training phase uses causal masks and real labels for guided training, while the inference phase uses an autoregressive method to generate characters one by one until the output terminator is reached or the maximum length is achieved.
[0119] In summary, the method of this invention constructs diverse training samples through data augmentation strategies during the model training phase. These samples, along with the original samples, are input into a deep neural network for training. Feature encoding is performed on the sample data, and the encoding results are fed into a hierarchical feature encoder for feature extraction. Regarding sample construction, diverse data augmentation strategies expand the distribution of training data, improving the model's generalization ability and robustness under different imaging conditions and complex scenes. In feature modeling, an NA-Mamba feature fusion module incorporating neighborhood attention significantly enhances the modeling ability for fine-grained features such as character strokes, edges, and local neighborhood structures while maintaining the advantages of efficient long-sequence modeling in state-space models. Global context fusion between multi-scale features is achieved through parameter-sharing and reusable Transformers, strengthening long-range dependencies and character sequence-level semantic consistency while effectively controlling the model parameter scale. In feature decoding, a dual-stream attention decoding mechanism is used for sequence modeling of encoded features. By establishing a stable attention interaction relationship between the query stream and the content stream, the accumulated error in the character prediction process is effectively reduced, enhancing the stability and consistency of character sequence generation. This, in turn, improves parameter efficiency and inference efficiency while ensuring high recognition accuracy.
[0120] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A scene text recognition method based on the Mamba framework that integrates and reuses Transformer, characterized in that, Includes the following steps: Step 1: Construct diverse training samples using data augmentation strategies, and input them along with the original samples into a deep neural network for training to generate augmented sample images; Step 2: Perform feature encoding on the augmented sample image and send the encoding result to the hierarchical feature encoder for feature extraction; The hierarchical feature encoder includes multiple cascaded feature encoding stages. Each stage adaptively adjusts the spatial resolution and channel dimension of the features through a downsampling module, and constructs multi-scale feature representations step by step. In the hierarchical feature encoding process, the Mamba framework based on the state space model is improved to form the NA-Mamba feature hybrid module, which models the text features of two-dimensional scenes. By introducing the parameter-sharing reused Transformer module, bidirectional global context modeling and iterative enhancement are performed on multi-stage features to extract scene text features. Step 3: The scene text features obtained in Step 2 are used as the content stream input and fed into the sequence decoder based on the dual-stream attention mechanism. The character sequence is generated through the cross-attention of the query stream and the content stream, and the final text recognition result is output. Step 2 specifically involves: Step 201: Input the augmented sample image generated in Step 1 into the feature encoder for feature extraction; The feature encoder processes the input image through the image block embedding module, dividing the input scene text image into multiple image blocks with fixed spatial sizes, and mapping the image blocks to initial feature representations that maintain spatial correspondence; Step 202: The initial feature representation is sequentially input into two feature encoding stages for multi-stage feature encoding. Each feature encoding stage consists of a downsampling module and several NA-Mamba feature mixing modules in sequence. After the two feature encoding stages, the first stage output features are obtained. Second-stage output features ; Step 203: Output features of the second stage The input is fed into the reusable Transformer module for global context modeling. First, the features are transformed and aligned by channel transformation. Then, the aligned features are input into the reusable Transformer encoder for global context modeling to obtain features and channel transformation to the original dimension. Step 204: Input the features modeled in Step 203 into the subsequent feature encoding stage. Perform feature modeling through the downsampling module and the NA-Mamba feature mixing module to obtain deep feature representations. Then, input the obtained deep features back into the reused Transformer encoder with the same parameter set as in Step 203 to perform global context modeling and enhance the long-range dependencies of the features. The NA-Mamba feature fusion module is built on the Mamba architecture based on the state-space model. It adaptively improves the feature fusion structure for scene-based text recognition tasks, including: Local bidirectional feature mixing is performed in the NA-Mamba feature mixing module. One-dimensional convolution replaces causal convolution to perform local bidirectional feature mixing on the input feature sequence. The process is represented as follows: ； In the formula, Given the input sequence, The length of the one-dimensional convolution kernel. Let be the local radius covered by the convolution kernel on both sides of the current position. For the convolution kernel in relative position The corresponding learnable weight parameters, The index of the current position in the input feature sequence. Position in the input sequence The feature vector at that location, This is the output feature sequence after local bidirectional feature mixing; The neighborhood attention branch is modeled outside the state space modeling path and introduced in parallel to adaptively aggregate features within the local neighborhood; for the position in the feature sequence Features of the place X Neighborhood attention The calculation process is as follows: ； ； In the formula, Indicated by position A local neighborhood window centered on the attention calculation is used to limit the scope of attention calculation; For position Its location within its neighborhood Normalized attention weights For position The value vector corresponding to the feature at that location. For position The query vector corresponding to the feature. For position The key vector corresponding to the feature at that location. The feature dimensions for query vectors and key vectors; This is the normalization function; the superscript T indicates transpose. The output features of the state space modeling path and the neighborhood attention path are dimensionally aligned using a mapping function and then fused. The fusion result is... Represented as: ； In the formula, The linear mapping function that acts on the output of the path modeling the state space. Let be a linear mapping function acting on the output of the neighborhood attention path. For neighborhood attention feature aggregation operation, Modeling operations for the characteristics of the state-space model; The fusion results are obtained through residual connection. With module input features Combining these, we obtain the final output of the NA-Mamba feature hybrid module.
2. The scene text recognition method according to claim 1, characterized in that, Step 1 specifically involves: Step 101: Store the training dataset in LMDB format and divide it into several subsets according to the recognition difficulty. Each subset has differences in text clarity, character scale and orientation, degree of deformation, occlusion, and background noise complexity. Step 102: Using the LMDB-based data reading method, read the training sample images from the training dataset, load the corresponding text annotation information, decode in RGB format, and filter empty annotation samples and excessively long character samples during the loading process; Step 103: Perform random augmentation operations on the read training sample images. Based on the RandAugment strategy, randomly select several augmentation operations from the preset set of augmentation operators and superimpose them in a random order. The augmentation operators include at least one or more of rotation, translation, blurring and noise perturbation. The rotation operation adopts the canvas expansion method. Step 104: Adjust the processed image to the preset input size and perform scaling using bicubic interpolation. The image is converted into a tensor representation and normalized. The processed image and related metadata are then packaged together and used as training samples to input into the deep neural network.
3. The scene text recognition method according to claim 1, characterized in that, The downsampling module processes the input features at the beginning of each feature encoding stage, and jointly adjusts the spatial resolution and channel capacity of the features to match the feature scale with the representation requirements of the current feature encoding stage. The downsampling module adopts the MBConv structure, which is built based on depthwise separable convolution. It sequentially increases the channel dimension through extended convolution, extracts spatial features through depthwise convolution, completes channel recalibration through SE blocks, and then performs dimensional projection and information fusion through pointwise convolution to realize spatial scale transformation and channel adjustment. This effectively preserves key local structural information related to character strokes and edges, and reduces feature redundancy.
4. The scene text recognition method according to claim 1, characterized in that, Step 202 specifically includes: In the In each feature encoding stage, the input features The intermediate features of the stage are obtained through downsampling module processing. : ； In the formula, Indicates downsampling; intermediate features of the stage Feature modeling is performed using several consecutively stacked NA-Mamba feature fusion modules; The output of each NA-Mamba feature mixing module is and satisfy the initial conditions. Then, each NA-Mamba feature mixing module executes the feature mixing process sequentially according to the following recursive relationship: ； In the formula, Indicates feature blending; the output of the last NA-Mamba feature blending module. As the first The output features of each feature encoding stage; After two feature encoding stages, the first stage output features are obtained respectively. Second-stage output features : ； in, , This is the output of the first and second stage NA-Mamba feature mixing modules.
5. The scene text recognition method according to claim 4, characterized in that, The reused Transformer module includes a self-attention sublayer and a feedforward network sublayer. By modifying the residual connection part of the self-attention sublayer and incorporating the SE mechanism, the intermediate features output by the self-attention sublayer are represented as follows: Then its intermediate output for: ； In the formula, This represents a global average pooling operation along the sequence dimension. This represents the Sigmoid activation function. Represents the linear rectified activation function. This indicates a channel-wise multiplication operation. , These are the weight parameters of the first fully connected layer and the weight parameters of the second fully connected layer, respectively, during the channel attention generation process.
6. The scene text recognition method according to claim 5, characterized in that, The multiplexed Transformer encoder operates as follows: ； In the formula, Indicates a set of identical parameters Multiplexed Transformer encoder, and These represent the channel transformations of the input and output, respectively. and express The stage applies to the input and output of the multiplexed Transformer encoder.
7. The scene text recognition method according to claim 1, characterized in that, Step 3 specifically involves: Step 301: The scene text features output in Step 2 are used as the content stream features on the encoder side and input to the dual-stream decoder; at the decoding end, dual-stream input features for character sequence modeling are constructed, including: content stream features for carrying semantic information of generated or target characters, and query stream features for providing position query and sequence structure modeling; Step 302: In each dual-stream decoding layer, after performing pre-normalization processing on the query stream features and content stream features respectively, self-attention modeling is performed on the dual-stream features in sequence to capture the long-range dependencies within the character sequence, and cross-attention modeling is performed under the guidance of the content stream features to fuse the visual global context information with the character sequence representation. Step 303: Apply a nonlinear transformation of the feedforward network to the decoded features after attention modeling, and fuse the attention output and the feedforward output through residual connections; after completing all decoding layer processing, normalize the decoded output to obtain the final sequence feature representation for character classification prediction, and generate the final character sequence recognition result based on linear mapping and probability normalization output character category distribution. The training phase uses causal masks and real labels for guided training, while the inference phase uses an autoregressive method to generate characters one by one until the output terminator is reached or the maximum length is achieved.
8. The scene text recognition method according to claim 7, characterized in that, The content stream features are obtained by superimposing character token embeddings and position encodings, and the query stream features are generated by learnable position query vectors or position encodings. In the final decoding layer, only the query stream features are updated, while the content stream features are not, reducing redundant computation and obtaining the final sequence representation for character prediction.