Mongolian handwritten character recognition method based on double-branch feature fusion and transformer sequence recognition

Abstract

The application discloses a Mongolian handwritten character recognition method based on double-branch feature fusion and Transformer sequence recognition, acquires a Mongolian handwritten character image dataset, and carries out multi-level data enhancement and data preprocessing; a Mongolian handwritten character recognition model based on the combination of double-branch feature fusion and Transformer sequence recognition is constructed, global features and detail features of the preprocessed data are respectively extracted by using a global branch and a detail branch, and dynamic weighted fusion is carried out based on sample characteristics by using a spatial attention module; based on the strategies of dynamic position coding, spatial perception sequence conversion and attention bias optimization, Transformer encoding and decoding sequence recognition is carried out according to the fused features. The double-branch feature fusion can improve the Mongolian handwritten character recognition accuracy and efficiency, and provides a systematic solution for relieving the technical bottlenecks of the Mongolian handwritten character recognition, such as the lack of dataset samples, the dense connection of characters and the difficulty in identifying similar characters.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to image and text recognition, and in particular to a method for recognizing Mongolian handwriting based on dual-branch feature fusion and Transformer sequence recognition. Background Technology

[0002] Mongolian handwriting recognition still faces technological bottlenecks due to issues such as complex character shapes, scarce datasets, and poor adaptability of traditional models. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition, so as to solve the above problems and improve the accuracy and efficiency of Mongolian handwriting recognition.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for recognizing Mongolian handwriting based on dual-branch feature fusion and Transformer sequence recognition, characterized by the following steps: Step 1: Obtain the Mongolian handwritten image dataset and perform multi-level data augmentation and data preprocessing; Step 2: Construct a Mongolian handwriting recognition model based on the combination of dual-branch feature fusion and Transformer sequence recognition. Use global branch and detail branch to extract global features and detail features of the preprocessed data respectively, and use spatial attention module to perform dynamic weighted fusion based on sample characteristics. Step 3: Based on the strategy of dynamic position encoding, spatially aware sequence transformation and attention bias optimization, Transformer encoding and decoding sequence recognition is performed according to the fusion features.

[0005] In one embodiment, step 1, multi-level data augmentation includes basic geometric transformation, style enhancement, and custom strategies; The basic geometric transformations employ directional rotation, vertical translation, and scaling to simulate the tilt and positional deviations in natural handwriting. The style enhancement is achieved by contrast perturbation, Gaussian or salt-and-pepper noise injection, and handwriting thickness adjustment to simulate the differences between different writing tools and scanning environments. The customized strategy enhances the adaptability to key morphological variations by using local affine transformations in the connected stroke area, subtle stroke perturbations of similar characters, and vertical edge clipping. When performing data augmentation, an online dynamic augmentation mechanism is adopted. Several transformations are randomly combined in each training iteration, and the parameters are fine-tuned through the validation set. At the same time, the augmented samples are screened to remove excessively distorted samples and retain effective data that can simulate real blur and deformation.

[0006] In one embodiment, the local affine transformation of the connected stroke region is implemented as follows: First, clearly mark the range of the connected stroke area, and generate a mask that covers the connected stroke area based on the standard that there is no obvious gap between characters. Non-connected stroke areas do not participate in the transformation. Then, a local affine transformation is performed on the masked area, including random scaling up and down by one time, to simulate the difference in the tightness of the strokes and the distortion at different angles, and to simulate the natural tilt during writing. The subtle stroke perturbation of similar characters is implemented as follows: We screened easily confused character pairs in Mongolian writing and designed targeted perturbations based on their core distinguishing features, including adjusting the length of vertical strokes, perturbing the angle of hooks or the curvature of arcs, and slightly adjusting the curvature to generate near-realistic yet clearly distinguishable variant samples. The vertical edge trimming is implemented as follows: The image is randomly cropped along its vertical edge, retaining the main characters in the center area, thus covering the real-world scenario where the top of the first character of a line is missing and the bottom of the last character is truncated.

[0007] In one embodiment, the global branch is based on the pure convolutional network ConvNeXt, and the Stages 2-5 of the network are replaced with 7×7 depthwise separable convolutions instead of 3×3 convolutions, covering the complete outline of the vertical strokes in one go and capturing the long-distance dependencies of the vertical strokes. The detail branch is based on the ResNeSt101 residual network with group attention. With the help of its group attention mechanism, the multi-dimensional channel is divided into 4 functional subsets, which focus on the difference in vertical stroke length, the direction of hook bend, the curvature of arc, and the position of additional point, respectively, to dynamically enhance the key difference features of similar characters. The detail branch uses the feature map output by its Stage4 as detail features.

[0008] In one embodiment, the method for dividing the multidimensional channel into four functional subsets is as follows: Subset 1 focuses on the vertical stroke shape, capturing the vertical edge density of short and long vertical strokes. By statistically analyzing the gradient response differences in the vertical direction, the length difference between the two is encoded, with short vertical strokes having weaker responses and long vertical strokes having stronger responses. Subset 2 adapts to hook-shaped bends and arcs, enhances the directional characteristics of non-perpendicular fine strokes, and distinguishes the morphological differences between hook shapes and straight lines; Subset 3 extracts local curvature information and distinguishes between gentle circular arcs (large radius of curvature, uniform response) and sharp hooks (small radius of curvature, abrupt local response) by Gaussian curvature calculation. Subset 4 locates character appendages and uses local extrema detection to mark the spatial position of appendages, helping to distinguish whether there are appendages and their positions.

[0009] In one embodiment, the method for using a spatial attention module to perform dynamic weighted fusion based on sample characteristics is as follows: Global and detailed features are compressed to a consistent channel dimension using 1×1 convolution, ensuring feature dimension alignment. The system generates attention weights α by summing features in two branches, compressing them with 1×1 convolution, global average pooling, and sigmoid activation. When α approaches 1, it enhances detailed features to adapt to similar character samples, and when α approaches 0, it highlights features to adapt to long cursive script samples, ultimately achieving dynamic weighted fusion.

[0010] In one embodiment, the dual-branch feature summation is achieved by adding the global features and detail features after channel compression pixel by pixel; the pixel-by-pixel summation result is then compressed by 1×1 convolution, global average pooling, and Sigmoid activation to generate attention weight α.

[0011] In one embodiment, step 3, reusing backbone network features to generate dynamic location codes, is implemented as follows: Deformation patterns are captured by compressing the spatial dimension of the fused features using 3×3 depthwise separable convolution and global average pooling. The compressed features are mapped to positional encoding vectors through a fully connected layer.

[0012] In one embodiment, step 3, the spatially aware sequence transformation, is implemented as follows: The fusion feature is divided into 3×3 non-overlapping blocks, which are then flattened into a sequence in a vertical order from top to bottom and from left to right, preserving the spatial structure within each block. Attach two-dimensional coordinates to each token, with row numbers as follows: Column number is The encoding is a low-dimensional vector concatenated with the token feature to enhance vertical spatial awareness. The formula is as follows: in, This is a coordinate embedding function that adapts to the top-to-bottom writing order of Mongolian script.

[0013] In one embodiment, step 3, attention bias optimization, is implemented as follows: In self-attention calculation, a line distance penalty is introduced to increase the attention weight in the vertical direction and suppress invalid associations across multiple lines. The strength of the suppression of attention weight by line distance is controlled by the following formula: in, These are the query and key vectors for the i-th and j-th tokens, respectively, where d is the dimension of the query and key vector. Used for scaling to prevent the softmax gradient from vanishing due to an excessively large inner product. Softmax normalizes the weights to [0,1], with a total sum of 1. The penalty coefficient is... This indicates the row index of the i-th and j-th tokens, i.e., which row the token belongs to in the vertical image.

[0014] Compared with existing technologies, this invention quantifies and verifies the effectiveness of multi-level data augmentation schemes, dual-branch feature fusion and Transformer sequence recognition hybrid models through core indicators such as recognition accuracy, generalization performance in small sample scenarios, model inference efficiency and robustness. It clearly demonstrates the advantages of dual-branch feature fusion in improving the accuracy and efficiency of Mongolian handwriting recognition, and provides a systematic solution to alleviate the technical bottlenecks of scarce dataset samples, dense strokes and confusion of similar characters in Mongolian handwriting recognition. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention.

[0016] Figure 2 This is a schematic diagram of ConvNeXt modules.

[0017] Figure 3 This is a schematic diagram of the ResNeSt Block.

[0018] Figure 4 This is a schematic diagram of the Split Attention module.

[0019] Figure 5 This is a schematic diagram of the spatial attention module.

[0020] Figure 6 This is a schematic diagram of the Transformer model structure. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0022] This invention relates to a method for recognizing Mongolian handwriting based on dual-branch feature fusion and Transformer sequence recognition. The main contents include: 1. Expand the Mongolian handwritten script dataset. Given the limitations of Mongolian handwritten image data, such as limited sample size, uniform writing style, and insufficient morphological diversity, this invention aims to expand the dataset and improve the model's generalization ability through a multi-level data augmentation scheme involving basic geometric transformations, style enhancement, and customized strategies.

[0023] Specifically, considering the vertical writing characteristics of Mongolian script, basic geometric transformations such as directional rotation, vertical translation, and scaling are employed to simulate the tilt and positional deviations in natural writing. Contrast perturbation, Gaussian / salt-and-pepper noise injection, and stroke thickness adjustment are used to simulate differences in writing tools and scanning environments. Focusing on the core challenges of dense cursive strokes and confusion between similar characters, customized strategies are designed, including local affine transformations in cursive regions, subtle stroke perturbations of similar characters (such as offsetting the position of additional points and adjusting the hook angle), and vertical edge clipping, to enhance the model's adaptability to key morphological variations. During implementation, an online dynamic enhancement mechanism is used, randomly combining 3-4 transformations in each training iteration. Parameters are fine-tuned using a validation set, and enhanced samples are screened to remove excessively distorted samples, retaining effective data that can simulate realistic blurring and deformation. These methods significantly improve the morphological diversity of the training data, effectively alleviate the problem of overfitting with small samples, lay a data foundation for subsequent dual-branch model training, and help improve the model's recognition accuracy in tilted, noisy, and cursive scenarios, thus enhancing overall robustness.

[0024] 2. Construct a hybrid architecture based on dual-branch feature fusion and Transformer sequence recognition as a Mongolian handwriting recognition model.

[0025] Given the inherent characteristics of Mongolian handwriting, such as dense vertical strokes and the ease with which similar characters can be confused, existing recognition models struggle to simultaneously handle global structural modeling and detailed feature capture, often resulting in a trade-off between accuracy and efficiency in practical applications. To address this issue, this invention constructs a hybrid model combining dual-branch feature fusion and Transformer sequence recognition, using this as the core recognition architecture to achieve performance breakthroughs. On one hand, ConvNeXt and ResNeSt101 are employed for dual-branch feature extraction. The pure convolutional network ConvNeXt captures the long-distance dependence of vertical strokes through large convolutional kernels, and its hierarchical Stage structure adapts to the multi-scale features of Mongolian characters, words, and sentences. The residual network ResNeSt101, with its grouped attention mechanism, leverages Split Attention (channel grouping and attention weighting) to focus on subtle stroke differences, accurately distinguishing similar characters. Both networks dynamically fuse detailed and global features through spatial attention in the attention gating module, forming a joint representation that balances global structure and detailed features. On the other hand, Transformer encoding and decoding are introduced for sequence recognition: the encoder models the temporal dependence of vertical characters to adapt to the top-to-bottom writing order of Mongolian script; the decoder outputs character sequences through Beam Search and combines directional position encoding to enhance adaptability to vertical layout.

[0026] This model alleviates the difficulties of capturing contiguous strokes and distinguishing similar characters by using complementary dual branches. It achieves end-to-end transcription by using Transformer sequence modeling and can still maintain high robustness in tilted, noisy and complex contiguous stroke scenarios, providing an efficient and accurate technical solution for Mongolian handwriting recognition.

[0027] refer to Figure 1 As shown, the implementation of this invention mainly includes the following steps: Step 1: Obtain the Mongolian handwritten image dataset and perform multi-level data augmentation and data preprocessing.

[0028] This step primarily addresses the issue of insufficient existing Mongolian handwritten data. Data augmentation is crucial for improving the generalization ability of Mongolian handwritten recognition models during training. Considering the characteristics of Mongolian handwritten characters, such as numerous vertical strokes and a relatively uniform writing style, this invention expands the data through basic geometric transformations, style enhancement, and customized strategies: basic geometric transformations include directional rotation, vertical translation, and scaling to simulate natural writing variations; style enhancement adapts to different tools and environments through contrast adjustment, noise injection, and variations in stroke thickness; customized strategies focus on core challenges such as deformation in connected stroke areas and fine-tuning of similar characters, strengthening the model's adaptability to key morphological features. During implementation, online dynamic augmentation is used, randomly combining transformation methods and selecting effective samples to avoid excessive distortion. This effectively improves data diversity, alleviates overfitting with small samples, and enhances the model's robustness in complex scenarios.

[0029] Specifically, the implementation method of the three-level framework of basic geometric transformation, style enhancement, and customization strategy for extending the Mongolian dataset in this invention is as follows: (1) Design basic geometric transformations to address the tilt, positional offset and size differences of Mongolian vertical writing.

[0030] This study focuses on the tilting characteristics of vertically written characters through directional rotation. Due to wrist movement habits during Mongolian handwriting, vertically written characters are more prone to tilting vertically. Therefore, random rotation is performed around the vertical axis of the image, i.e., the vertical direction. A suitable angle range can cover real-world scenarios where slight tilting does not affect overall legibility. After rotation, the boundaries are filled with pixels of the same color as the background to ensure the image size maintains its original pixel dimensions. This avoids edge truncation of cursive characters caused by rotation, preserving character integrity while enhancing the model's adaptability to tilting patterns.

[0031] Vertical translation simulates the natural deviation of inline positions. Mongolian script is written from top to bottom, and characters in a vertical layout often shift vertically due to the writing rhythm. To address this, a corresponding pixel-wise random translation is performed along the vertical y-axis. This pixel range accurately reflects the subtle fluctuations in the spacing between adjacent characters in actual writing without disrupting the overall continuity of the vertical text. By enhancing the diversity of vertical positions, the model is guided to de-emphasize absolute position dependence and focus more on the relative layout relationships between characters.

[0032] The scaling method balances differences in character size with the preservation of connected strokes. In Mongolian handwriting, the same character may appear different in size due to varying writing pressure. To address this, characters are randomly scaled up or down by approximately one time, and then the original size is restored through center cropping (for enlarged images) or boundary padding (for reduced images). The key is to ensure that connected strokes are not over-cropped. During scaling, the connected stroke area is used as the center anchor point, prioritizing the preservation of the connection structure between characters and avoiding breaks in the connected strokes due to scaling. This enhances the model's stability in recognizing characters of different sizes but with intact structures.

[0033] (2) Style transformation, simulate different handwriting and scanning noise, and improve the model's adaptability to non-ideal conditions.

[0034] Style enhancement focuses on real-world scenarios involving uneven lighting, scanning noise interference, and differences in handwriting tools used in Mongolian handwriting. It simulates image variations under non-ideal conditions through fine-grained manipulation. For lighting variations, different ratios of contrast adjustment and brightness shift are used to cover both low-light (low contrast) and high-light (high contrast) scenes. Because fine Mongolian strokes are easily damaged by extreme contrast, a range is limited to balance feature recognition and fine stroke preservation. For scanning noise, Gaussian noise is injected with adjusted probabilities to simulate sensor particle interference or salt-and-pepper noise, simulating discrete speckle contamination. Only one of these two is triggered to avoid excessively blurring the boundary features of fine strokes and connected structures.

[0035] To address the differences in writing tools, a 3×3 kernel morphological operation is performed on the binarized image with a 50% probability: erosion refines neat handwriting, while dilation coarsens the outlines of connected strokes. The 3×3 kernel size balances the preservation of local details with the range of morphological changes, adapting to the characteristics of different writing tools. Through these operations, style enhancement can cover variations across three dimensions: lighting, noise, and handwriting. It preserves the fine stroke details of Mongolian handwriting while simulating interference from the real environment, providing ample training samples for the model to learn anti-interference features.

[0036] (3) To address the challenges of complex Mongolian cursive writing structures and the ease with which similar characters can be confused, a targeted customized strategy was designed.

[0037] Local deformation of the connected stroke region addresses the issue of variations in the shape of connected strokes. First, the boundaries of the connected stroke region are clearly defined, using the absence of obvious gaps between characters as a standard, to generate a mask covering the connected stroke region. Non-connected stroke regions are not involved in the transformation. Then, a local affine transformation is performed on the masked area, including random scaling (one-fold vertically and horizontally) to simulate differences in the tightness and angle of the connected strokes, mimicking the natural tilt during writing. This operation enhances the morphological diversity of the connected stroke region, forcing the model to learn the elastic characteristics of the connected stroke outline and reducing segmentation errors caused by a lack of variation in the connected stroke shape.

[0038] Similar character variant generation addresses the issue of insufficient differentiation based on subtle features. It prioritizes the selection of the most easily confused character pairs in Mongolian writing and designs targeted perturbations based on their core distinguishing features, such as for the character "". "and" " (Mongolian script character, no specific meaning) Adjust the length of the vertical stroke to " "and" (Mongolian characters, without specific meaning) Perturb the hook angle or arc curvature, slightly adjusting the curvature. By generating near-realistic yet clearly distinguishable variant samples, the model's sensitivity to key features such as stroke length and hook angle is increased, reducing the probability of confusion between similar characters.

[0039] Vertical edge cropping simulates irregular writing boundaries. In Mongolian vertical writing, the beginning and end of lines often have partial gaps due to paper edges or writing habits, such as truncated strokes at the top or bottom of characters. To address this, the vertical edge region of the image is randomly cropped, preserving the main character in the center, thus covering real-world scenarios such as missing tops of characters at the beginning of lines and truncated bottoms of characters at the end of lines. This enhances the model's ability to recognize partially missing but structurally complete characters, improving its tolerance to irregular vertical text layouts.

[0040] In this step, the expanded data obtained after adding data is merged with the original handwritten dataset to serve as the dataset for subsequent training.

[0041] Step 2: Construct a Mongolian handwriting recognition model based on the combination of dual-branch feature fusion and Transformer sequence recognition. Use global branch and detail branch to extract global features and detail features of the preprocessed data respectively, and use spatial attention module to perform dynamic weighted fusion based on sample characteristics to form a joint representation that takes into account both global structure and detail features.

[0042] This step primarily addresses the relatively low recognition efficiency of Mongolian handwriting recognition models. The low recognition rate and accuracy of Mongolian handwriting stem from the challenges posed by the numerous connected strokes and the ease with which similar characters are confused. Single models struggle to simultaneously capture both global structure and detailed features. To improve recognition performance, this invention overcomes the limitations of traditional single-model approaches that cannot adequately address both global structure and detailed features. Targeting the two core pain points of dense vertical connected strokes and the ease with which similar characters are confused in Mongolian script, a hybrid model combining dual-branch feature fusion and Transformer sequence recognition is constructed. A customized dual-branch design with complementary functions—a global branch and a detailed branch—is designed. The global branch is based on the pure convolutional network ConvNeXt, which captures the global connected stroke structure. The detailed branch is based on ResNeSt101, which distinguishes subtle character differences. Features are dynamically weighted and fused through attention. Combined with Transformer encoding and decoding to model vertical sequence dependencies, this adapts to the writing style of Mongolian script and outputs a character sequence. The challenge of this invention lies in optimizing the dual-branch feature fusion strategy to balance global and detailed features, and adjusting the Transformer's positional encoding to adapt to the vertical positional relationships of vertically written characters, ensuring the accuracy of cursive segmentation and similar character differentiation. Furthermore, it is necessary to explore a collaborative mechanism between multi-stage optimization and feature fusion to further improve the model's adaptability to complex writing scenarios.

[0043] Specifically, the global branch utilizes the ConvNeXt network to focus on the global outline and word-level layout of the vertical strokes of Mongolian characters. The ConvNeXt network employs large-kernel convolutional layers to simulate a multi-head self-attention mechanism, enhancing long-range contextual dependency capture. Layer normalization is introduced after the convolutional block to continue the training stability optimization of the Transformer. An MLP structure with GELU activation replaces the traditional "convolution-normalization-activation" combination, strengthening feature representation capabilities. Borrowing from the deep hierarchical design of the ResNet network, multi-scale visual feature representations are constructed through progressive downsampling and feature dimension expansion, achieving hierarchical learning from low-level to high-level features, effectively balancing the modeling ability of local details and global structure. Its counterpart is... Figure 2 As shown.

[0044] The ConvNeXt structure is divided into 5 stages, each stacking multiple ConvNeXt Blocks. The parameters and functions of each stage are closely aligned with the morphology of Mongolian characters. This invention innovatively uses 7×7 Depthwise convolutions instead of traditional 3×3 convolutions in stages 2-5, deeply adapting to the morphological characteristics of vertically connected strokes in Mongolian script. From a structural capture perspective, the 7×7 convolution can cover the entire outline of long connected characters in one go, covering approximately 1 / 3 of the height in the vertical direction. This avoids the problem of broken strokes caused by splicing multiple layers of 3×3 convolutions, preserving the spatial continuity of the connected strokes. In terms of computational efficiency, the Depthwise convolution reduces the parameter computation to 7×7×C through a channel-by-channel independent convolution mechanism. In practice, combining it with 1×1 convolutions results in even fewer parameters and more direct spatial modeling. On a Mongolian script dataset with tens of thousands of samples, this significantly reduces the risk of overfitting while ensuring efficient model training. More importantly, it enhances feature completeness. The 7×7 Depthwise convolution accurately captures the vertical continuity of connected strokes. The convolution kernel simultaneously covers the connection areas between upper and lower strokes, learning structural features such as the tilt angle and stroke thickness variations of connected strokes. This provides more complete global structural support for subsequent connected stroke segmentation and recognition, significantly improving the accuracy of connected character recognition. Compared to the BN+ReLU combination used in ResNet Block, ConvNeXt Block overcomes the training bottleneck through a collaborative design of LN+GELU. Layer Norm (LN) normalizes the statistics for the channel dimension, and its formula is: Among them The mean and variance within the inner channel are optimized to eliminate batch size dependence and avoid training oscillations caused by small sample annotations of Mongolian handwritten characters; GELU activation is based on the Gaussian cumulative distribution function, the formula of which is: in For Gaussian CDF, negative gradients are preserved to smooth nonlinearities and enhance the " "and" (Mongolian script characters, no specific meaning) Length of vertical stroke, " "(Mongolian script character, no specific meaning) Captures subtle differences in strokes such as hook angles. Compared to the Swin Transformer Block, the ConvNeXt Block's pure convolutional architecture highlights its engineering advantages. The SwinBlock's moving window self-attention (MSA) computational complexity increases quadratically with resolution, and its sequential computation is not conducive to hardware parallelization. In contrast, the ConvNeXt Block's 7×7 depthwise convolution has a complexity of..." It exhibits linear growth, adapts to high-resolution vertical Mongolian input, and, combined with the hardware-friendly nature of convolution, can significantly improve training and inference speed, supporting the engineering implementation of dual-branch feature fusion.

[0045] The detail branch utilizes the ResNeSt101 network to specifically capture subtle differences between similar Mongolian characters. The core advantage of ResNeSt101 lies in its Split Attention mechanism (in this embodiment, radix=4). ResNeSt101 processes feature map groups through grouped attention blocks, ensuring that the feature representation of each group is obtained through a weighted combination of its subgroups, with weights based on global context information. This method effectively enhances the interaction of cross-channel information, thereby obtaining richer feature representations.

[0046] ResNeSt101 comprises five stages, from Stage 1 to Stage 4, with a cumulative downsampling factor of 16 and a stride of 2. Taking a 384×384 Mongolian script image as an example, through layer-by-layer convolution and pooling operations, the spatial size of the feature map is gradually compressed to 24×24, and the number of channels is gradually increased from 64 dimensions in Stage 1 to 512 dimensions in Stage 4. Group attention is used to enhance feature discriminativeness. The 24×24 spatial resolution preserves fine strokes at the character level, and the 512 channels encode discriminative features such as stroke direction, curvature, and the position of additional points, providing fine-grained basis for similar character recognition. Finally, the 24×24×512 feature map output from Stage 4 is selected as the detail feature. The 24×24×512 feature map output from Stage 4 is the core of fine-grained feature extraction, designed to deeply adapt to the shapes of fine strokes in Mongolian script, such as hooks and short vertical strokes, as well as the differences between similar characters, such as the length of vertical strokes and the position of additional points. Based on the input image size of Mongolian handwritten characters, ResNeSt101 compresses the spatial resolution to 24×24 through four downsampling steps (Stage 1–Stage 4), each with a step size of 2, so that each feature point corresponds to a 16×16 pixel region in the original image (calculation relationship: 384 / 24 = 16). The design logic of this scale can be analyzed from two aspects: the ability to preserve fine strokes and the advantage of scale contrast. One is that fine strokes of Mongolian characters, such as… The hook shape (Mongolian script character, without specific meaning) and the " The short vertical strokes (Mongolian script characters, without specific meaning) are mostly between 8 and 12 pixels in length. The 24×24 feature map of ResNeSt101 Stage4, through scale mapping, makes each feature point correspond to a 16×16 pixel area of ​​the original image, providing a spatial basis for preserving fine strokes. Regarding the capture of hook-shaped curved forms, The 45° bend of the character "" (Mongolian script, meaningless) can be analyzed using edge detection methods such as the Sobel operator to extract local gradient change patterns, which are related to the "". The gently sloping arcs (Mongolian script characters, without specific meaning) show significant differences in gradient distribution. The hook shape exhibits an abrupt gradient peak, while the arc shows a continuous gradient distribution. This regional scale effectively preserves this distinguishing feature. Regarding the difference in vertical stroke length... "and" The pixel length difference of the character "" (Mongolian script, meaningless) can be encoded by statistically analyzing the density differences of foreground pixels within a 16×16 region. Longer vertical lines, covering more pixels, have a higher feature response intensity, forming a stable discriminative signal. Secondly, from a scale comparison perspective, the 24×24 design further highlights its advantages: compared to Stage 5's 12×12 feature map, corresponding to a 32×32 pixel region, excessive compression would lead to... The hook shape (Mongolian script character, without specific meaning) is blurred into a "dot", losing its curved feature and exacerbating the problem. "and" The risk of confusion due to similar characters such as "(Mongolian script, without specific meaning)"; compared to the 48×48 feature map of Stage3, which corresponds to an 8×8 pixel area, the 24×24 model reduces the feature map area from 48... 2 Reduced to 24 2 The parameter size is reduced by 75%, which avoids the overfitting problem caused by parameter redundancy in Stage 3, and achieves a precise balance between detail preservation and computational efficiency. Its ResNeSt Block structure is as follows: Figure 3 As shown.

[0047] ResNeSt101's Split Attention divides the channel into four functional subsets, focusing on differences in vertical stroke length, hook bending direction, arc curvature, and attachment point position, dynamically strengthening key distinguishing features of similar characters to solve the problem of confusion with fine strokes. Split Attention dynamically strengthens features related to the distinctiveness of Mongolian script through an attention mechanism, with the specific adaptation logic revolving around the four dimensions of Mongolian script's distinctive features. Its subset functional division accurately matches Mongolian script features; the specific design and Split Attention module are as follows... Figure 4 As shown, the specific description is as follows: 1) Subset 1 (vertical direction): Focusing on the shape of vertical strokes, we capture the vertical edge density of short and long vertical strokes. By statistically analyzing the difference in gradient response in the vertical direction, we encode the length difference between the two strokes. Short vertical strokes have a weak response, while long vertical strokes have a strong response.

[0048] 2) Subset 2 (diagonal direction): It adapts to hook-shaped bends and arcs, enhancing the directional characteristics of these non-perpendicular fine strokes and distinguishing the morphological differences between hook shapes and straight lines.

[0049] 3) Subset 3 (Circular Curvature): Extract local curvature information and use Gaussian curvature calculation to distinguish between gentle circular arcs (large radius of curvature, uniform response) and sharp hooks (small radius of curvature, abrupt local response).

[0050] 4) Subset 4 (additional point location): Locate character appendages and use local extrema detection to mark the spatial position of appendages, thus helping to distinguish whether appendages exist and their positions.

[0051] When the model processes similar character pairs, the Split Attention mechanism dynamically increases the weights of the corresponding subsets: for example, when distinguishing the length of vertical strokes, the attention weight of subset 1 increases significantly, strengthening the contribution of vertical features; when distinguishing between hooks and arcs, the weights of subsets 2 and 3 are simultaneously enhanced. This task-driven feature selection mechanism allows the model to selectively strengthen the channels most relevant to the current recognition, providing a core basis for fine-grained recognition. Through spatial resolution adaptation to the scale of fine strokes, and the dynamic encoding of discriminative features by multi-dimensional channels and Split Attention, ResNeSt101's Stage4 features can accurately capture key information such as the shape, direction, curvature, and additional points of Mongolian fine strokes, laying the foundation for the contribution of detailed features in dual-branch feature fusion.

[0052] To achieve feature fusion between global and detailed features and avoid information redundancy caused by simple concatenation of bi-branch features, an attention gating module is designed, utilizing a spatial attention mechanism. Its structure is as follows: Figure 5 As shown. Branch weights are dynamically allocated based on the characteristics of the input samples, achieving adaptive fusion that emphasizes details for similar characters and the overall picture for long, connected strokes. The Stage4 output of the global branch needs to undergo channel compression to ensure dimensional matching and spatial correspondence. The specific process is as follows: The k-dimensional channels of the global branch Stage4 differ from the m-dimensional channels of the detail branch. Direct fusion would lead to dimensional redundancy, computational inefficiency, and even feature bias. Channel compression is achieved through 1×1 convolutions (stride=1, padding=0). 1×1 convolutions can preserve feature correlations through cross-channel weighted fusion without disrupting the spatial structure. For Mongolian script adaptation, the compressed n-dimensional channels are aligned, providing a basis for the subsequent channel-by-channel weight calculation of the spatial attention module. This avoids weight bias caused by differences in the number of channels and ensures a fair comparison of the feature contributions of connected strokes and fine strokes.

[0053] After channel compression, the detail branch outputs the feature map. Global branch output feature map The weights are calculated using the following steps: Step 1: Add and fuse the two features pixel by pixel to obtain the following formula: Step 2: Apply 1×1 convolution to... Compression, followed by global average pooling and Activation, generating attention weights Its formula is: The logic is as follows: The closer it is to 1, the higher the proportion of similar characters in the sample, and the more detailed branches need to be strengthened to adapt to similar character samples. The closer the value is to 0, the more prominent the long, connected strokes are in the sample, requiring strengthening of the global branch to adapt to long, connected stroke samples. Ultimately, the two are dynamically fused using an attention gating module, resulting in a weighted sum of features from both branches, avoiding feature bias from a single weight. The formula is as follows: Through a dual-branch feature extraction and channel compression design, along with dynamic fusion using attention gating, this model achieves three core objectives: First, it preserves details without losing fine strokes. Second, it encodes discriminative features: SplitAttention enhances fine-grained differences between similar characters. Third, it aligns the dual branches: channel matching ensures the correspondence between connected stroke areas and fine stroke positions. Ultimately, the detail branch provides fine-grained feature support for solving the problem of confusion between similar Mongolian characters, complementing the connected stroke structure modeling of the global branch, and jointly improving the robustness of Mongolian handwriting recognition.

[0054] Step 3: Based on the strategy of dynamic position encoding, spatially aware sequence transformation and attention bias optimization, Transformer encoding and decoding sequence recognition is performed according to the fusion features.

[0055] This step primarily addresses the issues encountered when using Transformer sequence recognition to handle the characteristics of Mongolian handwritten characters, such as dense vertical strokes and susceptibility to character deformation. These issues include the loss of spatial structure from two-dimensional features to one-dimensional sequences, and the inability of static positional encoding to adapt to deformed characters. The original Transformer model consists of an encoder and a decoder. Both the encoder and decoder layers incorporate multi-head self-attention mechanisms, offering advantages in parallel processing of long sequences. Its structure is as follows: Figure 6As shown, it mainly consists of four modules: an input module, an output module, an encoder, and a decoder, with Nx encoders and decoders stacked together. When dealing with the characteristics of Mongolian handwriting—dense vertical strokes and easily deformable characters—there are problems such as loss of spatial structure from two-dimensional features to one-dimensional sequences and static positional encoding failing to adapt to deformable characters. This invention addresses these issues by designing a model combining ResNeSt101 and ConvNeXt dual-branch feature fusion with a Transformer, employing strategies for dynamic positional encoding, spatially aware sequence transformation, and attention bias optimization to adapt to the characteristics of Mongolian handwriting.

[0056] To fuse the two-dimensional features of the dual-branch feature, the detail branch ResNeSt101 captures the differences in fine strokes, while the global branch ConvNeXt models the global structure of the strokes and converts it into a one-dimensional sequence that conforms to the writing order. It also accurately models the context dependency. The Transformer module of the model is designed around a three-layer architecture of spatially aware sequence transformation → dynamic position encoding → hierarchical encoding and decoding, which systematically adapts to the characteristics of Mongolian writing.

[0057] The original Transformer's position encoding is a static sine formula, as follows: This encoding method cannot adapt to the distortion and stroke deviation of Mongolian handwriting.

[0058] This invention reuses backbone network features to generate dynamic positional codes. It utilizes the features fused from dual-branch features, compresses the spatial dimension through 3×3 convolution and global average pooling, and captures deformation patterns. The formula is as follows: Then, a fully connected layer maps the pooled features into positional encoding vectors. Related to the dynamic changes in Mongolian script: in, These are trainable parameters, replacing static sinusoidal encoding. The core advantage of dynamic encoding lies in the fact that the distortion and tilting of strokes will change the characteristics of the intermediate layers. And thus Presenting differentiation, such as " (Mongolian script characters, no specific meaning) When tilted at 10° and 20°, The vector distributions are distinguishable, effectively improving the model's robustness to deformation.

[0059] The Transformer processes one-dimensional sequences by default, while the backbone network outputs a two-dimensional feature map. Direct flattening would lose the vertical alignment and adjacency relationships of strokes; therefore, a block-based flattening and position labeling strategy is designed. The specific design scheme is as follows: the fused two-dimensional feature map is divided into 3×3 non-overlapping blocks, and flattened into a sequence in a vertical order from top to bottom and left to right, preserving the spatial structure within each block. The formula is: in, The backbone network outputs features, corresponding to the fused features. A two-dimensional coordinate is appended to each token, with row numbers as follows: Column number is The encoding is a low-dimensional vector concatenated with the token feature to enhance vertical spatial awareness. The formula is as follows: in, This is a coordinate embedding function that adapts to the top-to-bottom writing order of Mongolian script.

[0060] The vertical dependence of vertically written Mongolian text is stronger than that of horizontally written text, thus enhancing the vertical weighting of self-attention. In the self-attention calculation, a line distance penalty is introduced to increase the vertical attention weight and suppress invalid associations across multiple lines. The penalty coefficient controls the inhibitory effect of line distance on attention weights, and its formula is as follows: in, These are the query and key vectors for the i-th and j-th tokens, respectively, where d is the dimension of the query / key vector. Used for scaling to prevent the Softmax gradient from vanishing due to an excessively large inner product. Softmax normalizes the weights to [0,1], with a total sum of 1. Mongolian script is written vertically, and the dependencies between characters / columns are strong in the vertical direction and weak in the horizontal direction. To strengthen this vertical dependency, a line distance penalty term is introduced into the formula. , This represents the row index of the i-th and j-th tokens, i.e., which row the token belongs to in the vertical image. This formula is designed to adapt to the characteristics of the vertical, connected strokes of Mongolian script. It enhances the vertical weights of standard self-attention, focuses on local vertical dependencies through row distance penalty, and suppresses invalid associations across multiple rows, thereby improving the model's ability to model the vertical structure of Mongolian script.

[0061] To address the short sequences and strong local dependencies characteristic of Mongolian handwritten characters, this invention employs a lightweight and efficient encoder-decoder structure. The encoder utilizes a 4-layer Transformer architecture with 8 self-attention heads. The 4-layer architecture is sufficient to capture the contextual dependencies of vertical sequences, avoiding computational redundancy caused by excessively deep networks. The 8-head self-attention extracts features in parallel from eight dimensions, including stroke direction, ligature position, row / column coordinates, and fine stroke length, enhancing fine-grained differentiation of similar characters. The hidden layer dimension is set to k-dimensional, balancing expressive power and computational cost, adapting to the small-sample training scenario of Mongolian characters, and avoiding over-parameterization. The decoder uses Beam Search (beam size=3), retaining the top 3 candidate sequences. In Mongolian handwritten character recognition, beam size=3 can improve accuracy by 1–2% while accelerating inference speed by 30% compared to beam size=5, meeting the efficiency requirements of practical applications. The output strictly follows the vertical writing order from top to bottom and left to right, ensuring that the decoding result of ligatures is consistent with the writing logic, avoiding semantic errors caused by sequence disorder.

[0062] By employing a three-layer design—spatial-aware sequence transformation to preserve vertical structure, dynamic positional encoding to adapt to deformation, and hierarchical encoding and decoding to model the context—the Transformer module of this invention achieves fine-grained differentiation of dual-branch feature fusion features and the transformation of global structure modeling into sequence-level association capabilities, providing core support for the final output of recognition results that conform to the Mongolian writing order.

Claims

1. A method for Mongolian handwriting recognition based on dual-branch feature fusion and Transformer sequence recognition, characterized in that, Includes the following steps: Step 1: Obtain the Mongolian handwritten image dataset and perform multi-level data augmentation and data preprocessing; Step 2: Construct a Mongolian handwriting recognition model based on the combination of dual-branch feature fusion and Transformer sequence recognition. Use global branch and detail branch to extract global features and detail features of the preprocessed data respectively, and use spatial attention module to perform dynamic weighted fusion based on sample characteristics. The global branch is based on the pure convolutional network ConvNeXt, and replaces the 3×3 convolution with 7×7 depthwise separable convolution in Stages 2-5 of the network, covering the complete outline of the vertical strokes in one go and capturing the long-distance dependence of the vertical strokes. The detail branch is based on the ResNeSt101 residual network with group attention. With the help of its group attention mechanism, the multi-dimensional channel is divided into 4 functional subsets, which focus on the difference in vertical stroke length, the direction of hook bend, the curvature of arc, and the position of additional points, respectively, to dynamically enhance the key difference features of similar characters; the detail branch uses the feature map output by its Stage4 as detail features. Step 3: Based on the strategy of dynamic position encoding, spatially aware sequence transformation and attention bias optimization, Transformer encoding and decoding sequence recognition is performed according to the fusion features; The dynamic location coding is generated by reusing backbone network features, and the implementation method is as follows: Deformation patterns are captured by compressing the spatial dimension of the fused features using 3×3 depthwise separable convolution and global average pooling. The compressed features are mapped to positional encoding vectors through a fully connected layer; The implementation method of spatially-aware sequence transformation is as follows: The fusion feature is divided into 3×3 non-overlapping blocks, which are then flattened into a sequence in a vertical order from top to bottom and from left to right, preserving the spatial structure within each block. Attach two-dimensional coordinates to each token, with row numbers as follows: Column number is The encoding is a low-dimensional vector concatenated with the token feature to enhance vertical spatial awareness. The formula is as follows: in, This is a coordinate embedding function that adapts to the top-to-bottom writing order of Mongolian script. The implementation method for attention bias optimization is as follows: In self-attention calculation, a line distance penalty is introduced to increase the attention weight in the vertical direction and suppress invalid associations across multiple lines. The strength of the suppression of attention weight by line distance is controlled by the following formula: in, These are the query and key vectors for the i-th and j-th tokens, respectively, where d is the dimension of the query and key vector. Used for scaling to prevent the softmax gradient from vanishing due to an excessively large inner product. Softmax normalizes the weights to [0,1], with a total sum of 1. The penalty coefficient is... This indicates the row index of the i-th and j-th tokens, i.e., which row the token belongs to in the vertical image.

2. The Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition according to claim 1, characterized in that, Step 1, multi-level data augmentation includes basic geometric transformation, style enhancement, and customized strategies; The basic geometric transformations employ directional rotation, vertical translation, and scaling to simulate the tilt and positional deviations in natural handwriting. The style enhancement is achieved by contrast perturbation, Gaussian or salt-and-pepper noise injection, and handwriting thickness adjustment to simulate the differences between different writing tools and scanning environments. The customized strategy enhances the adaptability to key morphological variations by using local affine transformations in the connected stroke area, subtle stroke perturbations of similar characters, and vertical edge clipping. When performing data augmentation, an online dynamic augmentation mechanism is adopted. Several transformations are randomly combined in each training iteration, and the parameters are fine-tuned through the validation set. At the same time, the augmented samples are screened to remove excessively distorted samples and retain effective data that can simulate real blur and deformation.

3. The Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition according to claim 2, characterized in that, The local affine transformation of the connected stroke region is implemented as follows: First, clearly mark the range of the connected stroke area, and generate a mask that covers the connected stroke area based on the standard that there is no obvious gap between characters. Non-connected stroke areas do not participate in the transformation. Then, a local affine transformation is performed on the masked area, including random scaling up and down by one time, to simulate the difference in the tightness of the strokes and the distortion at different angles, and to simulate the natural tilt during writing. The subtle stroke perturbation of similar characters is implemented as follows: We screened easily confused character pairs in Mongolian writing and designed targeted perturbations based on their core distinguishing features, including adjusting the length of vertical strokes, perturbing the angle of hooks or the curvature of arcs, and slightly adjusting the curvature to generate near-realistic yet clearly distinguishable variant samples. The vertical edge trimming is implemented as follows: The image is randomly cropped along its vertical edge, retaining the main characters in the center area, thus covering the real-world scenario where the top of the first character of a line is missing and the bottom of the last character is truncated.

4. The Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition according to claim 1, characterized in that, The method for dividing the multi-dimensional channel into four functional subsets is as follows: Subset 1 focuses on the vertical stroke shape, capturing the vertical edge density of short and long vertical strokes. By statistically analyzing the gradient response differences in the vertical direction, the length difference between the two is encoded, with short vertical strokes having weaker responses and long vertical strokes having stronger responses. Subset 2 adapts to hook-shaped bends and arcs, enhances the directional characteristics of non-perpendicular fine strokes, and distinguishes the morphological differences between hook shapes and straight lines; Subset 3 extracts local curvature information and distinguishes between gentle circular arcs (large radius of curvature, uniform response) and sharp hooks (small radius of curvature, abrupt local response) by Gaussian curvature calculation. Subset 4 locates character appendages and uses local extrema detection to mark the spatial position of appendages, helping to distinguish whether there are appendages and their positions.

5. The Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition according to claim 1, characterized in that, The method for dynamically weighted fusion based on sample characteristics using a spatial attention module is as follows: Global and detailed features are compressed to a consistent channel dimension using 1×1 convolution, ensuring feature dimension alignment. The system generates attention weights α by summing features in two branches, compressing them with 1×1 convolution, using global average pooling, and activating them with Sigmoid. When α approaches 1, it enhances detailed features and adapts to similar character samples. When α approaches 0, it highlights global features and adapts to long cursive script samples, ultimately achieving dynamic weighted fusion.

6. The Mongolian handwriting recognition method based on dual-branch feature fusion and Transformer sequence recognition according to claim 1, characterized in that, The dual-branch feature summation is achieved by adding the global features and detail features after channel compression pixel by pixel. The pixel-by-pixel summation result is compressed by 1×1 convolution, global average pooling, and Sigmoid activation to generate attention weight α.

Images