Natural scene-oriented end-to-end Tibetan text detection and recognition system

Abstract

The invention discloses a natural scene-oriented end-to-end Tibetan text detection and recognition system, and relates to the technical field of text image recognition, and the detection and recognition system comprises a multi-scale feature extraction module, a Tibetan special prompt mechanism module, a detection recognition calculation module and a loss optimization module. Aiming at low detection and recognition accuracy caused by the problems of lack of Tibetan natural image data, overlarge background noise of a natural scene, similar Tibetan character features and the like, a Tibetan detection and recognition public data set facing the natural scene is constructed, meanwhile, an end-to-end Tibetan detection and recognition method is provided, a TransformerEncoder architecture is combined, and the Tibetan detection and recognition accuracy is improved. According to the Tibetan detection and recognition method based on the time sequence characteristics, the time sequence characteristics of the Tibetan detection and recognition method are optimized, a window prompt and text prompt mechanism is innovatively introduced, a recognition module based on the time sequence characteristics is constructed, experimental results show that the detection F1 score of the model on an NSTD data set reaches 83.66%, and the effectiveness of the method in end-to-end Tibetan detection and recognition tasks is fully proved.

Description

Technical Field

[0001] This invention relates to the field of text image recognition technology, specifically to an end-to-end Tibetan text detection and recognition system for natural scenes. Background Technology

[0002] In the field of computer vision, extracting textual information from natural scene images and converting it into digital format is a crucial task. The technologies involved enable the digitization of textual data, which is significant for promoting human-computer interaction, improving machine translation quality, optimizing information retrieval systems, and advancing autonomous driving technology. Since text detection and recognition are the initial steps in the image translation process, their accuracy directly impacts the overall translation performance. However, text images in natural scenes exhibit high diversity. For example, there are too many irrelevant background features, and shooting conditions such as different weather and times of day can lead to low text contrast and uneven lighting. These factors all pose challenges to the accuracy of text detection and recognition. Therefore, developing text detection and recognition technologies with strong generalization capabilities to handle these complex situations is a highly challenging task in this field.

[0003] Tibetan is a long-standing and widely used cross-border script, used in Tibetan areas of my country as well as parts of Nepal, Bhutan, and India. The intelligent detection and recognition of Tibetan script has high research and application value.

[0004] Chinese Patent Publication No. CN110135419B discloses an end-to-end text recognition method in natural scenes, including training a framework using natural scene images and ground truth labels, and predicting text regions and content on natural scene images: In the training phase, images of natural scenes containing text are collected, a dataset containing text location and content is constructed, a standard end-to-end text recognition framework is defined, a detection part is trained using ground truth detection labels, a nearest neighbor correlation boundary optimization algorithm is used to optimize the detection region, the optimized detection region is input into the recognition part to train the recognition part parameters, and the trained framework parameters are saved to a data platform; In the testing phase, the trained framework parameters are read, a test image is input, text regions are detected in the detection phase, the nearest neighbor correlation boundary optimization algorithm is used to optimize the detection region, and the optimized detection region is sent to the recognition part for text recognition.

[0005] However, the following problems still exist:

[0006] On the one hand, there is a severe lack of high-quality Tibetan image datasets, and the existing datasets are much smaller than Chinese and English image datasets;

[0007] On the other hand, Tibetan script recognition in natural scenes is more complex than that in Chinese and English. This is due to the high noise environment caused by the complex and varied backgrounds of natural scenes, as well as the unique complex glyph structure of Tibetan characters, similar character features, and grammatical rules that differ from those of Chinese and English. These challenges result in poor performance of existing models in Tibetan script detection and recognition in natural scenes.

[0008] Therefore, this invention requires the design of an end-to-end Tibetan text detection and recognition system for natural scenes to solve the aforementioned problems. Summary of the Invention

[0009] The purpose of this invention is to provide an end-to-end Tibetan text detection and recognition system for natural scenes in order to solve the above-mentioned problems, thereby addressing the issues mentioned in the background art.

[0010] To address the above problems, the present invention provides a technical solution:

[0011] An end-to-end Tibetan text detection and recognition system for natural scenes, the detection and recognition system includes a multi-scale feature extraction module, a Tibetan-specific prompting mechanism module, a detection and recognition calculation module, and a loss optimization module. The multi-scale feature extraction module is used to extract multi-scale feature representations from the input image.

[0012] The Tibetan-specific prompting mechanism module is connected to the multi-scale feature extraction module and is used to generate text prompts and window prompts to enhance feature representation.

[0013] The detection and recognition calculation module is connected to the Tibetan-specific prompting mechanism module and is used to perform text detection and recognition based on the enhanced features.

[0014] The loss optimization module is connected to the detection and recognition calculation module and is used to calculate multi-task loss and optimize model parameters;

[0015] The detection and recognition system achieves end-to-end Tibetan text detection and recognition through the collaborative work of its various modules. Staff can access the detection and recognition system by logging into the loss optimization module, and then view the real-time operating data of the multi-scale feature extraction module, the Tibetan-specific prompting mechanism module, and the detection and recognition calculation module.

[0016] In a preferred embodiment of the present invention, the multi-scale feature extraction module is used to perform hierarchical feature extraction on the input image and output feature maps of multiple resolutions.

[0017] The Tibetan-specific prompting mechanism module is used to generate text prompts based on the Tibetan character library and to generate window prompts by randomly cropping them, thereby enhancing the model's ability to perceive Tibetan text.

[0018] The detection and recognition calculation module is used to encode enhanced features based on the Transformer architecture and output text detection boxes and recognition results;

[0019] The loss optimization module is used to integrate multiple loss functions and jointly optimize the detection and recognition tasks through gradient backpropagation.

[0020] In a preferred embodiment of the present invention, the multi-scale feature extraction module includes a SwingTransformer basic architecture unit, a multiple feature capture unit, a proposal anchor box generation unit, and a bilinear interpolation feature aggregation unit. The output of the SwingTransformer basic architecture unit is communicatively connected to the input of the multiple feature capture unit, the output of the multiple feature capture unit is communicatively connected to the input of the proposal anchor box generation unit, and the output of the proposal anchor box generation unit is communicatively connected to the input of the bilinear interpolation feature aggregation unit.

[0021] In a preferred embodiment of the present invention, the SwinTransformer infrastructure unit is used to perform hierarchical window attention calculation on the input image and realize information interaction between adjacent windows through the Window-MSA and SW-MSA alternation mechanism.

[0022] The multi-feature capture unit is used to output feature tensors with four different resolutions: 56×56, 28×28, 14×14, and 7×7, which are used to simultaneously capture the overall outline and local detail features of Tibetan characters.

[0023] The proposal anchor box generation unit is used to generate k quadrilateral proposal anchor boxes with coordinates (x1, y1, x2, y2) based on the feature map, which are used to guide subsequent feature selection.

[0024] The bilinear interpolation feature aggregation unit is used to extract features from feature tensors at different levels based on the proposed anchor frame, and to concatenate the 192 aggregated features extracted at each level into 768 aggregated features with a length and width of 7.

[0025] The bilinear interpolation feature aggregation unit specifically includes:

[0026] The feature grid partitioning subunit divides each proposed feature boundary into multiple feature grids, with the coordinate center point (xij, yij) as the unique identifier.

[0027] The weight calculation subunit calculates the feature weight wmn of four adjacent pixels (xm, yn) based on the coordinates of the center point of the feature grid.

[0028] The aggregated feature generation subunit calculates the aggregated features using a formula, and splices the four layers of features into a unified 768-dimensional feature representation.

[0029] In a preferred embodiment of the present invention, the Tibetan-specific prompting mechanism module includes a Tibetan script library construction unit, a text prompting generation unit, a window prompting generation unit, and a random prompting strategy control unit. The text prompting generation unit is integrated within the Tibetan script library construction unit. The text prompting generation unit and the window prompting generation unit are bidirectionally connected, and the Tibetan script library construction unit and the random prompting strategy control unit are bidirectionally connected.

[0030] In a preferred embodiment of the present invention, the Tibetan character database construction unit is used to construct a semantic prior knowledge base containing 3371 deduplicated characters based on VCC vertical segmentation technology.

[0031] The text prompt generation unit is used to randomly select complete semantic segments of the annotated text, convert them into character table encoding to form prompt vector I, and ensure dimensionality consistency through linear projection and zero padding.

[0032] The text prompt generation unit specifically includes:

[0033] The semantic segment selection subunit randomly selects a complete semantic segment from the N annotated texts of the instances to be identified in the current image;

[0034] The encoding conversion subunit converts the selected text into a 3371-dimensional character table encoding representation;

[0035] The dimension alignment processing subunit processes the cue vector I into a fixed-length vector by zero padding to ensure that it is consistent with the dimension of the image features.

[0036] The feature fusion calculation subunit combines text prompt features with image features according to a formula;

[0037] The window prompt generation unit is used to output the coordinates of the cropping quadrilateral through the self-attention module, randomly crop the image within the anchor frame, and the cropping area is 20%-40% of the original image area. After processing by the multilayer perceptron, the window prompt features are formed.

[0038] The window prompt generation unit specifically includes:

[0039] The attention-guided coordinate prediction subunit outputs the four coordinates of the clipped quadrilateral through a self-attention module and a linear layer;

[0040] The area control cropping subunit controls the cropping area within 20%-40% of the image area within the anchor frame based on a random number.

[0041] The local feature extraction subunit extracts features from the cropped image crop(E) to generate window prompt features that enhance spatial awareness.

[0042] The combined prompt fusion subunit simultaneously fuses text prompts, window prompts, and original image features according to a formula;

[0043] The random prompting strategy control unit is used to control the activation of the prompting mechanism according to a probability distribution of 40% no prompts, 30% text-only prompts, and 30% combined prompts.

[0044] In a preferred embodiment of the present invention, the detection and recognition calculation module includes a TransformerEncoder encoding unit, a detection result output unit, and a recognition result output unit. The output end of the TransformerEncoder encoding unit is communicatively connected to the input end of the detection result output unit, and the output end of the detection result output unit is communicatively connected to the input end of the recognition result output unit.

[0045] The TransformerEncoder encoding unit is used to employ a multi-head self-attention mechanism and a feedforward network to capture the complex relationship between image and text features;

[0046] The detection result output unit is used to output the detection results of M=100 instance points through the multilayer perceptron. Each instance contains N=2 coordinate points (top left corner and bottom right corner) with a dimension of (batch, M, N, 2).

[0047] The detection result output unit specifically includes:

[0048] The confidence calculation subunit processes the Plogits value using the sigmoid function and calculates the confidence score for each instance according to the formula.

[0049] The binary classification subunit performs binary classification of text or background on M=100 proposal instances;

[0050] The coordinate regression subunit outputs two coordinate points for each instance, representing the upper left and lower right corners of the detection box, respectively.

[0051] The recognition result output unit is used to output Tibetan character recognition results with a maximum sequence length L=25 through linear projection and softmax processing.

[0052] In a preferred embodiment of the present invention, the loss optimization module includes an algorithm matching unit, a multi-task loss calculation unit, and a gradient backpropagation unit. The output of the algorithm matching unit is communicatively connected to the input of the multi-task loss calculation unit, and the output of the multi-task loss calculation unit is communicatively connected to the input of the gradient backpropagation unit.

[0053] The algorithm matching unit is used to filter detection results based on confidence scores and calculate the minimum distance between the predicted anchor box and the real annotation.

[0054] The multi-task loss calculation unit is used to integrate GIoU loss, L1 distance loss, CTC loss and alignment loss, and the total loss function is the sum of the four.

[0055] The multi-task loss calculation unit specifically includes:

[0056] The GIoU loss calculation subunit addresses the issue of excessively large differences between predicted and labeled points in the early stages of training, resulting in an IoU of 0 and the inability to provide gradients. The calculation formula is as follows:

[0057] The L1 distance loss calculation subunit calculates the distance loss between the predicted anchor point Pi and the labeled anchor point Gj.

[0058] The CTC loss calculation subunit processes the recognition of text sequences of variable length and calculates the difference between the predicted text and the target text sequence using softmax and CTC loss functions.

[0059] The alignment loss calculation subunit dynamically learns and adjusts the prompt sampling based on the second norm loss value to enhance the alignment effect between the text and the window prompt.

[0060] The gradient backpropagation unit is used to explicitly optimize the selection process of shared aggregated features by jointly optimizing the detection and recognition tasks.

[0061] In a preferred embodiment of the present invention, the data preprocessing and enhancement module is further included, wherein the data preprocessing and enhancement module includes:

[0062] The image size standardization unit is used to uniformly process the input image to a 3-channel 224×224 size;

[0063] The perspective transformation enhancement unit is used to select images from the SCUT-CTW1500 dataset, remove the original text, and then render Tibetan text.

[0064] The fusion unit is used to ensure a natural transition in color and texture between the enhanced text area and the original background.

[0065] The multilingual environment simulation unit is used to generate training samples for Tibetan-Chinese bilingual and Tibetan-English bilingual mixed scenarios.

[0066] In a preferred embodiment of the present invention, the hardware operating environment of the system includes:

[0067] Processor units: Two GeForce RTX 3090 graphics cards;

[0068] Software framework: PyTorch 1.8 deep learning framework;

[0069] Optimized configuration: Adam optimizer, batch size 16, initial learning rate 0.002, training for 1000 epochs, learning rate decay in the 600th epoch;

[0070] Inference configuration: Detection cross-union ratio threshold of 0.5, post-processing mechanism based on confidence score and nonmaximum suppression.

[0071] The beneficial effects of this invention are as follows: Addressing the low detection and recognition accuracy caused by the scarcity of Tibetan natural image data, excessive background noise in natural scenes, and similar features of Tibetan characters, this invention constructs a public dataset for Tibetan detection and recognition in natural scenes. Simultaneously, it proposes an end-to-end Tibetan detection and recognition method. This method uses SwinTransformer as the feature extraction module, combined with the TransformerEncoder architecture, and innovatively introduces window prompts and text prompts to construct a recognition module based on temporal features. Experimental results show that the proposed model achieves an F1 score of 83.66% and a character recognition accuracy of 81.92% on the NSTD dataset. Furthermore, ablation experiments verify the effectiveness of the prompting method: compared to the baseline model without prompts, the model's precision, recall, F1 score, and dictionary-based average recognition accuracy improved by 1.25%, 1.98%, 1.67%, and 2.57%, respectively, fully demonstrating the effectiveness of the proposed method in end-to-end Tibetan detection and recognition tasks. Attached Figure Description

[0072] For ease of explanation, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0073] Figure 1 This invention is an end-to-end Tibetan text detection and recognition system for natural scenes;

[0074] Figure 2 The images are natural scene images collected by the end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention.

[0075] Figure 3 This invention provides an image annotation process for an end-to-end Tibetan text detection and recognition system for natural scenes.

[0076] Figure 4 This is an example of a dataset for different scenarios in an end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention;

[0077] Figure 5 This is a schematic diagram illustrating a Tibetan data enhancement example of an end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention.

[0078] Figure 6 This is a visual comparison diagram of the results of an end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention;

[0079] Figure 7 This is a schematic diagram of the comparative experimental results in Experiment 1 of the end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention;

[0080] Figure 8 This is a schematic diagram of the comparison of detection accuracy in Experiment 4 of the end-to-end Tibetan text detection and recognition system for natural scenes according to the present invention. Detailed Implementation

[0081] like Figures 1-8 As shown, the specific implementation adopts the following technical solution:

[0082] An end-to-end Tibetan text detection and recognition system for natural scenes, the detection and recognition system includes a multi-scale feature extraction module, a Tibetan-specific prompting mechanism module, a detection and recognition calculation module, and a loss optimization module. The multi-scale feature extraction module is used to extract multi-scale feature representations from the input image.

[0083] The Tibetan-specific prompting mechanism module is connected to the multi-scale feature extraction module and is used to generate text prompts and window prompts to enhance feature representation.

[0084] The detection and recognition calculation module is connected to the Tibetan-specific prompting mechanism module and is used to perform text detection and recognition based on the enhanced features.

[0085] The loss optimization module is connected to the detection and recognition calculation module and is used to calculate multi-task loss and optimize model parameters;

[0086] The detection and recognition system achieves end-to-end Tibetan text detection and recognition through the collaborative work of its various modules. Staff can access the detection and recognition system by logging into the loss optimization module, and then view the real-time operating data of the multi-scale feature extraction module, the Tibetan-specific prompting mechanism module, and the detection and recognition calculation module.

[0087] The hardware operating environment of the system includes:

[0088] Processor units: Two GeForce RTX 3090 graphics cards;

[0089] Software framework: PyTorch 1.8 deep learning framework;

[0090] Optimized configuration: Adam optimizer, batch size 16, initial learning rate 0.002, training for 1000 epochs, learning rate decay in the 600th epoch;

[0091] Inference configuration: Detection cross-union ratio threshold of 0.5, post-processing mechanism based on confidence score and nonmaximum suppression.

[0092] The multi-scale feature extraction module is used to perform hierarchical feature extraction on the input image and output feature maps of multiple resolutions;

[0093] The multi-scale feature extraction module includes a SwingTransformer basic architecture unit, a multiple feature capture unit, a proposal anchor box generation unit, and a bilinear interpolation feature aggregation unit. The output of the SwingTransformer basic architecture unit is communicatively connected to the input of the multiple feature capture unit, the output of the multiple feature capture unit is communicatively connected to the input of the proposal anchor box generation unit, and the output of the proposal anchor box generation unit is communicatively connected to the input of the bilinear interpolation feature aggregation unit.

[0094] In a preferred embodiment of the present invention, the SwinTransformer infrastructure unit is used to perform hierarchical window attention calculation on the input image and realize information interaction between adjacent windows through the Window-MSA and SW-MSA alternation mechanism.

[0095] The multi-feature capture unit is used to output feature tensors with four different resolutions: 56×56, 28×28, 14×14, and 7×7, which are used to simultaneously capture the overall outline and local detail features of Tibetan characters.

[0096] The multiple feature capture unit consists of the output layers of Stage 1 to Stage 4 of the SwinTransformer backbone network, which output feature maps with resolutions of 56×56, 28×28, 14×14, and 7×7, respectively.

[0097] The proposal anchor box generation unit is used to generate k quadrilateral proposal anchor boxes with coordinates (x1, y1, x2, y2) based on the feature map, which are used to guide subsequent feature selection.

[0098] The bilinear interpolation feature aggregation unit is used to extract features from feature tensors at different levels based on the proposed anchor frame, and to concatenate the 192 aggregated features extracted at each level into 768 aggregated features with a length and width of 7.

[0099] The bilinear interpolation feature aggregation unit specifically includes:

[0100] The feature grid partitioning subunit divides each proposed feature boundary into multiple feature grids, with the coordinate center point (xij, yij) as the unique identifier.

[0101] The weight calculation subunit calculates the feature weight wmn of four adjacent pixels (xm, yn) based on the coordinates of the center point of the feature grid.

[0102] The aggregated feature generation subunit calculates the aggregated features using a formula, and concatenates the four layers of features into a unified 768-dimensional feature representation;

[0103] The Tibetan-specific prompting mechanism module is used to generate text prompts based on the Tibetan character library and to generate window prompts by randomly cropping them, thereby enhancing the model's ability to perceive Tibetan text.

[0104] The Tibetan-specific prompting mechanism module includes a Tibetan script library construction unit, a text prompting generation unit, a window prompting generation unit, and a random prompting strategy control unit. The text prompting generation unit is integrated within the Tibetan script library construction unit. The text prompting generation unit and the window prompting generation unit are bidirectionally connected, and the Tibetan script library construction unit and the random prompting strategy control unit are bidirectionally connected.

[0105] In a preferred embodiment of the present invention, the Tibetan character database construction unit is used to construct a semantic prior knowledge base containing 3371 deduplicated characters based on VCC vertical segmentation technology.

[0106] The text prompt generation unit is used to randomly select complete semantic segments of the annotated text, convert them into character table encoding to form prompt vector I, and ensure dimensionality consistency through linear projection and zero padding.

[0107] The text prompt generation unit specifically includes:

[0108] The semantic segment selection subunit randomly selects a complete semantic segment from the N annotated texts of the instances to be identified in the current image;

[0109] The encoding conversion subunit converts the selected text into a 3371-dimensional character table encoding representation;

[0110] The dimension alignment processing subunit processes the cue vector I into a fixed-length vector by zero padding to ensure that it is consistent with the dimension of the image features.

[0111] The feature fusion calculation subunit combines text prompt features with image features according to a formula;

[0112] The window prompt generation unit is used to output the coordinates of the cropping quadrilateral through the self-attention module, randomly crop the image within the anchor frame, and the cropping area is 20%-40% of the original image area. After processing by the multilayer perceptron, the window prompt features are formed.

[0113] The window prompt generation unit specifically includes:

[0114] The attention-guided coordinate prediction subunit outputs the four coordinates of the clipped quadrilateral through a self-attention module and a linear layer;

[0115] The area control cropping subunit controls the cropping area within 20%-40% of the image area within the anchor frame based on a random number.

[0116] The local feature extraction subunit extracts features from the cropped image crop(E) to generate window prompt features that enhance spatial awareness.

[0117] The combined prompt fusion subunit simultaneously fuses text prompts, window prompts, and original image features according to a formula;

[0118] The random prompting strategy control unit is used to control the activation of the prompting mechanism according to a probability distribution of 40% no prompts, 30% text-only prompts, and 30% combined prompts;

[0119] The detection and recognition calculation module is used to encode enhanced features based on the Transformer architecture and output text detection boxes and recognition results;

[0120] The detection and recognition calculation module includes a TransformerEncoder encoding unit, a detection result output unit, and a recognition result output unit. The output end of the TransformerEncoder encoding unit is communicatively connected to the input end of the detection result output unit, and the output end of the detection result output unit is communicatively connected to the input end of the recognition result output unit.

[0121] The TransformerEncoder encoding unit is used to employ a multi-head self-attention mechanism and a feedforward network to capture the complex relationship between image and text features;

[0122] The detection result output unit is used to output the detection results of M=100 instance points through the multilayer perceptron. Each instance contains N=2 coordinate points (top left corner and bottom right corner) with a dimension of (batch, M, N, 2).

[0123] The detection result output unit specifically includes:

[0124] The confidence calculation subunit processes the Plogits value using the sigmoid function and calculates the confidence score for each instance according to the formula.

[0125] The binary classification subunit performs binary classification of text or background on M=100 proposal instances;

[0126] The coordinate regression subunit outputs two coordinate points for each instance, representing the upper left and lower right corners of the detection box, respectively.

[0127] The recognition result output unit is used to output Tibetan character recognition results with a maximum sequence length L=25 through linear projection and softmax processing;

[0128] The loss optimization module is used to integrate multiple loss functions and jointly optimize the detection and recognition tasks through gradient backpropagation.

[0129] The loss optimization module includes an algorithm matching unit, a multi-task loss calculation unit, and a gradient backpropagation unit. The output of the algorithm matching unit is communicatively connected to the input of the multi-task loss calculation unit, and the output of the multi-task loss calculation unit is communicatively connected to the input of the gradient backpropagation unit.

[0130] The algorithm matching unit is used to filter detection results based on confidence scores and calculate the minimum distance between the predicted anchor box and the real annotation.

[0131] The multi-task loss calculation unit is used to integrate GIoU loss, L1 distance loss, CTC loss and alignment loss, and the total loss function is the sum of the four.

[0132] The multi-task loss calculation unit specifically includes:

[0133] The GIoU loss calculation subunit addresses the issue of excessively large differences between predicted and labeled points in the early stages of training, resulting in an IoU of 0 and the inability to provide gradients. The calculation formula is as follows:

[0134] The L1 distance loss calculation subunit calculates the distance loss between the predicted anchor point Pi and the labeled anchor point Gj.

[0135] The CTC loss calculation subunit processes the recognition of text sequences of variable length and calculates the difference between the predicted text and the target text sequence using softmax and CTC loss functions.

[0136] The alignment loss calculation subunit dynamically learns and adjusts the prompt sampling based on the second norm loss value to enhance the alignment effect between the text and the window prompt.

[0137] The gradient backpropagation unit is used to explicitly optimize the selection process of shared aggregated features by jointly optimizing the detection and recognition tasks.

[0138] It also includes the data preprocessing and enhancement module, which includes:

[0139] The image size standardization unit is used to uniformly process the input image to a 3-channel 224×224 size;

[0140] The perspective transformation enhancement unit is used to select images from the SCUT-CTW1500 dataset, remove the original text, and then render Tibetan text.

[0141] The fusion unit is used to ensure a natural transition in color and texture between the enhanced text area and the original background.

[0142] The multilingual environment simulation unit is used to generate training samples for Tibetan-Chinese bilingual and Tibetan-English bilingual mixed scenarios.

[0143] Example

[0144] Data acquisition (multi-scale feature extraction module and Tibetan-specific prompting mechanism module running in real time):

[0145] S101. Using a mobile phone, real-world photos were taken on the streets of Lhasa. The photos mainly included road signs, shop signs, stone tablets, banners, etc. These images varied greatly in terms of shooting angle, lighting conditions, and occlusion, making detection and recognition quite difficult. Some images are attached. Figure 2 As shown;

[0146] S102. A detailed comparison of currently known Tibetan datasets in natural scenes is shown in Table 1 below:

[0147] Table 1 Comparison of existing Tibetan datasets for natural scenes

[0148]

[0149] S103. As shown in Table 1, this study systematically compares and analyzes existing datasets from the dimensions of data source geographical distribution, sample size, collection method and language coverage. The results show that the dataset constructed in this study has significant advantages in terms of total data volume and scene authenticity. The three existing comparison datasets mainly adopt horizontal perspective collection, and the number of text instances in a single image is limited. Most of them are single text targets and lack densely distributed samples of multiple texts in complex backgrounds, resulting in relatively low complexity of detection and recognition tasks. In addition, the language diversity of existing datasets is insufficient. The maximum number of languages ​​covered is the bilingual configuration of Hong Song's team, which is difficult to meet the model training needs in a multilingual environment.

[0150] S104. This study's dataset not only covers pure Tibetan text but also systematically includes Tibetan-Chinese bilingual and Tibetan-English bilingual mixed scenarios, effectively simulating the real multilingual ecological environment of Tibetan-inhabited areas. This significantly improves the dataset's ecological adaptability and application value. The dataset also encompasses Tibetan text instances under various lighting conditions and highly complex background environments, comprehensively reflecting the technical challenges faced by Tibetan text recognition and detection in real-world application scenarios. It should be noted that, to objectively evaluate the specific performance of the model in Tibetan text detection and recognition tasks, this study adopts a Tibetan-specific annotation strategy, precisely annotating only the Tibetan text regions in the images.

[0151] S105. Image Annotation: In this study, a total of 2644 Tibetan texts from 2000 images were annotated using quadrilaterals. The annotation process for each natural scene image is shown in the attached figure. Figure 3 As shown:

[0152] First, a bounding box is drawn around the Tibetan text using a long rectangle formed by connecting the coordinates of the four boundary vertices for detection and annotation. Each box contains only one line of text. Then, the text within the bounding box is identified and annotated.

[0153] S106. Data Statistics: The Natural Scene Tibetan Image Dataset (NSTD) constructed in this paper contains 2000 images, covering 2644 valid Tibetan texts. These images exhibit high diversity, demonstrating the various complex characteristics of Tibetan texts in natural scenes. The texts in the dataset include horizontal text, slanted text, and text with strong background interference. Furthermore, the Tibetan texts in the images face challenges such as uneven lighting and blurriness, which may significantly hinder character recognition. The dataset also includes multilingual text scenes, where Tibetan text coexists with texts in other languages, further increasing the complexity of the image content, as shown in the appendix. Figure 4 As shown, the images in the dataset cover these challenging scenarios, providing a rich testing foundation for subsequent text recognition research;

[0154] S107. The image is first used for feature extraction through the SwinTransformer module. This paper selects this model for the following reasons:

[0155] (1) Complexity of Tibetan character structure: Tibetan is different from Latin character languages ​​such as English. Its character structure is complex, with characters containing a variety of components. The number of vertically segmented characters can reach thousands, resulting in complex text boundary features in natural scenes. The model needs to have a strong ability to extract local features. SwinTransformer's window attention mechanism can accurately capture these local structural features.

[0156] (2) The impact of Tibetan word order characteristics on text recognition: Tibetan adopts the subject-object-verb (SOV) structure, which is different from the subject-verb-object (SVO) structure of Chinese. In SOV word order, the verb is located at the end of the sentence. This means that to fully understand the semantics of a sentence, it is necessary to process the complete information flow from the subject at the beginning of the sentence to the verb at the end of the sentence. When Tibetan text is long, there may be a large spatial distance between the subject and the verb. This places higher demands on the context modeling ability of the text recognition model. SwinTransformer, through its hierarchical window attention mechanism, can expand the receptive field layer by layer, gradually expanding from the local window to the global scope. This helps to capture the dependency relationship between characters in different positions in the text, thus better adapting to the word order characteristics of Tibetan.

[0157] S108. The specific process involves using the self-attention mechanism in the SwinTransformer and employing local windowing and hierarchical design to effectively capture spatial features in the image at different scales. The model then generates a set of ProposalAnchors to guide the model in accurately extracting the Region of Interest (ROI), i.e., regions that the model believes may contain text descriptions. Subsequently, the model uses window prompts or text prompts under random number generation to increase its attention to text-related features in the image. The joint features resulting from the prompts and feature connections are fed into the TransformerEncoder for computation. Each encoder layer consists of a multi-head self-attention mechanism and a feed-forward network to capture the complex relationships between image and text features. Finally, the TransformerEncoder outputs a feature vector, employing an instance matching strategy based on the Hungarian algorithm and edit distance to match the output with the image annotation instances, ultimately outputting the recognition and detection results.

[0158] Feature extraction (multi-scale feature extraction module runs in real time):

[0159] S201. The feature extraction section first processes the image into 3 channels, with each channel having a width and height of 224. Then, it feeds the image into a SwinTransformer for feature extraction. To improve the model's ability to extract detailed features, the model extracts the image features into four feature tensors of different sizes. The generated proposal anchor boxes are then used to guide the model in feature selection within these different feature tensors. Specifically, the model generates k proposal anchor boxes (whose coordinates are...) , , , Based on the size, features are mapped to different levels, generating proposed feature boundaries that match the feature dimensions. Each feature boundary is divided into multiple feature grids, with its coordinate center point serving as a unique identifier. Assuming a proposed feature boundary's... Okay, number The coordinates of the center point of the feature cell in the column are ( , The proposed feature boundaries are extracted using bilinear interpolation to obtain aggregated features, each with the same size. The specific calculation process is shown in equations (1) and (2).

[0160]

[0161]

[0162] in The feature weights corresponding to each feature cell. and Center point ( , The coordinates of four adjacent pixels, not the coordinates of the grid center, are the nearest points selected during bilinear interpolation.

[0163] S202. Set the number of aggregated features extracted at each level to 192, then concatenate them into 768 aggregated features with a width and height of 7. Proceed to the prompting process. Successfully apply the proposed anchor boxes to the generation of guiding features. The loss optimization of recognition and detection results will also optimize the calculation error of the proposed anchor boxes and explicitly optimize the selection process of aggregated features shared by recognition and detection.

[0164] S203. Mechanism Design: Unlike existing suggestion mechanisms (such as TESTR's general query suggestions), this study specifically designed a suggestion mechanism tailored to the linguistic characteristics of Tibetan.

[0165] Tibetan structure-aware text prompts: Considering the complex character structure of Tibetan vertical segmentation, this study performs VCC vertical segmentation on all labeled text in the dataset and removes duplicates to form a character table of size 3371. This prompt design based on the inherent character structure of Tibetan can more accurately represent the combination relationship and semantic dependency between Tibetan characters compared to general query vectors.

[0166] Window prompts based on SOV word order adaptation: In Tibetan subject-object-verb (SOV) word order, the verb is located at the end of the sentence. When Tibetan text is long, there is a large spatial distance between the subject and the verb. Window prompts use an attention mechanism to guide the cropping of the image within the anchor box (with an area of ​​20%-40%), enabling the model to focus on the key local features of the text and effectively address the challenge of long-distance Tibetan semantic dependencies.

[0167] Randomized strategy for low-resource languages: Considering the characteristics of Tibetan as a low-resource language, a probabilistic strategy of 40% no prompts, 30% text prompts, and 30% combined prompts was designed to avoid overfitting the model to prompt patterns and improve its adaptability to diverse Tibetan scenarios. The specific random prompt strategy is as follows:

[0168] ω40% probability: No hints are given; only zero-feature tensors are concatenated with features to preserve the basic capabilities of the model.

[0169] ω30% probability: Use text prompts to enhance feature representations using Tibetan semantic information;

[0170] ω30% probability: Use aligned text hints and window hints in combination to enhance spatial positioning capabilities through local visual features;

[0171] In each training iteration, a uniformly random number p between 0 and 1 is generated for each sample in the current batch. If p < 0.4, no hint is used; if 0.4 ≤ p < 0.7, only text hint is used; if p ≥ 0.7, combined hint is used.

[0172] S204. In the generation of text prompts, all labeled texts in the dataset are vertically segmented and deduplicated into a character table with a size of 3371. Let N be the number of texts to be recognized in the current image. The text model randomly selects one of the labeled texts from the N instances to be recognized and assigns it to the character table encoding. Let the complete Tibetan character table annotation vector corresponding to the selected text be the prompt vector. And fill the remaining length with zeros to ensure For a fixed length, when only text prompts are used, the feature fusion method is as shown in equation (3):

[0173]

[0174] in Features calculated using the SwinTransformer framework. This represents a zero-dimensional vector that has the same dimension as I calculated through a linear layer. This indicates that it incorporates features of text prompts;

[0175] S205. In the combined generation of aligned text and window tooltips, the model inputs the extracted image features into a self-attention module and a linear layer, and outputs the four coordinates of the cropped quadrilateral. Based on these coordinates, the internal image of the anchor box is labeled. Cropping is performed to obtain the cropped image. The cropping area is adjusted according to random numbers to 20% to 40% of the area of ​​the image within the anchor frame. After that, the cropped image features are processed by a multilayer perceptron and combined with textual feature prompts and features calculated by the SwinTransformer framework. The specific formula is shown in the following formula (4).

[0176]

[0177] As can be seen from the above, the text prompt is based on the design of the vertical segmentation character table, which enables it to distinguish similar characters. Since the character table contains 3371 deduplicated smallest representation units, the prompt vector can encode similar characters to provide prompts. The window prompt, guided by the attention mechanism, forces the model to focus on the key distinguishing parts of the characters, improving the model's sensitivity to local feature differences. At the same time, the text prompt uses linear projection and zero-padding mechanisms to ensure dimensionality consistency. After the window prompt is generated under the guidance of the attention mechanism, relevant information is fused through feature concatenation.

[0178] S206. To ensure semantic alignment of images and text during training when using both text and window prompts simultaneously, this study implements the following strategy:

[0179] Context-aware sampling: The text prompt contains a random and complete labeled text fragment to be identified, maintaining the semantic coherence between Tibetan characters;

[0180] An alignment loss function is introduced: the cue sampling is dynamically learned and adjusted based on the second norm loss value to enhance the alignment effect between the text and the window cue. The alignment loss function is calculated as shown in equation (5).

[0181]

[0182] In summary, the model successfully incorporated the features. It integrates with text and window prompts, and thanks to the role of random numbers, it avoids the model from overfitting to the prompts;

[0183] S207. The cue vector selection strategy takes into account the special characteristics of Tibetan detection and recognition: A continuous and complete annotation embedding representation is randomly selected from the labeled text as the cue vector. This method can capture the semantic relationships and structural features between Tibetan characters, thereby enhancing the model's adaptability to Tibetan semantics. The text cue scope covers the complete 3371-dimensional Tibetan character semantic feature space. The character table constructed through vertical segmentation provides refined semantic prior knowledge, which is applied to the model's semantic understanding level. The window cue scope is limited to 20%-40% of the space within the anchor box. Random pruning guided by the attention mechanism strengthens local discriminative features, which is applied to the model's spatial perception level. The quality control of the cue vector is mainly achieved through two mechanisms: firstly, the cue vector comes from high-quality labeled data, ensuring the accuracy of semantics and the region to be detected; secondly, the random number mechanism avoids the model's dependence on specific cue patterns. The implementation of this strategy gives the model multimodal perception and processing capabilities, enabling it to establish an effective cross-modal mapping relationship between visual feature extraction and language semantic understanding.

[0184] The synergistic effect of the S208, window prompts, and text prompts modules significantly enhances the model's capabilities in the following three aspects:

[0185] (1) Dataset adaptability: Through the random prompting mechanism, the model can quickly adapt to Tibetan texts in different scenarios, reducing the dependence on large-scale pre-training data, which is especially important for the recognition of low-resource languages ​​such as Tibetan.

[0186] (2) Semantic recognition and disambiguation ability: Tibetan vertical character combination (VCC) is the rule for the composition of Tibetan characters. It refers to the vertical combination of components such as superscript, base, and subscript within a single Tibetan character. For example, a character may contain components in different positions such as the head character "ར", the base character "ག", and the foot character "ས". This vertical superposition structure leads to a high degree of overlap in the feature space between different characters. Specifically, it can be explained as follows: Let the vector of the introduced prompting mechanism be... The input image is Traditional image features Difficulty in distinguishing similar characters, such as "ག" and "པ" which differ only slightly in some strokes, or "ང" and "མ" which are extremely similar in overall outline, can be addressed by redistributing attention. By separating similar characters in the feature space, the model's discriminative ability against similar characters is enhanced. The mechanism at the information theory level is as follows: Let the set of Tibetan characters in the study be... Natural scene noise is Then the recognition probability Due to severe noise pollution, a warning has been introduced. Afterwards, posterior probability Reconstructed using Bayes' theorem ,in Providing prior constraints reduces computational complexity and significantly reduces recognition entropy. In summary, this mechanism can make full use of the semantic association between characters and contextual constraint information, significantly improving the model's ability to distinguish easily confused characters, thereby maintaining high recognition accuracy and stability in challenging scenarios such as blurred character boundaries, missing strokes, or morphological variations.

[0187] (3) Spatial perception ability in complex backgrounds: Under the guidance of the attention mechanism, the window prompts enable the model to focus on the local visual features of the text by cropping the image within the anchor box, which effectively reduces the interference of highly complex backgrounds on detection and recognition. Especially in scenes with high image background complexity, this module can significantly improve detection accuracy and recognition robustness.

[0188] S209. Detection and recognition computation framework based on TransformerEncoder:

[0189] The detection and recognition computation framework uses TransformerEncoder as the feature extraction module. The features are flattened and embedded with positional encoding before being input into the Transformer framework. Multi-head self-attention is used to simultaneously focus on multiple different regions of the input features and capture the complex dependencies between these regions. Through multiple attention heads, the model can learn the correlation between features from different subspaces. A feedforward network is used to perform nonlinear transformation on the features at each position. This includes two fully connected layers with nonlinearity introduced between them through an activation function. This process allows the features to better capture complex semantic information. The feedforward network processes the features after each attention layer. By increasing the depth and nonlinear transformation, more discriminative features are further extracted. The computation results are then processed by a multilayer perceptron to obtain the final detection and recognition results.

[0190] In the loss function calculation section, the number of proposal instances is set to... The number of each instance point is uniformly set to ( The maximum length for instance recognition is set to... The anchor box dimension of the detection result is (batch, , ,2), In this study, the coordinates of the instance points are set as the coordinates of the upper left corner and the lower right corner of the anchor frame, respectively;

[0191] The test results include the number of proposals. Binary classification (text or background) is performed on the linear projection of the vertices of a rectangle. During model testing, this study takes the number of proposed instances as a function of binary classification. The average of the results is used as the confidence score for each instance. The calculation formula is shown in equation (6):

[0192]

[0193] The matching strategy for the detection results is based on the Hungarian algorithm. First, detection results with excessively low confidence scores are filtered out. Then, the proposed anchor boxes output in each batch are matched with... Match each instance sequentially, in Choose the proposal with the smallest average distance from the labeled point. The anchor frame detection results are set as follows: Anchor frame proposal point Anchor frame annotation Distance metric To obtain the corresponding instance The resulting detection result identifier, k, represents the number of features proposed for detection for each image. The specific calculation process is shown in equations (7) and (8):

[0194]

[0195]

[0196] To address the issue of insufficient model training in the early stages leading to a large discrepancy between predicted and labeled points, resulting in an IoU (Intersection over Union) of 0 and the inability to provide gradients, a variant of the intersection-over-union loss function between predicted anchor boxes and ground truth labels is chosen for anchor box detection loss. Compare it with the predicted anchor point anchor points Distance loss function The detection loss function is obtained by summing the results; the specific calculation formulas are shown in equations (9) and (10):

[0197]

[0198]

[0199] in This refers to the intersection-union ratio of the actual anchor frame annotations and the predicted anchor frames. It is the area of ​​the smallest bounding box that includes both the predicted bounding box and the ground truth bounding box. It is the size of the union of the actual anchor frame area and the predicted anchor frame area;

[0200] S210. Regarding the recognition loss, the predicted text is matched with the labeled text instances based on edit distance, and optimized using the connection-based temporal classification CTC loss function. This can handle text sequences of indefinite length. The advantage of CTC loss is that it can effectively solve the problem of mismatch between the length of the text sequence and the image feature, making it suitable for free-form text recognition tasks. Specifically, the sequence predicted by the model is first... conduct The process involves obtaining the probability distribution of each character at each time step, followed by applying the CTC loss function. To calculate the difference between the predicted text and the target text sequence, and to avoid the impact of padding on the loss, a dynamic target length calculation method was used in the study. That is, padding was filtered out based on the actual text length of each instance. The specific calculation formulas are shown in equations (11) and (12):

[0201]

[0202]

[0203] in By in the target sequence Insert whitespace characters and allow all possible paths to generate duplicate characters;

[0204] In summary, the total loss of model detection and recognition is the anchor box distance loss. Anchor frame intersection and loss , Alignment loss function The sum of the four.

[0205] Specific experiments (Tibetan-specific prompting mechanism module, detection and recognition calculation module, and loss optimization module running in real time):

[0206] In the S301 experiment, the algorithm was implemented using the deep learning framework PyTorch 1.8, and the experiment was conducted on two GeForce RTX 3090 GPUs. The optimizer used was Adam, the batch size was set to 16, and the initial learning rate was set to 0.002. After training for 600 epochs, the learning rate was reduced to one-tenth of the initial value. Natural images containing Tibetan text were uniformly sized to 224*224 pixels, and the training lasted for 1000 epochs. The natural scene Tibetan text dataset contained 2000 images and 2644 Tibetan texts. Regarding the construction of the training set, the study used image text erasure, perspective transformation, and text rendering techniques to randomly select a portion of images from the SCUT-CTW1500 dataset and used DBNet++ text detection technology to accurately locate and erase the images. The original text region was first processed, then color and texture filling was performed based on the edge texture of the erased region to generate a text-similar color region. Semantically complete text fragments were randomly selected from a Tibetan text library. A perspective transformation matrix was calculated based on the vertex coordinates of the original text region and adaptively adjusted according to the target region size. Text was then selected from a Tibetan font library for rendering. A Poisson fusion algorithm was used to ensure that the transformed text maintained consistency with the shape, angle, and spatial layout of the original text region. In this way, the team combined 1750 original images with the newly synthesized text images, expanding the dataset to nearly 30,000 images. These images have rich background variations, simulating the presentation of Tibetan text in various natural scenes, increasing the diversity of the training data. Samples of some of the synthesized data are attached. Figure 5 As shown, for the test set, 250 images from the original dataset were selected for testing, and the test set contained 360 texts.

[0207] Poisson fusion is performed using OpenCV's default parameters. The Tibetan font library includes 10 common Tibetan fonts such as Magpie and Qiong to simulate the diversity of fonts in real-world scenarios.

[0208] S302. Evaluation Metrics: In the detection task, setting the intersection-union ratio (IU) threshold to 0.5 means that for true quadrilateral bounding boxes... and predictive anchor boxes If the intersection-to-union ratio of the two is satisfied If the value is greater than 0.5, then it is considered... and If a match is successful, record it as a true example; if... or If a match is consistently unsuccessful, it is recorded as a false negative and a false positive, respectively. Finally, the precision, recall, and F1 score are calculated based on the matching data as evaluation indicators for text detection. The specific calculation formulas are shown in equations (13)-(15):

[0209]

[0210]

[0211]

[0212] Where P, R, and F1 represent precision, recall, and F1 score, respectively, and TP, FP, and FN represent the number of true positives, false positives, and false negatives. In the recognition task, the accuracy of text recognition is evaluated on the basis of the correct detection box. The recognition accuracy acc is the percentage of the number of correctly predicted characters to the number of characters. This paper performs vertical segmentation and deduplication on the labeled recognition text, adds common punctuation marks, and constructs a character table of size 3371. This study also constructs a strong dictionary of size 2644 based on the dataset text, which contains all the recognition words in the dataset and is a word list with complete semantics. This study conducted comparative experiments with and without a dictionary.

[0213] S303. Experimental Results and Analysis: The four experiments in this paper were conducted under the same hardware and software parameters. The cross-union ratio was set to 0.5. The accuracy, recall and F1 score of the detection were recorded. In the recognition part, None represents the recognition accuracy under the condition of no dictionary and Full represents the recognition accuracy using a strong dictionary.

[0214] S304, Experiment 1: On the end-to-end detection and recognition task, a comparative experiment was conducted with the currently publicly available Tibetan text detection and recognition models in natural scenes. This study selected five baseline models. The model selection was mainly based on the following considerations: CNN+RNN+Attn and CNN+BiLSTM+Attn were selected as representatives of basic deep learning methods to verify the advantages of the Transformer architecture compared with traditional methods. In the current mainstream detection and recognition framework, DBNET+CRNN

[19] was selected as the representative of the currently widely used two-stage detection and recognition method. End-to-end methods based on Transformer, such as TESTR and SwinTextSpotter, were selected. TESTR adopted a query prompt mechanism to unify text detection and recognition tasks. The purpose of selecting this model is to verify whether the text prompt and window prompt mechanism designed for Tibetan text in this paper is superior to other general models that adopt prompt mechanisms in low-resource language recognition tasks. SwinTextSpotter considers common modules, that is, it is also based on the SwinTransformer architecture, which is convenient for verifying the effectiveness of the feature extraction method in this paper. The relevant experimental results are attached. Figure 7 As shown;

[0215] According to the experimental results, in terms of detection accuracy, the proposed method achieved a precision of 85.35%, which is 5.13 percentage points higher than DBNET+CRNN. ​​This indicates that by introducing the SwinTransformer and TransformerEncoder architecture, the model can more accurately identify Tibetan characters and reduce false positives. In terms of recall, the proposed model is 7.51 percentage points higher than DBNET+CRNN. ​​This improvement reflects the model's advantage in detecting missed characters, capturing more target characters and thus improving the model's overall performance. As a comprehensive evaluation indicator, the F1 score is 6.40 percentage points higher than DBNET+CRNN and 3.19 percentage points higher than the SwinTextSpotter model. This significant improvement shows that the proposed method has a clear advantage in balancing precision and recall, and can better handle the challenges in Tibetan character detection and recognition.

[0216] In terms of character recognition accuracy, the average recognition accuracy of this study with and without a strong dictionary is 14.53, 11.87 and 6.94 percentage points higher than the first three baseline models, respectively, and 2.58 percentage points higher than the SwinTextSpotter model. This result shows that the method of introducing TransformerEncoder and prompts in this paper effectively improves the recognition accuracy of Tibetan characters.

[0217] The purpose of choosing TESTR as the comparison model is to verify the advantages of the prompting mechanism designed for Tibetan language characteristics in this paper compared with the general query prompting mechanism. TESTR uses a unified learnable query vector for text detection and recognition, while the text prompts in this paper are constructed based on the Tibetan VCC character table, and the window prompts are designed for the word order characteristics of Tibetan SOV. This targeted prompting mechanism design makes the method in this paper improve the F1 score by 3.22 percentage points and the average recognition accuracy by 3.00 percentage points compared with TESTR on the Tibetan recognition task.

[0218] In summary, the method proposed in this study outperforms existing mainstream models in all major metrics, fully demonstrating its effectiveness and advantages in Tibetan text detection and recognition tasks in natural scenes.

[0219] S305, Experiment 2: This study also verifies the effectiveness of the proposed text and window prompt-based model performance enhancement method through ablation experiments. Experiments were conducted on the NSTD Tibetan text detection and recognition dataset constructed in this paper, using models with and without text and window prompts respectively, to verify the performance of the proposed method in the two major tasks of natural scene text detection and recognition. Specific experimental results are shown in Table 2 below:

[0220] Table 2 Ablation Experiment Results

[0221]

[0222] As shown in Table 2, without prompts, the model's precision was 84.10, recall was 79.99, and F1 score was 81.99. When text prompts were introduced, precision, recall, and F1 score increased by 0.46, 0.18, and 0.33 percentage points, respectively, and the average accuracy with and without a dictionary increased by 0.635 percentage points. This indicates that the text prompt function improved the model's detection and recognition performance to a certain extent. When both text prompts and window prompts were enabled, compared with no prompts, the F1 score and the average accuracy with and without a dictionary increased by 1.67 and 2.57 percentage points, respectively. This shows that the combination of text prompts and window prompts effectively enhanced the model's performance, and their synergistic effect helped improve the overall detection and recognition results.

[0223] To better demonstrate the improvement of the model's ability to identify and disambiguate similar characters in Tibetan script recognition by the suggestion mechanism in this paper, the study statistically analyzed the changes in the recognition confusion error rate of the most frequently occurring similar characters in the ablation experiment, that is, the ratio of character recognition confusion error to the number of times the character appears. The specific statistical results are shown in Table 3 below.

[0224] Table 3 shows the results of the disambiguation ablation experiment.

[0225]

[0226] As shown in Table 3, the prompting mechanism significantly improved the recognition of similar-looking characters. The average error rate of the five pairs of similar-looking characters decreased from 17.40 before the prompts were used to 8.19 after the prompts were used, with an average reduction of 9.21 percentage points and a relative improvement rate of 53 percentage points. The degree of improvement varied significantly among different characters, with the character “བ” showing the most significant improvement, with the error rate decreasing from 29.33 to 10.88, a reduction of 18.45 percentage points. Further analysis of the relationship between similarity and improvement revealed that the prompting mechanism also showed significant improvement for characters with high similarity, such as “ད” and “ང”. This indicates that the character table encoding in the text prompts and the local feature enhancement mechanism of the window prompts can effectively capture subtle morphological differences. The experimental results fully demonstrate the effectiveness of the prompting mechanism proposed in this paper in solving the problem of recognizing similar-looking characters in Tibetan, and provide an effective technical path for character-level fine recognition in low-resource languages.

[0227] S306, Experiment 3: To verify the effectiveness of the two core architectural components, SwinTransformer and TransformerEncoder, this study conducted an ablation experiment on the architectural components, in which SwinTransformer was replaced with ResNet-50. As a feature extraction module, the TransformerEncoder is replaced with a traditional CNN. For comparison, it should be noted that this paper uses a linear layer to align the parameter dimensions of the computation and feature fusion process of the replacement component suggestion mechanism with the proposed model, ensuring the effectiveness of the suggestion method. Specific experimental results are shown in Table 4:

[0228] Table 4 Ablation Experiment Results of Architecture Components

[0229]

[0230] The experimental results show that, compared with ResNet-50+CNN and ResNet-50+TransformerEncoder, the F1 score improved from 73.80 to 77.60 after introducing TransformerEncoder, an improvement of 3.80 percentage points. The recognition accuracy improved from 51.87 and 56.44 to 71.00 and 79.01 with and without a dictionary, respectively, an average improvement of 20.85 percentage points. This verifies the significant advantages of TransformerEncoder over CNN models in capturing sequence dependencies and long-range semantic modeling.

[0231] Effectiveness of SwinTransformer: Comparing the results of ResNet-50+CNN and SwinTransformer+CNN, SwinTransformer, as a feature extraction module, improved the F1 score from 73.80 to 76.32, an improvement of 2.52 percentage points. The average recognition accuracy improved by approximately 12.13 percentage points. This demonstrates its ability to progressively expand the receptive field through its hierarchical window attention mechanism, capturing the dependencies between characters at different positions in the text, and thus better adapting to the word order characteristics of Tibetan.

[0232] The synergistic effect of the proposed method's architecture: Compared to the basic ResNet-50+CNN architecture, the complete SwingTransformer+TransformerEncoder architecture improves the F1 score by 9.86 percentage points, and the recognition accuracy improves by 21.03 percentage points and 25.48 percentage points with and without a dictionary, respectively, with an average improvement of 23.255 percentage points. This significant improvement fully demonstrates the synergistic effect of the two core components. SwingTransformer provides powerful visual feature extraction capabilities, while TransformerEncoder enhances sequence modeling and contextual understanding capabilities. The combination of the two provides the optimal architecture design for Tibetan text detection and recognition tasks.

[0233] From the perspective of individual contributions, the introduction of TransformerEncoder brought greater performance improvement, which shows that in Tibetan script recognition tasks, sequence modeling and contextual understanding are more critical than simple visual feature extraction, which is consistent with the characteristics of Tibetan characters having complex structure and special word order.

[0234] S307, Experiment 4: This paper conducts a detection comparison experiment under different cross-union ratio (CUI) thresholds to test the model's detection performance at high accuracy. The comparison results are attached. Figure 8 As shown;

[0235] The experimental results show that our model exhibits stronger detection capabilities and better balanced performance compared to other models at multiple intersection-union (IU) thresholds. At an mIoU threshold of 0.6, our model outperforms the CNN+RNN+Attn model in precision, recall, and F1 score by 18.0%, 1.5%, and 9.6%, respectively; compared to the CNN+BILSTM+Attn model, it outperforms by 13.5%, 3.4%, and 7.7%, respectively; and compared to the DBNET+CRNN model, it improves precision by 6% and F1 score by 5%. This indicates that our model demonstrates more balanced performance at this threshold, especially with a significant lead in F1 score. At an mIoU threshold of 0.7, although the performance of all models decreases... However, our model still outperforms other models. Compared to the SwinTextSpotter model, our model achieves 2.3% higher accuracy, 0.9% higher recall, and a 1.2% higher F1 score. At higher intersection-union (IU) thresholds, the performance of other models drops sharply, while our model maintains stable and strong detection capabilities. When the threshold is further increased to mIoU=0.8, our model still performs superiorly, achieving 2.8% higher accuracy and a 2.5% lower F1 score compared to SwinTextSpotter. This demonstrates that despite the greater difficulty of high IU tasks, our model maintains a significant advantage in recall and F1 score, exhibiting strong adaptability, particularly in capturing relevant features and information for high IU tasks.

[0236] To more intuitively demonstrate the performance improvement of the suggestion method presented in this paper, this study conducted a visual comparison of the results between the model using the suggestion method and the SwinTextSpotter model, as shown in the attached figure. Figure 6As shown, the incorrectly identified characters are marked in red in this study. The visualized recognition results indicate that the model without prompting methods shows a decrease in its sensitivity to text features in multi-instance images with complex backgrounds. It performs poorly in recognizing small characters such as "་" and "།" in Tibetan text at image edges, and its ability to define text edges is also low. Furthermore, it has poor ability to distinguish between Tibetan characters with similar features such as "ང" and "ད" and "པ" and "བ". This suggests that the model's detection and recognition performance is affected by noise in highly complex natural scenes. However, the addition of text prompts... After the suggested method was used, the model's recognition accuracy improved, but its ability to define boundaries remained unclear, with three errors in the detection and recognition of edge text. After using both text and window models simultaneously, the model's recognition and detection capabilities were further improved, effectively enhancing the model's generalization ability in detection and recognition under complex background conditions. The recognition accuracy for similar characters was significantly improved. It was also found that the SwinTextSpotter model recognized "པ" as "བ" and "ཆ" as "ཚ". Compared with this model, the recognition capabilities of the model presented in this paper are superior in all aspects.

[0237] However, further analysis revealed that the model, relying solely on anchor frames for detection and recognition, has limitations in handling deformed instances such as curves. Because the anchor frames cannot effectively enclose these deformed instances, the feature selection portion fed into the recognition module becomes biased, affecting recognition accuracy. To address this issue, future research will combine polygon points and Bézier curves to further enhance the model's adaptability to deformed characters and curved text. This improvement aims to more accurately fit the morphological changes of Tibetan text in natural scenes, thereby achieving more efficient detection and recognition in complex backgrounds and deformed environments. Furthermore, the combination of polygons and Bézier curves not only more accurately encloses target characters but also improves the ability to capture details, further enhancing the model's robustness and accuracy in practical applications.

[0238] S4. In summary, the main improvements of this invention can be summarized into the following three points:

[0239] (1) Dataset Construction: The first large-scale Tibetan image dataset of natural scenes was constructed. Real street scene images were collected in Lhasa, mainly including road signs, banners, and shop signs. A total of 2,000 images were labeled using quadrilateral anchor boxes, containing 2,644 corresponding Tibetan texts. More than 500 images have at least two instances of Tibetan text to be identified, providing a solid data foundation for Tibetan text detection and recognition research.

[0240] (2) End-to-end detection and recognition method: An end-to-end Tibetan text detection and recognition architecture based on SwinTransformer and TransformerEncoder is proposed. The hierarchical window attention mechanism of SwinTransformer is particularly suitable for handling the complex structure and long-distance dependency of Tibetan text. Its multi-scale feature extraction capability is achieved through four feature layers with different resolutions (56×56, 28×28, 14×14, 7×7), which enables the model to capture the overall outline and local detail features of characters at the same time. For example, "པ" and "བ" are similar in the whole but have different local strokes. Through the Window-MSA and SW-MSA shift window alternation mechanism, it ensures the information interaction between adjacent windows and avoids the loss of character boundary features. This is crucial for distinguishing boundary-sensitive characters such as "ང" and "ད". In terms of Tibetan text recognition method, this paper uses a Tibetan text recognition framework based on temporal features.

[0241] (3) Innovative prompting mechanism: A random prompting mechanism combining text prompts and window prompts was designed for the word order and semantic characteristics of Tibetan, which effectively improved the model’s generalization ability for text detection and recognition in natural scenes. It performed particularly well when dealing with complex backgrounds and similar character features. In terms of Tibetan recognition method, this paper also adopted a Tibetan recognition framework based on temporal features, which further improved the recognition accuracy.

[0242] To address the low detection and recognition accuracy caused by the scarcity of Tibetan natural image data, excessive background noise in natural scenes, and similar features of Tibetan characters, a public dataset for Tibetan detection and recognition in natural scenes (NSTD) was constructed. An end-to-end Tibetan detection and recognition method was proposed, employing the SwinTransformer as the feature extraction module, combined with the TransformerEncoder architecture, and innovatively introducing window and text prompting mechanisms to construct a recognition module based on temporal features. Experimental results show that the proposed model achieves an F1 score of 83.66% and a character recognition accuracy of 81.92% on the NSTD dataset. Furthermore, ablation experiments validated the effectiveness of the prompting method: compared to the baseline model without prompting, the model using prompting improved precision, recall, F1 score, and dictionary-based average recognition accuracy by 1.25%, 1.98%, 1.67%, and 2.57%, respectively, fully demonstrating the effectiveness of the proposed method in end-to-end Tibetan detection and recognition tasks.

[0243] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0244] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, equipment, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0245] The modules for multi-scale feature extraction, Tibetan-specific prompting mechanism, detection and recognition calculation, and loss optimization may or may not be physically separate. The components displayed as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0246] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0247] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0248] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end natural scene oriented Tibetan text detection and recognition system, characterized in that, The detection and recognition system comprises a multi-scale feature extraction module, a Tibetan special prompt mechanism module, a detection and recognition calculation module and a loss optimization module, the multi-scale feature extraction module is used for multi-scale feature extraction module, and is used for extracting multi-scale feature representation from an input image; The Tibetan special prompt mechanism module is connected with the multi-scale feature extraction module, and is used for generating text prompts and window prompts to enhance the feature representation; The detection and recognition calculation module is connected with the Tibetan special prompt mechanism module, and is used for text detection and recognition based on the enhanced feature; The loss optimization module is connected with the detection and recognition calculation module, and is used for calculating multi-task loss and optimizing model parameters; Wherein the detection and recognition system realizes end-to-end Tibetan text detection and recognition through the cooperative work of each module, and the staff enters the inside of the detection and recognition system by logging in the loss optimization module, so as to view the real-time running data of the multi-scale feature extraction module, the Tibetan special prompt mechanism module and the detection and recognition calculation module one by one.

2. The end-to-end Tibetan text detection and recognition system for natural scenes according to claim 1, wherein: The multi-scale feature extraction module is used for hierarchical feature extraction of the input image, and outputs feature maps of multiple resolutions; The Tibetan special prompt mechanism module is used for generating text prompts based on the Tibetan character library, and generating window prompts through random cropping to enhance the perception ability of the model to Tibetan text; The detection and recognition calculation module is used for encoding the enhanced feature based on the Transformer architecture, and outputting the text detection box and the recognition result; The loss optimization module is used for integrating multiple loss functions, and jointly optimizing the detection and recognition tasks through gradient back propagation.

3. The end-to-end natural scene oriented Devanagari text detection and recognition system as claimed in claim 1, wherein: The multi-scale feature extraction module comprises a SwinTransformer basic architecture unit, a multiple feature capture unit, a proposal anchor box generation unit and a bilinear interpolation feature aggregation unit, the output end of the SwinTransformer basic architecture unit is in communication connection with the input end of the multiple feature capture unit, the output end of the multiple feature capture unit is in communication connection with the input end of the proposal anchor box generation unit, and the output end of the proposal anchor box generation unit is in communication connection with the input end of the bilinear interpolation feature aggregation unit.

4. The end-to-end Tibetan text detection and recognition system for natural scenes according to claim 3, wherein: The SwinTransformer basic architecture unit is used for hierarchical window attention calculation of the input image, and realizes information interaction between adjacent windows through an alternating mechanism of Window-MSA and SW-MSA; The multiple feature capture unit is used for outputting feature tensors of four different resolutions of 56x56, 28x28, 14x14 and 7x7; The proposal anchor box generation unit is used for generating k quadrilateral proposal anchor boxes according to the feature map, which is used to guide the subsequent feature selection; The bilinear interpolation feature aggregation unit is used for feature extraction in different level feature tensors according to the proposed anchor frame, and 192 aggregated features of each level are spliced into 768 aggregated features with a length and width of 7.

5. The end-to-end natural scene oriented Devanagari text detection and recognition system as claimed in claim 1, wherein: The Tibetan special prompt mechanism module includes a Tibetan character library construction unit, a text prompt generation unit, a window prompt generation unit, and a random prompt strategy control unit. The text prompt generation unit is integrated in the Tibetan character library construction unit. The text prompt generation unit and the window prompt generation unit are bidirectionally connected. The Tibetan character library construction unit and the random prompt strategy control unit are bidirectionally connected.

6. The end-to-end Tibetan text detection and recognition system for natural scenes according to claim 5, wherein: The Tibetan character library construction unit is used to construct a semantic prior knowledge base containing 3371 de-duplicated character blocks based on the VCC longitudinal segmentation technology; The text prompt generation unit is used to randomly select a complete semantic segment of the labeled text, convert it into a character block table code to form a prompt vector I; The window prompt generation unit is used to output a clipping quadrilateral coordinate through a self-attention module to randomly clip the image in the anchor frame; The random prompt strategy control unit is used to control the prompt mechanism activation according to a probability distribution of 40% no prompt, 30% only text prompt, and 30% combined prompt.

7. The end-to-end Tibetan text detection and recognition system for natural scenes according to claim 1, wherein: The detection and recognition calculation module includes a TransformerEncoder encoding unit, a detection result output unit, and a recognition result output unit. The output end of the TransformerEncoder encoding unit is communicatively connected to the input end of the detection result output unit. The output end of the detection result output unit is communicatively connected to the input end of the recognition result output unit; The TransformerEncoder encoding unit is used to adopt a multi-head self-attention mechanism and a feedforward network to capture the complex relationship between image and text features; The detection result output unit is used to output a detection result of M=100 instance points through a multi-layer perception machine; The recognition result output unit is used to output a Tibetan character recognition result of a maximum sequence length L=25 through linear projection and softmax processing.

8. The end-to-end Tibetan text detection and recognition system for natural scenes according to claim 7, wherein: The loss optimization module includes an algorithm matching unit, a multi-task loss calculation unit, and a gradient backpropagation unit. The output end of the algorithm matching unit is communicatively connected to the input end of the multi-task loss calculation unit. The output end of the multi-task loss calculation unit is communicatively connected to the input end of the gradient backpropagation unit; The algorithm matching unit is used to filter the detection result based on a confidence score and calculate the minimum distance matching between the predicted anchor frame and the real labeled anchor frame; The multi-task loss calculation unit is used to integrate GIoU loss, L1 distance loss, CTC loss, and alignment loss. The total loss function is the sum of the four losses; The gradient back propagation unit is used to optimize the selection process of the shared aggregated features by jointly optimizing the detection and recognition tasks. 9.The natural scene oriented end-to-end Devanagari text detection and recognition system according to claim 1, wherein: The data preprocessing and enhancement module is also included, which comprises: The image size normalization unit is used to uniformly process the input image into a 3-channel 224x224 size; The perspective transformation enhancement unit is used to select images from the SCUT-CTW1500 dataset; The fusion unit is used to ensure that the enhanced text region and the original image background are naturally transitioned in color and texture; The multi-language environment simulation unit is used to generate training samples of Tibetan-Chinese bilingual and Tibetan-English bilingual mixed scenes.

10. The end-to-end natural scene oriented Devanagari text detection and recognition system as claimed in claim 9, wherein: The hardware running environment of the system includes: Processor unit: two GeForce RTX3090 graphics cards; Software framework: PyTorch1.8 deep learning framework; Optimization configuration: Adam optimizer, batch size 16, initial learning rate 0.002, training 1000 epochs, learning rate decay at 600th epoch; Inference configuration: detection intersection over union threshold 0.5, post-processing mechanism based on confidence score and non-maximum suppression.

Images