Southeast Asia language scene text image recognition method based on multi-scale feature fusion

Abstract

The invention provides a Southeast Asia language scene text image recognition method based on multi-scale feature fusion, and belongs to the field of natural language processing. According to the invention, the model can identify the Southeast Asia language text more accurately in a complex scene. The method comprises the steps of multi-scale feature extraction, progressive cross-scale interaction and adaptive feature fusion. Specifically, firstly, text image features of different scales are extracted through a multi-scale encoder, an edge feature map is introduced to enhance representation capability, and complementary information of a text is captured; secondly, further enhancing the features of each scale by using progressive cross-scale interaction; and then, fusing the features of different scales to generate final text feature representation. And finally, accurately decoding the Southeast Asia language characters through a text decoder. The method provided by the invention is excellent in performance in a Southeast Asia language text recognition task, and the recognition accuracy is obviously improved while the model parameter quantity is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a Southeast Asian language scene text image recognition method based on multi-scale feature fusion BACKGROUND

[0002] Text image recognition in natural scenes is an important research direction in the field of computer vision, and has wide application value, such as photo translation, image information retrieval, automatic driving, etc. The existing recognition methods have achieved good recognition effect on Southeast Asian language text images in most real scenes, but in actual application, the text images in natural scenes are often affected by various interference factors, such as geometric deformation of the text, such as inclination, distortion, rotation, and quality problems such as low image resolution, uneven illumination, complex background, artistic font, etc. These factors limit the model in the edge fine-grained feature extraction of the text image, especially in processing tone marks, vowel extension symbols and other small target characters, which is prone to recognition errors or omissions, thereby affecting the overall recognition accuracy and robustness. The existing text recognition model usually relies on large-scale high-quality data set for training, and has weak robustness to damaged or low-quality images. In view of this problem, many studies have been devoted to improving the text recognition ability of the model in complex scenes. For example, a super-resolution network module is introduced, which preprocesses the text image before encoding the visual features to improve the resolution of the blurred image; by integrating the attention mechanism into the thin plate spline transformation, the control points are dynamically generated to better correct and recognize irregular text. Although these methods improve the text recognition accuracy in complex scenes to some extent, they mainly focus on improving the overall quality of the image, and do not model the local features of the text characters. In addition, these methods do not fully utilize the fine-grained visual features extracted from the image, resulting in insufficient representation of small target characters such as superscripts, tone marks, vowel extension symbols and other small target characters when processing Southeast Asian language text, and further causing recognition errors due to the lack of key edge features. SUMMARY

[0003] The technical problem solved by the application is that the Southeast Asian language scene text image recognition method based on multi-scale feature fusion is proposed to solve the problem of recognition errors or omissions when recognizing small target characters such as tone marks and vowel extension symbols in Southeast Asian language text images; the Southeast Asian language scene text image features of multiple scales are fused to improve the recognition effect of Southeast Asian language scene text images.

[0004] The technical solution of the application is: a Southeast Asian language scene text image recognition method based on multi-scale feature fusion, comprising the following steps:

[0005] Step 1, Southeast Asian language image preprocessing:

[0006] The input Southeast Asian language text image is denoised and the height of the image is unified, and the width is adaptively adjusted according to the size of the original image.

[0007] Step2, through the Southeast Asian language image multi-scale feature extraction module, a multi-scale feature extraction method is adopted, and a multi-scale encoder is used to capture the global features and local features of the text image respectively.

[0008] Step3, through the Southeast Asian language image progressive cross-scale interaction module, an progressive cross-scale interaction mechanism is introduced, and through a recursive cross-attention structure, the feature interaction between different scales is enhanced.

[0009] Specifically, the interaction between character details and overall structure in Southeast Asian language text is captured, a cross-scale Transformer attention mechanism is designed, and the feature representation of the current scale is optimized by interacting with the feature information of the adjacent scale.

[0010] Step4, through the adaptive fusion module of Southeast Asian language image multi-scale features, after optimizing the features of the current scale, an adaptive fusion mechanism is adopted, and on the basis of multi-scale feature extraction and cross-scale interaction, the features of different scales are weighted and fused through a dynamic weighting mechanism.

[0011] Through dynamic weighting of features of each scale, the contribution of each scale can be automatically adjusted according to the importance of each scale feature in recognition, and finally a more rich and accurate text feature representation, i.e. the final image visual feature, is generated.

[0012] Step5, through the Southeast Asian language character recognition module, after obtaining the final image visual feature, a four-layer Transformer decoder is adopted, a multi-head attention mechanism is introduced to enhance the modeling ability of complex relationships between encoded features and improve the alignment accuracy of output characters and encoded features, and the final image visual feature is mapped to the corresponding Southeast Asian language character sequence.

[0013] Further, the specific steps of Step2 are:

[0014] Step2.1, given a text image I∈R H×W×3 where H and W are the height and width of the image respectively, a lightweight convolutional network EfficientNet is used as the backbone network for feature map extraction, and the original image features of the image are preliminarily extracted;

[0015] Step2.2, introduce edge information, apply a Canny edge detector to extract edge features of the text image I to generate an edge map E∈R H×W×1wherein the value of the edge pixel is 1 and the value of the non-edge pixel is 0;

[0016] Subsequently, the edge map E is also subjected to feature extraction using a lightweight convolutional network EfficientNet, so as to further capture the relationship between the edge pixels and generate edge features conducive to text recognition;

[0017] Step 2.3, in the output feature map of the backbone network and the edge map, adaptive 2D position encoding is added respectively, so as to help the model to better locate and align the text information in the image in space; the specific calculation formula is as follows:

[0018]

[0019]

[0020] wherein p represents the pixel position, i is the dimension index, f is the feature dimension, TE x (p,2i) is the feature map after adding position encoding, TE x (p,2i+1) is the edge map after adding position encoding, and canny represents the binary image after canny algorithm;

[0021] Step 2.4, by recursively downsampling the original image features, image features are extracted on three scales F0∈H / 4×H / 4×D, F1∈H / 8×H / 8×D, F2∈H / 16×H / 16×D, in this way, image detail information and global structure at different scales are gradually captured;

[0022] Step 2.5, in order to further improve the feature representation of each scale, a self-attention module MSA based on Transformer is used to refine the features of each scale, and the specific calculation formula is as follows:

[0023]

[0024] wherein F i represents the i-th scale feature map, respectively represent the weight matrix of Q, K and V at different scales, FFN represents the feedforward network, and MSA represents the multi-head self-attention; through this multi-level feature extraction method which fuses local and global information, text features with rich details can be effectively extracted.

[0025] Further, the Step 3 comprises:

[0026] Step 3.1, by using a progressive cross-scale interaction mechanism to enhance the feature interaction between different scales, the interaction between character details and overall structure in the Southeast Asian language text is captured.

[0027] Specifically, assuming the feature representation of the i-th layer is F i , the improved feature representation is obtained by interacting with the feature F i-1 of the previous scale through the cross-scale attention module; the interaction process is described by the following formula:

[0028]

[0029] Where CA represents the cross-attention function, F i-1 and F i are the features of adjacent scales, and σ is the scaling factor; the progressive cross-scale interaction mechanism enables the model to enhance the recognition ability of Southeast Asian language characters according to the relationship between the global features of low frequency and the local features of high frequency.

[0030] Further, the specific steps of Step4 are as follows:

[0031] Step4.1, an adaptive fusion mechanism is used to weight and fuse the features of different scales; by dynamically weighting the features of each scale, the importance of each scale in recognition can be automatically adjusted to adjust the contribution of each scale, and finally a more rich and accurate text feature representation, i.e. the final image visual feature F, is generated; the feature fusion is realized by the following formula:

[0032] A=Sigm(W0[F′0,F′1,F′2])

[0033] F=A⊙(W1[F′0,F′1,F′2])

[0034] Where [F′0,F′1,F′2] represents the splicing of features of different scales, W0 and W1 are learned weight matrices, Sigm is the sigmoid activation function, and ⊙ represents the element-level multiplication operation.

[0035] Through the adaptive fusion mechanism, the model can find the most useful information in the multi-scale features and obtain the final image visual feature through weighted fusion.

[0036] Further, the specific steps of Step5 are as follows:

[0037] Step5.1, after obtaining the final image visual feature F, the final image visual feature F is passed through the multi-head attention mechanism to enhance the modeling ability of the complex relationship between the encoded features and improve the alignment accuracy of the output characters and the encoded features; finally, the cross-entropy loss function is used as the optimization function to map to the corresponding Southeast Asian language character sequence, and the formula of the optimization function is:

[0038]

[0039] wherein M represents an input Southeast Asian language image, θ represents a set of learnable parameters of a recognition model, and y t represents a true label corresponding to the t th feature sequence;

[0040] The loss function guides the model training by minimizing the difference between the predicted distribution and the true label, thereby improving the recognition accuracy.

[0041] The application also provides a Southeast Asian language scene text image recognition system based on multi-scale feature fusion, which comprises a module for executing the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0042] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0043] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0044] The application also provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0045] The application has the following beneficial effects:

[0046] 1. The application extracts text image features of different scales through a multi-scale encoder, and introduces an edge feature map to enhance the representation ability and capture complementary information of the text.

[0047] 2. The application further enhances the features of each scale using progressive cross-scale interaction, then fuses the features of different scales to generate the final text feature representation, and accurately decodes Southeast Asian language characters through a text decoder.

[0048] 3. The method proposed by the application performs excellently in Southeast Asian language text recognition tasks, and the recognition accuracy is also significantly improved while reducing the number of model parameters. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The Southeast Asian language scene text image recognition method based on multi-scale feature fusion provided by the embodiments of the application is shown in the flowchart.

[0050] Figure 2 The method flowchart in the application is shown.

[0051] Figure 3 These are sample test results for low-quality image scenarios in embodiments of the present invention. Detailed Implementation

[0052] The embodiments of the present invention will now be described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0053] like Figures 1-3 As shown, Figure 1 This is an overall framework diagram of the Southeast Asian language scene text image recognition method based on multi-scale feature fusion provided in this embodiment of the invention. The Southeast Asian language scene text image recognition method based on multi-scale feature fusion includes the following steps:

[0054] Step 1: Southeast Asian language image preprocessing:

[0055] To avoid distortion of complex characters such as Thai and Burmese due to forced image scaling, this invention uniformly performs noise reduction processing on the input Southeast Asian language text images, and preprocesses the Southeast Asian text images by uniformly setting the image height to 32 pixels and adaptively adjusting the width from the original image size.

[0056] Step 2: Using the Southeast Asian language image multi-scale feature extraction module, a multi-scale feature extraction method is adopted to capture the global and local features of the text image through a multi-scale encoder.

[0057] To address the issue of insufficient fine-grained edge feature extraction in traditional recognition methods for Southeast Asian language text images in complex scenes, Burmese text image recognition in natural scenes is often hampered by factors such as text tilt, uneven lighting, low image resolution, and artistic fonts. This leads to limitations in the model's ability to extract fine-grained edge features from text images, especially for small-scale characters such as tone marks and vowel extension symbols, which are prone to recognition errors or feature loss (e.g., in recognizing...). Easily lost or (The presence of superscript and subscript information affects the overall recognition accuracy). To address this issue, this invention employs a multi-scale feature extraction method. A multi-scale encoder captures both global features (low-frequency global structure) and local features (high-frequency local details) of the text image, combining complementary information to improve the recognition accuracy of complex Southeast Asian language texts. Low-frequency features typically represent the global structure of the text, while high-frequency features capture local details. By capturing information at different levels in Southeast Asian language text images, complementary information can be obtained from both global and local perspectives, thereby improving the accuracy of text recognition.

[0058] Further, the specific steps of Step2 are:

[0059] Step2.1, Given a text image I∈R H×W×3 Where H and W are the height and width of the image respectively, a lightweight convolutional network EfficientNet is used as the backbone network for feature extraction, and the original image features are extracted;

[0060] Step2.2, In order to enhance the model's perception of text structure, edge information is introduced, and a Canny edge detector is applied to extract edge features from the text image I to generate an edge map E∈R H×W×1 Where the value of the edge pixel is 1 and the value of the non-edge pixel is 0;

[0061] Subsequently, a lightweight convolutional network EfficientNet is also used to extract features from the edge map E, thereby further capturing the relationship between edge pixels and generating edge features that are helpful for text recognition;

[0062] Step2.3, In the output feature map of the backbone network and the edge map, adaptive 2D position encoding is added respectively, which helps the model to better locate and align the text information in the image in space; The specific calculation formula is as follows:

[0063]

[0064] Where p represents the pixel position, i is the dimension index, f is the feature dimension, TE x (p,2i) is the feature map after adding position encoding, TE x (p,2i+1) is the edge map after adding position encoding, and canny represents the binary image after canny algorithm;

[0065] Step2.4, By recursively downsampling the original image features, image features are extracted on three scales F0∈H / 4×H / 4×D, F1∈H / 8×H / 8×D, F2∈H / 16×H / 16×D. In this way, image detail information and global structure at different scales are gradually captured;

[0066] Step2.5, In order to further improve the feature representation of each scale, a Transformer-based self-attention module (Multi-Head Self-Attention, MSA) is used to refine the features at each scale, and the specific calculation formula is as follows:

[0067]

[0068] Among them, F i The feature map representing the i-th scale. These represent the weight matrices of Q, K, and V at different scales, respectively. FFN represents the feedforward network, and MSA represents multi-head self-attention. Through this multi-level feature extraction method that integrates local and global information, the model can effectively extract text features with rich details.

[0069] Step 3: Through the Southeast Asian language image progressive cross-scale interaction module, after extracting the multi-scale features of Southeast Asian language text images, a progressive cross-scale interaction mechanism is introduced. Through a recursive cross-attention structure, the feature interaction between different scales is enhanced.

[0070] Specifically, we capture the interaction between character details and overall structure in Southeast Asian language texts, design a cross-scale Transformer attention mechanism, and optimize the feature representation at the current scale by interacting with feature information from adjacent scales. This improves the accuracy of local features in Southeast Asian language images while ensuring the consistency of the global text structure.

[0071] Furthermore, Step 3 includes:

[0072] Step 3.1: Enhance feature interaction between different scales through a progressive cross-scale interaction mechanism to capture the interaction between character details and overall structure in Southeast Asian language texts;

[0073] Specifically, suppose the feature representation of the i-th layer is F i By using a cross-scale attention module and the features F from the previous scale i-1 The interaction process yields an improved feature representation; this interaction is described by the following formula:

[0074]

[0075] Where CA represents the cross-attention function, F i-1 and F i σ represents the features at adjacent scales, and σ is the scaling factor. The progressive cross-scale interaction mechanism enables the model to enhance the recognition ability of Southeast Asian language characters based on the relationship between low-frequency global features and high-frequency local features.

[0076] Step 4: After optimizing the features at the current scale through the adaptive fusion module of multi-scale features of Southeast Asian language images, the adaptive fusion mechanism is adopted to perform weighted fusion of features at different scales based on multi-scale feature extraction and cross-scale interaction through a dynamic weighting mechanism.

[0077] By dynamically weighting each scale feature, the contribution of each scale can be automatically adjusted according to the importance of each scale feature in recognition, and finally a more rich and accurate text feature representation, i.e. the final image visual feature, is generated.

[0078] Further, the specific steps of Step4 are:

[0079] Step4.1, an adaptive fusion mechanism is used to weight and fuse features of different scales; by dynamically weighting each scale feature, the contribution of each scale can be automatically adjusted according to the importance of each scale feature in recognition, and finally a more rich and accurate text feature representation, i.e. the final image visual feature F, is generated; feature fusion is realized by the following formula:

[0080] A = Sigm(W0[F'0, F'1, F'2])

[0081] F = A O (W1[F'0, F'1, F'2])

[0082] Where [F'0, F'1, F'2] represents splicing features of different scales, W0 and W1 are learned weight matrices, Sigm is a sigmoid activation function, and O represents an element-level multiplication operation.

[0083] Through the adaptive fusion mechanism, the model can find the most useful information in the multi-scale features and obtain the final image visual feature through weighted fusion.

[0084] Step5, through the Southeast Asian language and character recognition module, after obtaining the final image visual feature, a four-layer Transformer decoder is used to introduce a multi-head attention mechanism to enhance the modeling ability of complex relationships between encoding features and improve the alignment accuracy of output characters and encoding features, and finally map the final image visual feature to the corresponding Southeast Asian language character sequence.

[0085] Further, the specific steps of Step5 are:

[0086] Step5.1, after obtaining the final image visual feature F, the final image visual feature F is passed through a multi-head attention mechanism to enhance the modeling ability of complex relationships between encoding features and improve the alignment accuracy of output characters and encoding features, and finally a cross-entropy loss function is used as an optimization function to map to the corresponding Southeast Asian language character sequence, and the formula of the optimization function is:

[0087]

[0088] Where M represents an input Southeast Asian language image, θ represents a set of learnable parameters of the recognition model, and y represents the true label corresponding to the tth feature sequence.

[0089] The loss function guides the model training by minimizing the difference between the predicted distribution and the true label, thereby improving the recognition accuracy.

[0090] The application also provides a Southeast Asian language scene text image recognition system based on multi-scale feature fusion, which comprises:

[0091] A Southeast Asian language image preprocessing module is configured to perform noise reduction processing on the input Southeast Asian language text image, and uniformly adjust the height of the image and adaptively adjust the width according to the size of the original image to preprocess the Southeast Asian text image.

[0092] A global feature and local feature extraction module is configured to capture global features and local features of the text image by using a multi-scale feature extraction method and a multi-scale encoder.

[0093] A Southeast Asian language image progressive cross-scale interaction module is configured to introduce a progressive cross-scale interaction mechanism, and enhance the feature interaction between different scales through a recursive cross-attention structure.

[0094] Specifically, the interaction between character details and overall structure in Southeast Asian language text is captured, a cross-scale Transformer attention mechanism is designed, and the feature representation of the current scale is optimized by interacting with the feature information of adjacent scales.

[0095] A Southeast Asian language image multi-scale feature adaptive fusion module is configured to, after optimizing the features of the current scale, adopt an adaptive fusion mechanism to weight and fuse the features of different scales based on multi-scale feature extraction and cross-scale interaction through a dynamic weighting mechanism.

[0096] Through dynamic weighting of features of each scale, the contribution of each scale can be automatically adjusted according to the importance of the features of each scale in recognition, and finally a more rich and accurate text feature representation, i.e., the final image visual feature, is generated.

[0097] A Southeast Asian language character recognition module is configured to, after obtaining the final image visual feature, adopt a four-layer Transformer decoder, introduce a multi-head attention mechanism to enhance the modeling ability of complex relationships between encoded features and improve the alignment accuracy of output characters and encoded features, and map the final image visual feature to a corresponding Southeast Asian language character sequence.

[0098] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the Southeast Asian language scene text image recognition method based on multi-scale feature fusion when executing the program.

[0099] The application further provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0100] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the Southeast Asian language scene text image recognition method based on multi-scale feature fusion.

[0101] In order to verify the effectiveness of the Southeast Asian language scene text image recognition method based on multi-scale feature fusion proposed by the application, the following experiments are carried out for detailed analysis.

[0102] Experiment 1: main experiment analysis

[0103] The main experimental results are shown in Table 1.

[0104] Table 1 is the main comparative experiment results

[0105]

[0106]

[0107] The performances of various methods in Southeast Asian language scene text recognition (STR) tasks are compared, covering CTC, Seq2Seq and Transformer-based model architectures. The experimental results show that there are significant differences in recognition accuracy (SA) and parameter quantity (Parameters) among different methods, and the specific analysis is as follows.

[0108] (1) The method proposed by the application performs excellently in Southeast Asian language text recognition tasks, with an accuracy of 97.6%, which is significantly better than most of the comparative methods. A good balance between accuracy and parameter quantity is achieved, and compared with the Transformer-based method, the method of the application is close to the optimal MGP-Base model (97.2%) in accuracy, but the parameter quantity is only 46.5M, which is much lower than the 95.1M of MGP-Base, showing higher efficiency. In addition, compared with the CTC and Seq2Seq methods, the method of the application leads by a large margin in accuracy (such as VGG16+BiLSTM+CTC is 90.4%, and VGG16+BiLSTM+Attention is 90.6%), while maintaining a relatively low parameter quantity, indicating its advantages in feature extraction and sequence modeling.

[0109] In the specific method for image recognition of Southeast Asian languages, the accuracy (97.6%) of the method of the present application is close to the current optimal LBR-B model (97.9%), but the parameter amount (43.5M) is significantly lower than the 97.2M of LBR-B, further embodying the balance of the method of the present application in performance and efficiency. Compared with the method proposed by He et al. (97.3%, 115.1M), the parameter amount of the method of the present application is reduced by about 62% under the condition of equivalent accuracy, which is more suitable for actual application scenarios.

[0110] Experiment two: multi-scale feature extraction network ablation experiment

[0111] To verify the effectiveness of the method of the present application and the contribution of each module, an ablation study of the system is designed and implemented. By controlling the variable method, the key components of the model are removed in turn, and the influence of each module on the final recognition performance is quantitatively evaluated, so as to deeply understand the operation mechanism of the model and the importance of each part. The following is an introduction to the removed modules:

[0112] (1) Cross-scale interaction module (Csi): This module is used to perform cross-scale feature interaction on multi-scale visual features, further enhancing the visual features of each scale.

[0113] (2) Character contour feature fusion (Cof): This module is used to extract and fuse the contour features of characters to enhance the model's perception of Southeast Asian language character shapes and structures.

[0114] Table 2 is the ablation experiment results of multi-granularity features

[0115]

[0116] The experimental results show that after removing the Csi module, the recognition accuracy (SA) of the model decreases from 97.6% to 96.7%, a decrease of 0.9 percentage points. This result shows that the Csi module plays a significant role in improving the model's ability to fuse multi-scale features. Through cross-scale interaction, the model can better capture the local details and global structure information of characters, thereby improving the recognition performance. Especially when dealing with complex text or low-resolution images, the role of the Csi module is more obvious.

[0117] And the character contour feature fusion module (Cof) is used for extracting and fusing the contour features of the character, so as to enhance the perception ability of the model to the shape and structure of the Southeast Asian language character. The experimental results show that after removing the Cof module, the recognition accuracy of the model decreases from 97.6% to 97.3%, a decrease of 0.3%. Although the decrease is small, the Cof module still plays an important role in improving the recognition ability of the model to complex characters. Especially in dealing with Southeast Asian language characters, due to their complex stroke structure and morphological changes, contour features can provide additional shape information to the model, making up for the shortcomings of traditional texture features.

[0118] Experiment three: multi-scale representation ablation experiment of text image;

[0119] Table 3 is the experimental results of different scale feature fusion

[0120]

[0121] Table 3 shows the results of the multi-scale representation ablation experiment of the text image, which verifies the effectiveness of the multi-scale feature fusion strategy by comparing the recognition accuracy (SA) under different scale combinations. As can be seen from the table, with the increase of the number of scales, the recognition performance of the model gradually improves. Specifically, when using only a single scale (Scale 0), the recognition accuracy of the model is 95.1%. Although this result has reached a high level, there is still some limitation because a single scale is difficult to capture both the global structure and the local details of the text. When the second scale (Scale 0,1) is introduced, the recognition accuracy increases to 96.3%, which shows that by fusing the features of two scales, the model can better balance the global information and local details, thereby improving the recognition performance. After further introducing the third scale (Scale 0,1,2), the recognition accuracy reaches 97.3%, which verifies the effectiveness of the multi-scale feature fusion strategy, indicating that more scales of features can provide richer text representation, thereby further improving the recognition ability of the model.

[0122] Finally, based on the fusion of three scales, the recognition accuracy is further improved to 97.6% through the progressive cross-scale interaction and adaptive feature fusion mechanism. This result shows that the multi-scale feature fusion framework proposed in the present application can not only effectively integrate features of different scales, but also further optimize feature representation through dynamic interaction and weighted fusion mechanism, so as to achieve optimal performance in the text recognition task.

[0123] Experiment four: comparison and analysis of recognition accuracy of multiple mainstream models on real data set;

[0124] Table 4 is the experimental results of multiple models on real scene data set

[0125]

[0126] In Table 4, the method proposed in the present application achieves an identification accuracy (SA) of 95.7% on the real data set, which is the best among all the methods listed, slightly higher than 95.5% of the LBR-B model and 95.3% of the method proposed by He et al. This result shows that the method in the present application has a significant advantage in the Southeast Asian language text recognition task, and its performance improvement is mainly due to the design of the multi-scale feature fusion framework and the introduction of the progressive cross-scale interaction and adaptive feature fusion mechanism, which strengthens the recognition accuracy of the model in real scenarios.

[0127] To verify the application effect of the method proposed in the present application in real complex scenarios, the detailed test results of the model in actual scene text images are shown below, as shown in Figure 3

[0128] The experimental results are shown in Figure 3 . The figure contains four test examples, which show the input image samples, the contour maps generated by the method in the present application, the final recognition results of LBR-B and the method in the present application. By comparing the contour maps, the advantages of the method in the present application in edge feature extraction can be intuitively evaluated, especially in artistic fonts, tilting, complex backgrounds and low resolution. Specifically, due to the deformation of artistic fonts and image tilting, the recognition results of part of the character clusters or syllables in the first and second groups of samples have errors. The third and fourth groups of samples are affected by light changes and image blurring, and the existing methods are difficult to completely retain the edge features of the image, resulting in small target characters (such as tone marks and vowel extension symbols) being easily missed, thereby affecting the final recognition accuracy. The experimental results show that the method in the present application has a significant advantage in multi-scale feature fusion and edge feature extraction, and can effectively enhance the recognition accuracy and robustness in complex scenarios.

[0129] The specific embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.​

Claims

1. A Southeast Asian language scene text image recognition method based on multi-scale feature fusion, characterized in that, Comprising the following steps: Step1, Southeast Asian language image preprocessing: The input Southeast Asian language text image is denoised, and the height of the image is unified, and the width is adaptively adjusted according to the size of the original image to preprocess the Southeast Asian text image; Step2, through the Southeast Asian language image multi-scale feature extraction module, a multi-scale feature extraction method is adopted, and the global features and local features of the text image are captured through a multi-scale encoder; Step3, through the Southeast Asian language image gradual cross-scale interaction module, a gradual cross-scale interaction mechanism is introduced, and through a recursive cross-attention structure, the feature interaction between different scales is enhanced; Specifically, the interaction between character details and overall structure in Southeast Asian language text is captured, a cross-scale Transformer attention mechanism is designed, and the feature representation of the current scale is optimized through interaction with the feature information of adjacent scales; Step4, through the adaptive fusion module of Southeast Asian language image multi-scale features, after optimizing the features of the current scale, an adaptive fusion mechanism is adopted, and based on multi-scale feature extraction and cross-scale interaction, the features of different scales are weighted and fused through a dynamic weighting mechanism; Through dynamic weighting of each scale feature, the contribution of each scale can be automatically adjusted according to its importance in recognition, and a more rich and accurate text feature representation, i.e. the final image visual feature, is generated; Step5, through the Southeast Asian language character recognition module, after obtaining the final image visual feature, a four-layer Transformer decoder is used to introduce a multi-head attention mechanism to enhance the modeling ability of complex relationships between encoded features and improve the alignment accuracy of output characters and encoded features. The final image visual feature is mapped to the corresponding Southeast Asian language character sequence.

2. The method of claim 1, wherein the method is a multi-scale feature fusion based scene text image recognition method for Southeast Asian languages. The specific steps of Step2 are as follows: Step2.1、Given a text image I∈R H×W×3 Where H and W are the height and width of the image respectively, a lightweight convolutional network EfficientNet is used as the backbone network for feature extraction, and the original image features are preliminarily extracted. Step 2.2, introduce edge information, apply Canny edge detector to text image I to extract edge features to generate an edge map E e R H×W×1 where the value of edge pixels is 1 and the value of non-edge pixels is 0; Then, the edge map E is also feature-extracted using a lightweight convolutional network EfficientNet, thereby further capturing the relationship between edge pixels and generating edge features that are conducive to text recognition; Step2.3, adaptive 2D position encoding is added to the output feature map of the backbone network and the edge map, thereby helping the model to better locate and align the text information in the image in space; The specific calculation formula is as follows: where p represents a pixel position, i is a dimension index, f is a feature dimension, TE x (p,2i) is a feature map with position encoding added, TE x (p,2i+1) is an edge map with position encoding added, canny represents a binary image after canny algorithm Step2.4, by recursively downsampling the original image features, image features are extracted on three scales F0∈H / 4×H / 4×D, F1∈H / 8×H / 8×D, F2∈H / 16×H / 16×D, and in this way, image detail information and global structure at different scales are gradually captured; Step2.5, in order to further improve the feature representation of each scale, a Transformer-based self-attention module MSA is used to refine the features of each scale, and the specific calculation formula is as follows: where F i represent the feature maps of the i-th scale, represent the weight matrices of Q, K and V at different scales, respectively, FFN represents a feed-forward network, and MSA represents multi-head self-attention; through this multi-level feature extraction mode that fuses local and global information, text features with rich details can be effectively extracted. 3.The method of claim 1, wherein the method further comprises: determining a plurality of feature maps of the input image; and determining a plurality of feature maps of the input image based on the plurality of feature maps of the input image. Step3 includes: Step3.1, the feature interaction between different scales is enhanced by a progressive cross-scale interaction mechanism to capture the interaction between character details and overall structure in Southeast Asian language text; Specifically, assume the feature representation of the i-th layer is F i , through the cross-scale attention module and the previous scale feature F i-1 , the improved feature representation is obtained; the interaction process is described by the following formula: where CA denotes the cross-attention function, F i-1 and F i are the features of adjacent scales, and σ is a scaling factor; the progressive cross-scale interaction mechanism enables the model to enhance the recognition ability of Southeast Asian language characters according to the relationship between the low-frequency global features and the high-frequency local features.

4. The method of claim 1, wherein the method is a scene text image recognition method for Southeast Asian languages based on multi-scale feature fusion. The specific steps of Step4 are: Step4.1, an adaptive fusion mechanism is used to weight and fuse features of different scales; by dynamically weighting the features of each scale, the importance of each scale feature in recognition can be automatically adjusted to adjust the contribution of each scale, and finally more rich and accurate text feature representation, i.e., the final image visual feature F, is generated; feature fusion is realized by the following formula: A=Sigm(W0[F′0,F′1,F′2]) F=A⊙(W1[F′0,F′1,F′2]) Where [F′0,F′1,F′2] represents splicing features of different scales, W0 and W1 are learned weight matrices, Sigm is a sigmoid activation function, and ⊙ represents an element-level multiplication operation; Through the adaptive fusion mechanism, the model can find the most useful information in the multi-scale features and obtain the final image visual feature through weighted fusion.

5. The method of claim 1, wherein the method is a scene text image recognition method for Southeast Asian languages based on multi-scale feature fusion. The specific steps of Step5 are: Step5.1, after obtaining the final image visual feature F, the final image visual feature F is passed through a multi-head attention mechanism to enhance the modeling ability of complex relationships between encoded features and improve the alignment accuracy of output characters and encoded features. Finally, a cross-entropy loss function is used as an optimization function to map to the corresponding Southeast Asian language character sequence, and the formula of the optimization function is: Where M represents the input Southeast Asian language image, θ represents the set of learnable parameters of the recognition model, and y represents the true label corresponding to the t-th feature sequence. The loss function guides the model training by minimizing the difference between the predicted distribution and the true label, thereby improving the recognition accuracy.

6. A Southeast Asian language scene text image recognition system based on multi-scale feature fusion, characterized in that, The system comprises a module for executing the Southeast Asian language scene text image recognition method based on multi-scale feature fusion as claimed in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the Southeast Asian language scene text image recognition method based on multi-scale feature fusion as claimed in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the Southeast Asian language scene text image recognition method based on multi-scale feature fusion as claimed in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the Southeast Asian language scene text image recognition method based on multi-scale feature fusion as claimed in any one of claims 1 to 5.

Images